Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

read data from hdfs #1

Open
formath opened this issue Aug 31, 2016 · 1 comment
Open

read data from hdfs #1

formath opened this issue Aug 31, 2016 · 1 comment
Assignees
@reyoung
Labels
question status/developing 开发中

Comments

@formath
Copy link

formath commented Aug 31, 2016

"Different node should owns different parts of all Train data. This simple script did not do this job, so you should prepare it at last. " I saw this in cluster training wiki. So, could paddle read data from hdfs and distribute data to each node automatically?
@reyoung reyoung self-assigned this Aug 31, 2016
@reyoung
Copy link
Collaborator

reyoung commented Aug 31, 2016

Distribute data to cluster is not added in PaddlePaddle now. You can read data directly from a HDFS file path by PyDataProvider2.

PaddlePaddle not handle how to get data file remotely, just pass the file path into a Python function. It is user's job to OPEN the file (or SQL connection string, or HDFS path), and get each
sample one by one from it.

It is welcome to contribute a script to distribute data to cluster. Or we may add it soon if this feature is very necessary.

qingqing01 pushed a commit that referenced this issue Sep 14, 2016
@emailweixu
Merge pull request #1 from baidu/master
5c6ecb2 
Update from the original
reyoung referenced this issue in reyoung/Paddle Sep 21, 2016
@gangliao
Merge pull request #1 from reyoung/mac_port
4315a38 
revise CMake for MAC OS
wangkuiyi added a commit that referenced this issue Dec 1, 2016
@wangkuiyi
Merge pull request #1 from reyoung/docker_openssh
bb13328 
Refine cmake version tag.
reyoung pushed a commit that referenced this issue Dec 5, 2016
@hohdiy
Merge pull request #1 from PaddlePaddle/develop
021b3a4 
merge PaddlePaddle/Paddle
luotao1 pushed a commit that referenced this issue Dec 28, 2016
@CrossLee1
Merge pull request #1 from PaddlePaddle/develop
4a0cb3d 
update from origin
backyes pushed a commit that referenced this issue Dec 30, 2016
@tizhou86
Merge pull request #1 from wangkuiyi/aws-yi
7ca14b6 
Rephrase the first paragraph
@sarawon sarawon mentioned this issue Feb 21, 2017
Closed
qingqing01 pushed a commit that referenced this issue Mar 20, 2017
@Noplz
Merge pull request #1 from PaddlePaddle/develop
2a9d71a 
Update from the origin
@sdujq sdujq mentioned this issue May 5, 2017
Closed
@xiehongweiscut xiehongweiscut mentioned this issue Aug 3, 2017
Closed
This was referenced Aug 17, 2017
Closed
Closed
@NHZlX NHZlX mentioned this issue Aug 22, 2017
Closed
reyoung pushed a commit that referenced this issue Sep 5, 2017
@qingqing01
Merge pull request #1 from reyoung/grad_test_for_multi_inputs
95955cc 
Invoke check_grad many times for no_grad_set
@fty8788 fty8788 mentioned this issue Sep 6, 2017
Closed
@fsfszongming256 fsfszongming256 mentioned this issue Sep 8, 2017
Closed
zchen0211 added a commit that referenced this issue Sep 14, 2017
@zchen0211
Merge pull request #1 from reyoung/czy_elemwise
b2d9c91 
Fix CI test
@wanghaoshuang wanghaoshuang mentioned this issue Sep 19, 2017
Closed
qingqing01 pushed a commit that referenced this issue Sep 19, 2017
@Noplz
Merge pull request #1 from PaddlePaddle/develop
462c2ed 
Update
luotao1 pushed a commit that referenced this issue Oct 16, 2017
@cxysteven
Merge pull request #1 from PaddlePaddle/master
4bc5bf9 
Master
wanghaox added a commit that referenced this issue Nov 24, 2017
@wanghaox
Merge pull request #1 from PaddlePaddle/develop
ca988bd 
update from origin
typhoonzero added a commit that referenced this issue Dec 2, 2017
@typhoonzero
Merge pull request #1 from seiriosPlus/tangw/simple_pipe_reader
cd36531 
fix pipe_reader unimport packages
@tensor-tang tensor-tang mentioned this issue Dec 5, 2017
Merged
@yyhlvdl yyhlvdl mentioned this issue Dec 11, 2017
Closed
@fty8788 fty8788 mentioned this issue Jan 23, 2018
Closed
@pkuyym pkuyym mentioned this issue Jan 24, 2018
Closed
@marsbzp marsbzp mentioned this issue Jan 11, 2023
Open
@VBPython VBPython mentioned this issue Jan 29, 2023
Closed
qizhaoaoe pushed a commit to qizhaoaoe/Paddle that referenced this issue Mar 3, 2023
@guru4elephant
Merge pull request PaddlePaddle#1 from sandyhouse/init_release
f487345 
init commit
@ShaominFan ShaominFan mentioned this issue Mar 3, 2023
Open
Silv3S pushed a commit to Silv3S/Paddle that referenced this issue Mar 14, 2023
@HulekJakub
Merge pull request PaddlePaddle#1 from Silv3S/fused_elementwise_ops
e4be4cc 
Fused elementwise ops
@liumy1111 liumy1111 mentioned this issue Mar 15, 2023
Closed
tsocha pushed a commit to tsocha/Paddle that referenced this issue Mar 16, 2023
@jczaja
Merge pull request PaddlePaddle#1 from jakpiase/3.0_hard_swish_fix
bbf9a73 
[OneDNN 3.0] Fix for hard_swish
yuanlehome referenced this issue in yuanlehome/Paddle Mar 27, 2023
@jiweibo
Merge pull request #1 from Shixiaowei02/dev/pass_2
5edbbd9 
adds the base classes for analyses
@chlyzzo chlyzzo mentioned this issue Mar 29, 2023
Closed
@hw446 hw446 mentioned this issue Mar 31, 2023
Open
PeiyuLau pushed a commit to PeiyuLau/Paddle that referenced this issue Jun 8, 2023
@fwenguang
[MLU] fix bn and enable tf32 (PaddlePaddle#1)
4ef39c2
feifei-111 referenced this issue in feifei-111/Paddle Jun 14, 2023
@2742195759
Merge pull request #1 from NotHaozi/pr54202
632b36a 
Pr54202
@LangJiHuangSha LangJiHuangSha mentioned this issue Jun 19, 2023
Open
@ga1008 ga1008 mentioned this issue Jul 26, 2023
Closed
@MyAngelAyase MyAngelAyase mentioned this issue Aug 28, 2023
Closed
@cqli0905 cqli0905 mentioned this issue Sep 8, 2023
Open
@HelloRay123 HelloRay123 mentioned this issue Sep 9, 2023
Closed
@paddle-bot paddle-bot bot added the status/developing 开发中 label Sep 22, 2023
@paddle-bot paddle-bot bot reopened this Sep 22, 2023
@kane-hu kane-hu mentioned this issue Sep 23, 2023
Closed
@kane-hu kane-hu mentioned this issue Oct 8, 2023
Closed
@hubimaso hubimaso mentioned this issue Nov 3, 2023
Closed
tianyan01 pushed a commit to tianyan01/Paddle that referenced this issue Nov 23, 2023
@laipaang
Merge pull request PaddlePaddle#1 from tianyan01/v2.4.2
052c62e 
modify for dynamic zeus
lizexu123 pushed a commit to lizexu123/Paddle that referenced this issue Feb 23, 2024
@wanghaoshuang
Merge pull request PaddlePaddle#1 from wanghaoshuang/doc
775aa12 
Rename docs-src to docs and rename demo to tutorials.
hanhaowen-mt referenced this issue in hanhaowen-mt/Paddle Feb 29, 2024
@caizhi-mt @mt-robot
Merge pull request #1 from mthreads/mingyuanw/ci
8625655 
[MTAI-489] build(ci): test CI
NKNaN pushed a commit to NKNaN/Paddle that referenced this issue Mar 3, 2024
@ekelsen
Merge pull request PaddlePaddle#1 from jaredcasper/master
f7d1190 
Correct license in rockspec file.
Fridge003 pushed a commit to Fridge003/Paddle that referenced this issue Mar 20, 2024
@feifei-111
Merge pull request PaddlePaddle#1 from Fridge003/cinn_tmp
dfee88f 
spliting
@YuanRisheng YuanRisheng mentioned this issue Apr 22, 2024
Open
@Juruobudong Juruobudong mentioned this issue May 14, 2024
Open
@lemonish lemonish mentioned this issue May 23, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@reyoung reyoung

Labels
question status/developing 开发中
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@reyoung @gangliao @formath