Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

read data from hdfs #1

Closed
formath opened this issue Aug 31, 2016 · 2 comments
Closed

read data from hdfs #1

formath opened this issue Aug 31, 2016 · 2 comments
Assignees
@reyoung
Labels
question status/developing 开发中

Comments

@formath
Copy link

formath commented Aug 31, 2016

"Different node should owns different parts of all Train data. This simple script did not do this job, so you should prepare it at last. " I saw this in cluster training wiki. So, could paddle read data from hdfs and distribute data to each node automatically?
@reyoung reyoung self-assigned this Aug 31, 2016
@reyoung
Copy link
Collaborator

reyoung commented Aug 31, 2016

Distribute data to cluster is not added in PaddlePaddle now. You can read data directly from a HDFS file path by PyDataProvider2.

PaddlePaddle not handle how to get data file remotely, just pass the file path into a Python function. It is user's job to OPEN the file (or SQL connection string, or HDFS path), and get each
sample one by one from it.

It is welcome to contribute a script to distribute data to cluster. Or we may add it soon if this feature is very necessary.

qingqing01 pushed a commit that referenced this issue Sep 14, 2016
@emailweixu
Merge pull request #1 from baidu/master
5c6ecb2 
Update from the original
reyoung referenced this issue in reyoung/Paddle Sep 21, 2016
@gangliao
Merge pull request #1 from reyoung/mac_port
4315a38 
revise CMake for MAC OS
wangkuiyi added a commit that referenced this issue Dec 1, 2016
@wangkuiyi
Merge pull request #1 from reyoung/docker_openssh
bb13328 
Refine cmake version tag.
reyoung pushed a commit that referenced this issue Dec 5, 2016
@hohdiy
Merge pull request #1 from PaddlePaddle/develop
021b3a4 
merge PaddlePaddle/Paddle
luotao1 pushed a commit that referenced this issue Dec 28, 2016
@CrossLee1
Merge pull request #1 from PaddlePaddle/develop
4a0cb3d 
update from origin
backyes pushed a commit that referenced this issue Dec 30, 2016
@tizhou86
Merge pull request #1 from wangkuiyi/aws-yi
7ca14b6 
Rephrase the first paragraph
@sarawon sarawon mentioned this issue Feb 21, 2017
Closed
qingqing01 pushed a commit that referenced this issue Mar 20, 2017
@Noplz
Merge pull request #1 from PaddlePaddle/develop
2a9d71a 
Update from the origin
@sdujq sdujq mentioned this issue May 5, 2017
Closed
@xiehongweiscut xiehongweiscut mentioned this issue Aug 3, 2017
Closed
This was referenced Aug 17, 2017
Closed
Closed
@NHZlX NHZlX mentioned this issue Aug 22, 2017
Closed
reyoung pushed a commit that referenced this issue Sep 5, 2017
@qingqing01
Merge pull request #1 from reyoung/grad_test_for_multi_inputs
95955cc 
Invoke check_grad many times for no_grad_set
@fty8788 fty8788 mentioned this issue Sep 6, 2017
Closed
@fsfszongming256 fsfszongming256 mentioned this issue Sep 8, 2017
Closed
zchen0211 added a commit that referenced this issue Sep 14, 2017
@zchen0211
Merge pull request #1 from reyoung/czy_elemwise
b2d9c91 
Fix CI test
@wanghaoshuang wanghaoshuang mentioned this issue Sep 19, 2017
Closed
qingqing01 pushed a commit that referenced this issue Sep 19, 2017
@Noplz
Merge pull request #1 from PaddlePaddle/develop
462c2ed 
Update
luotao1 pushed a commit that referenced this issue Oct 16, 2017
@cxysteven
Merge pull request #1 from PaddlePaddle/master
4bc5bf9 
Master
wanghaox added a commit that referenced this issue Nov 24, 2017
@wanghaox
Merge pull request #1 from PaddlePaddle/develop
ca988bd 
update from origin
typhoonzero added a commit that referenced this issue Dec 2, 2017
@typhoonzero
Merge pull request #1 from seiriosPlus/tangw/simple_pipe_reader
cd36531 
fix pipe_reader unimport packages
@tensor-tang tensor-tang mentioned this issue Dec 5, 2017
Merged
@yyhlvdl yyhlvdl mentioned this issue Dec 11, 2017
Closed
@fty8788 fty8788 mentioned this issue Jan 23, 2018
Closed
@pkuyym pkuyym mentioned this issue Jan 24, 2018
Closed
@hw446 hw446 mentioned this issue Mar 31, 2023
Closed
PeiyuLau pushed a commit to PeiyuLau/Paddle that referenced this issue Jun 8, 2023
@fwenguang
[MLU] fix bn and enable tf32 (PaddlePaddle#1)
4ef39c2
feifei-111 referenced this issue in feifei-111/Paddle Jun 14, 2023
@2742195759
Merge pull request #1 from NotHaozi/pr54202
632b36a 
Pr54202
@LangJiHuangSha LangJiHuangSha mentioned this issue Jun 19, 2023
Closed
@ga1008 ga1008 mentioned this issue Jul 26, 2023
Closed
@MyAngelAyase MyAngelAyase mentioned this issue Aug 28, 2023
Closed
@cqli0905 cqli0905 mentioned this issue Sep 8, 2023
Closed
@HelloRay123 HelloRay123 mentioned this issue Sep 9, 2023
Closed
@paddle-bot paddle-bot bot added the status/developing 开发中 label Sep 22, 2023
@paddle-bot paddle-bot bot reopened this Sep 22, 2023
@kane-hu kane-hu mentioned this issue Sep 23, 2023
Closed
@kane-hu kane-hu mentioned this issue Oct 8, 2023
Closed
@hubimaso hubimaso mentioned this issue Nov 3, 2023
Closed
tianyan01 pushed a commit to tianyan01/Paddle that referenced this issue Nov 23, 2023
@laipaang
Merge pull request PaddlePaddle#1 from tianyan01/v2.4.2
052c62e 
modify for dynamic zeus
lizexu123 referenced this issue in lizexu123/Paddle Feb 23, 2024
@wanghaoshuang
Merge pull request #1 from wanghaoshuang/doc
775aa12 
Rename docs-src to docs and rename demo to tutorials.
hanhaowen-mt referenced this issue in hanhaowen-mt/Paddle Feb 29, 2024
@caizhi-mt @mt-robot
Merge pull request #1 from mthreads/mingyuanw/ci
8625655 
[MTAI-489] build(ci): test CI
NKNaN pushed a commit to NKNaN/Paddle that referenced this issue Mar 3, 2024
@ekelsen
Merge pull request PaddlePaddle#1 from jaredcasper/master
f7d1190 
Correct license in rockspec file.
Fridge003 pushed a commit to Fridge003/Paddle that referenced this issue Mar 20, 2024
@feifei-111
Merge pull request PaddlePaddle#1 from Fridge003/cinn_tmp
dfee88f 
spliting
@YuanRisheng YuanRisheng mentioned this issue Apr 22, 2024
Closed
@Juruobudong Juruobudong mentioned this issue May 14, 2024
Open
@lemonish lemonish mentioned this issue May 23, 2024
Open
kircle888 pushed a commit to kircle888/Paddle that referenced this issue Jul 7, 2024
@kuizhiqing
CAPI for paddle (PaddlePaddle#1)
5994ce0 
* provide capi for flash attention

* cuda enforce; flash error

* fix api

* fix zero tensors

* fix softmax_ptr size
kircle888 added a commit to kircle888/Paddle that referenced this issue Jul 7, 2024
@kircle888
Merge pull request PaddlePaddle#1 from kircle888/smask_up
c3212be 
Smask up ready
@saysbye saysbye mentioned this issue Jul 8, 2024
Closed
@Linda-Deng Linda-Deng mentioned this issue Jul 18, 2024
Open
ckl117 pushed a commit to ckl117/Paddle that referenced this issue Jul 18, 2024
@ming1753
Merge pull request PaddlePaddle#1 from ckl117/adi_fm
933d218 
TRT index_put
Layssy pushed a commit to Layssy/Paddle that referenced this issue Jul 24, 2024
@lizexu123
Merge pull request PaddlePaddle#1 from Layssy/lw_trt
fd0959e 
添加castop
@gaecom gaecom mentioned this issue Aug 22, 2024
Open
@paddle-bot paddle-bot bot closed this as completed Sep 24, 2024
@paddle-bot Paddle-bot
Copy link

paddle-bot bot commented Sep 26, 2024

Since you haven't replied for more than a year, we have closed this issue/pr.
If the problem is not solved or there is a follow-up one, please reopen it at any time and we will continue to follow up.
由于您超过一年未回复，我们将关闭这个issue/pr。
若问题未解决或有后续问题，请随时重新打开，我们会继续跟进。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@reyoung reyoung

Labels
question status/developing 开发中
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@reyoung @gangliao @formath