euler如何支持hdfs #31

SuperbDong · 2019-02-15T09:49:19Z

看到吗现在会判断hdfs_prefix='hdfs://'
若何使用viewfs:// ?

SuperbDong · 2019-02-16T12:14:11Z

指定hdfs 报错
no hdfs file io factory register
代码：
if (euler::common::factory_map.find("hdfs") ==
euler::common::factory_map.end()) {
LOG(ERROR) << "no hdfs file io factory register";
return false;
}

zakheav · 2019-02-16T12:24:34Z

如果需要支持hdfs数据加载，则需要修改项目根目录下的CMakeLists.txt，把option(USE_HDFS "Option for using hdfs" OFF)改为option(USE_HDFS "Option for using hdfs" ON)，详细过程请看编译安装部分

SuperbDong · 2019-02-16T12:28:10Z

如果需要支持hdfs数据加载，则需要修改项目根目录下的CMakeLists.txt，把option(USE_HDFS "Option for using hdfs" OFF)改为option(USE_HDFS "Option for using hdfs" ON)，详细过程请看编译安装部分

怎么用viewfs呢？开启这个option(USE_HDFS "Option for using hdfs" ON) 就可以了么

zakheav · 2019-02-16T12:44:40Z

目前我们只测试了hdfs://这种方式，不太清楚viewfs://会有什么问题。
至于打docker，
git clone --recursive https://github.com/alibaba/euler.git # 执行完这个命令之后修改一下根目录的CMakeLists.txt，设置option(USE_HDFS "Option for using hdfs" ON)，然后执行下面两句命令应该就好了，Dockerfile不用修改
cd euler
docker build --net=host -f tools/docker/Dockerfile .

SuperbDong · 2019-02-16T12:56:44Z

目前我们只测试了hdfs://这种方式，不太清楚viewfs://会有什么问题。
至于打docker，
git clone --recursive https://github.com/alibaba/euler.git # 执行完这个命令之后修改一下根目录的CMakeLists.txt，设置option(USE_HDFS "Option for using hdfs" ON)，然后执行下面两句命令应该就好了，Dockerfile不用修改
cd euler
docker build --net=host -f tools/docker/Dockerfile .

hdfs_prefix = 'hdfs://'
if not directory.startswith(hdfs_prefix):
raise ValueError('Only hdfs graph data is support for shared graph.')
directory = directory[len(hdfs_prefix):]
看代码是不支持viewfs://这种方式的，我们目前无法使用hdfs:// 因为用了federation，看下能不能支持下viewfs这种方式呢？

chengenbao · 2019-02-17T08:21:20Z

On 16 Feb 2019, at 8:13 PM, SuperbDong <notifications@github.com> wrote:指定hdfs 报错 no hdfs file io factory register—You are receiving this because you are subscribed to this thread.Reply to this email directly, view it on GitHub, or mute the thread.

SuperbDong · 2019-02-18T01:44:45Z

目前我们只测试了hdfs://这种方式，不太清楚viewfs://会有什么问题。
至于打docker，
git clone --recursive https://github.com/alibaba/euler.git # 执行完这个命令之后修改一下根目录的CMakeLists.txt，设置option(USE_HDFS "Option for using hdfs" ON)，然后执行下面两句命令应该就好了，Dockerfile不用修改
cd euler
docker build --net=host -f tools/docker/Dockerfile .

指定这个依然报no hdfs file io factory register
我把demo里面的ppi 上传到hdfs上面了，是数据不对么

chengenbao · 2019-02-18T02:51:02Z

你修改了CMakeLists.txt吗？有没有把option(USE_HDFS "Option for using hdfs" OFF)改为option(USE_HDFS "Option for using hdfs" ON)？

…

On 18 Feb 2019, at 9:44 AM, SuperbDong ***@***.***> wrote: 目前我们只测试了hdfs://这种方式，不太清楚viewfs://会有什么问题。至于打docker， git clone --recursive https://github.com/alibaba/euler.git <https://github.com/alibaba/euler.git> # 执行完这个命令之后修改一下根目录的CMakeLists.txt，设置option(USE_HDFS "Option for using hdfs" ON)，然后执行下面两句命令应该就好了，Dockerfile不用修改 cd euler docker build --net=host -f tools/docker/Dockerfile . 指定这个依然报no hdfs file io factory register 我把demo里面的ppi 上传到hdfs上面了，是数据不对么 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#31 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD20y35-_OM9iLbN94H4nY6YdyWC_eqRks5vOgWPgaJpZM4a9TEF>.

zakheav · 2019-02-18T03:20:36Z

如果编译没有问题，启动的时候会打印“hdfs file io factory register”这个日志

zakheav · 2019-02-18T03:41:02Z

如果只打印local file io factory register，说明编译没有把hdfs部分编译进去。我们在euler/filo_io_factory.cc中用宏做了选择编译：
"
#ifdef HDFS
#include "euler/common/hdfs_file_io_factory.h"
static euler::common::HdfsFileIOFactoryReg hdfs_reg;
#endif
"
如果没有在根目录的CMakeLists.txt中把设置option(USE_HDFS "Option for using hdfs" OFF)改为option(USE_HDFS "Option for using hdfs" ON)，是不会编译hdfs部分的。

zakheav · 2019-02-18T03:59:39Z

能把你的编译步骤详细再说明一下吗

chengenbao · 2019-02-18T07:24:23Z

你能确定下你的执行环境是你修改之后build的docker吗？

…

On 18 Feb 2019, at 12:00 PM, SuperbDong ***@***.***> wrote: 能把你的编译步骤详细再说明一下吗我直接build的DockerFile — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#31 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD20yzpqUlCKiXFihQyCvRSWyQzXw4Qdks5vOiVfgaJpZM4a9TEF>.

SuperbDong · 2019-02-18T08:44:13Z

希望后续能支持一下viewfs 还有xdl好像也不支持viewfs

2shou · 2019-02-18T09:30:59Z

如果需要支持hdfs数据加载，则需要修改项目根目录下的CMakeLists.txt，把option(USE_HDFS "Option for using hdfs" OFF)改为option(USE_HDFS "Option for using hdfs" ON)，详细过程请看编译安装部分

如果用pip安装的，需要手动开启这个开关不

chengenbao · 2019-02-18T09:33:37Z

pip安装的不支持hdfs

…

On 18 Feb 2019, at 5:31 PM, Gavin Zhang ***@***.***> wrote: 如果需要支持hdfs数据加载，则需要修改项目根目录下的CMakeLists.txt，把option(USE_HDFS "Option for using hdfs" OFF)改为option(USE_HDFS "Option for using hdfs" ON)，详细过程请看编译安装部分如果用pip安装的，需要手动开启这个开关不 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#31 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD20y2jvOq8p3Ejg6uUvRdKbjyRic42bks5vOnLUgaJpZM4a9TEF>.

chengenbao · 2019-02-18T09:34:52Z

这个我们会评估下，其实文件系统这块我们提供了抽象，用户可以自己扩展

…

On 18 Feb 2019, at 4:44 PM, SuperbDong ***@***.***> wrote: 希望后续能支持一下viewfs 还有xdl好像也不支持viewfs — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#31 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD20yzZWIntWYEB0eVW5Cskee1dugt7Xks5vOmfegaJpZM4a9TEF>.

SuperbDong · 2019-02-19T06:19:05Z

docker 运行模式单worker没问题如果多worker会报
NotFoundError (see above for traceback): ckpt/model.ckpt-0_temp_763e08a969e04f0eb55c86f367e10977; No such file or directory
每个worker在一台机器上

chengenbao · 2019-02-19T07:49:51Z

可以说一下执行过程吗？

…

On 19 Feb 2019, at 2:19 PM, SuperbDong ***@***.***> wrote: 0_temp_763e08a969e04f0eb55c86f367e10977

SuperbDong · 2019-02-19T08:32:06Z

https://stackoverflow.com/questions/53828895/does-distributed-tensorflow-requires-a-distributed-file-system-to-save-checkpoin
和这个一样的case

SuperbDong · 2019-02-20T02:19:35Z

需要指定额外的参数么，checkpoint的时候会失败，当worker大于1的时候
NotFoundError (see above for traceback): ckpt/model.ckpt-0_temp_763e08a969e04f0eb55c86f367e10977; No such file or directory

chengenbao · 2019-02-20T03:02:07Z

这个问题看起来是由于ps在不同的机器上导致的，不同机器的ps需要有共享的文件系统，如果你是这种情况可以考虑hdfs或者nfs。你看看是不是跟这个issue上的情形类似， tensorflow/tensor2tensor#394

…

On 20 Feb 2019, at 10:19 AM, SuperbDong ***@***.***> wrote: 需要指定额外的参数么，checkpoint的时候会失败，当worker大于1的时候 NotFoundError (see above for traceback): ckpt/model.ckpt-0_temp_763e08a969e04f0eb55c86f367e10977; No such file or directory — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#31 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD20ywVydPvHx8JtRKKtFdoiWZdfKLVKks5vPLC4gaJpZM4a9TEF>.

zys2017 · 2019-04-10T12:29:09Z

pip安装的不支持hdfs
…
On 18 Feb 2019, at 5:31 PM, Gavin Zhang @.***> wrote: 如果需要支持hdfs数据加载，则需要修改项目根目录下的CMakeLists.txt，把option(USE_HDFS "Option for using hdfs" OFF)改为option(USE_HDFS "Option for using hdfs" ON)，详细过程请看编译安装部分如果用pip安装的，需要手动开启这个开关不 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#31 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/AD20y2jvOq8p3Ejg6uUvRdKbjyRic42bks5vOnLUgaJpZM4a9TEF.

我通过pip安装是支持hdfs的

xiongzhiyuan · 2019-08-01T13:28:16Z

pip 安装的支持 hdfs么？我这通过whl文件安装的出现下面的错误
E0801 20:48:20.770622 68588 graph_engine.cc:75] no hdfs file io factory register I0801 20:48:20.770738 68588 graph_service.cc:179] service init finish E0801 20:48:20.770808 68588 graph_service.cc:157] service error

xiongzhiyuan · 2019-08-02T08:43:59Z

有没有支持hdfs的euler whl包

zys2017 · 2019-08-02T08:50:33Z

Python 有支持hdfs的 whl包，或者可以在euler 的dockerfile文件中增加ssh、format等相关内容，如：用户在pod中就可以直接操作hdfs相关内容。在2019年08月2日 16:44，xiongzhiyuan<notifications@github.com> 写道：有没有支持hdfs的euler whl包 — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

xiongzhiyuan · 2019-08-02T08:54:59Z

euler_gl-0.1.2-cp27-cp27mu-manylinux1_x86_64.whl 这个whl支持么，我这边重新编译的代价有点大，网络，你们那边可以提供一下能支持hdfs的whl包么？

AnuradhaSK mentioned this issue Nov 28, 2019

Distributed training #179

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

euler如何支持hdfs #31

euler如何支持hdfs #31

SuperbDong commented Feb 15, 2019

SuperbDong commented Feb 16, 2019 •

edited

zakheav commented Feb 16, 2019

SuperbDong commented Feb 16, 2019

zakheav commented Feb 16, 2019

SuperbDong commented Feb 16, 2019

chengenbao commented Feb 17, 2019 via email

SuperbDong commented Feb 18, 2019

chengenbao commented Feb 18, 2019 via email

zakheav commented Feb 18, 2019

zakheav commented Feb 18, 2019

zakheav commented Feb 18, 2019

chengenbao commented Feb 18, 2019 via email

SuperbDong commented Feb 18, 2019

2shou commented Feb 18, 2019

chengenbao commented Feb 18, 2019 via email

chengenbao commented Feb 18, 2019 via email

SuperbDong commented Feb 19, 2019 •

edited

chengenbao commented Feb 19, 2019 via email

SuperbDong commented Feb 19, 2019

SuperbDong commented Feb 20, 2019

chengenbao commented Feb 20, 2019 via email

zys2017 commented Apr 10, 2019

xiongzhiyuan commented Aug 1, 2019

xiongzhiyuan commented Aug 2, 2019

zys2017 commented Aug 2, 2019 via email •

edited

xiongzhiyuan commented Aug 2, 2019

euler如何支持hdfs #31

euler如何支持hdfs #31

Comments

SuperbDong commented Feb 15, 2019

SuperbDong commented Feb 16, 2019 • edited

zakheav commented Feb 16, 2019

SuperbDong commented Feb 16, 2019

zakheav commented Feb 16, 2019

SuperbDong commented Feb 16, 2019

chengenbao commented Feb 17, 2019 via email

SuperbDong commented Feb 18, 2019

chengenbao commented Feb 18, 2019 via email

zakheav commented Feb 18, 2019

zakheav commented Feb 18, 2019

zakheav commented Feb 18, 2019

chengenbao commented Feb 18, 2019 via email

SuperbDong commented Feb 18, 2019

2shou commented Feb 18, 2019

chengenbao commented Feb 18, 2019 via email

chengenbao commented Feb 18, 2019 via email

SuperbDong commented Feb 19, 2019 • edited

chengenbao commented Feb 19, 2019 via email

SuperbDong commented Feb 19, 2019

SuperbDong commented Feb 20, 2019

chengenbao commented Feb 20, 2019 via email

zys2017 commented Apr 10, 2019

xiongzhiyuan commented Aug 1, 2019

xiongzhiyuan commented Aug 2, 2019

zys2017 commented Aug 2, 2019 via email • edited

xiongzhiyuan commented Aug 2, 2019

SuperbDong commented Feb 16, 2019 •

edited

SuperbDong commented Feb 19, 2019 •

edited

zys2017 commented Aug 2, 2019 via email •

edited