Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

利用docker部署分布式EasyML集群后,hadoop集群中的文件无法下载 #104

Closed
feiruyun opened this issue May 7, 2019 · 2 comments

Comments

@feiruyun
Copy link

feiruyun commented May 7, 2019

按照https://github.com/ICT-BDA/EasyML/wiki/%E5%88%A9%E7%94%A8docker%E9%83%A8%E7%BD%B2%E5%88%86%E5%B8%83%E5%BC%8FEasyML%E9%9B%86%E7%BE%A4 的步骤,在三台电脑上部署分布式后,log中也没出现错误,hadoop集群50070端口访问正常,两个datanode也正常,但是在Browse the file system里面对文件进行下载时,提示:
无法访问此网站
找不到 hadoop-slave1.weave.local 的服务器 IP 地址。
image
image
跳转网址:http://hadoop-slave1.weave.local:50075/webhdfs/v1/user/root/input/restart.sh?op=OPEN&namenoderpcaddress=hadoop-master:9000&offset=0

本地host文件已添加(如下):
10.10.21.85 hadoop-master
10.10.21.85 mysql
10.10.21.80 hadoop-slave1
10.10.21.86 hadoop-slave2

下载的文件为运行sh run-wordcount.sh后生成的file1.txt

请问,这是什么原因呢?感谢解答!

@feiruyun
Copy link
Author

feiruyun commented May 8, 2019

50075端口未映射导致的该问题,教程中slave容器创建命令:
docker run -itd -e WEAVE_CIDR=10.32.0.5/12 --name=hadoop-slave2 --hostname hadoop-slave2.weave.local -p 8043:8042 -p 50011:50011 -p 50021:50021 --add-host mysql.weave.local:10.32.0.2 --add-host hadoop-master.weave.local:10.32.0.3 --add-host hadoop-slave1.weave.local:10.32.0.4 --add-host hadoop-slave2.weave.local:10.32.0.5 \cluster /bin/bash
加一个50075端口的映射:-p 50075:50075
这样就在50070界面进行文件下载就正常了,另外本地host文件最好修改如下:
10.10.21.85 hadoop-master hadoop-master.weave.local
10.10.21.85 mysql mysql.weave.local
10.10.21.80 hadoop-slave1 hadoop-slave1.weave.local
10.10.21.86 hadoop-slave2 hadoop-slave2.weave.local

@feiruyun
Copy link
Author

feiruyun commented May 8, 2019

这个问题解决了,但是又有了新的问题:
#105

@feiruyun feiruyun closed this as completed May 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant