Skip to content
This repository has been archived by the owner on Feb 27, 2023. It is now read-only.

dfget 本地下载报错 #18

Closed
meglory opened this issue Dec 15, 2017 · 15 comments
Closed

dfget 本地下载报错 #18

meglory opened this issue Dec 15, 2017 · 15 comments

Comments

@meglory
Copy link

meglory commented Dec 15, 2017

RT，我在本地启动两个dfget进程，同时下载一个在线文件，报了如下错误，请问是什么原因呢

本地通过docker，启了supernode进程

docker run -d -p 8001:8001 -p 8002:8002 f28c73445883

我的dragonfly.conf配置如下
[node]
address=127.0.0.1,127.0.0.1

dfclient 报错日志：

[2017-12-15 18:56:47,844] WARNING sign:70462-1513335389.992 lineno:167 : has not available pieceTask,maybe resource lack
[2017-12-15 18:56:47,848] WARNING sign:70462-1513335389.992 lineno:167 : has not available pieceTask,maybe resource lack
[2017-12-15 18:56:47,851] INFO sign:70462-1513335389.992 lineno:110 : pull piece task result:{u'msg': u'piece resource lack', u'code': 602} and sleep 1.432 ...
[2017-12-15 18:56:47,938] ERROR sign:70469-1513335392.125 lineno:332 : piece range:0-5242879 error,realMd5:70461da8b94c6ca5d2fda3260c5a8c3b,expectedMd5:436a148497d25f524d222019b174f54f,dstIp:127.0.0.1,total:162
[2017-12-15 18:56:47,943] ERROR sign:70469-1513335392.125 lineno:332 : piece range:10485760-15728639 error,realMd5:70461da8b94c6ca5d2fda3260c5a8c3b,expectedMd5:7d05eb939cd37dc6f612d09a9b6319cc,dstIp:127.0.0.1,total:162
[2017-12-15 18:56:47,944] ERROR sign:70469-1513335392.125 lineno:332 : piece range:5242880-10485759 error,realMd5:70461da8b94c6ca5d2fda3260c5a8c3b,expectedMd5:d99f987ccca3083eaac61338d800da60,dstIp:127.0.0.1,total:162
[2017-12-15 18:56:47,947] WARNING sign:70469-1513335392.125 lineno:167 : has not available pieceTask,maybe resource lack
[2017-12-15 18:56:47,958] WARNING sign:70469-1513335392.125 lineno:167 : has not available pieceTask,maybe resource lack
[2017-12-15 18:56:47,965] ERROR sign:70469-1513335392.125 lineno:332 : piece range:10485760-15728639 error,realMd5:70461da8b94c6ca5d2fda3260c5a8c3b,expectedMd5:7d05eb939cd37dc6f612d09a9b6319cc,dstIp:127.0.0.1,total:162
[2017-12-15 18:56:47,970] INFO sign:70469-1513335392.125 lineno:110 : pull piece task result:{u'msg': u'piece resource lack', u'code': 602} and sleep 0.665 ...

求大神帮忙看下是什么原因，谢谢了^_^
@chenchaobing
Copy link
Contributor

hi，只是下载这个文件会出问题吗？
另外，这个问题，可能是超级节点中的状态信息不对导致的，先试着重启一次超级节点看看是否可以解决。

@meglory
Copy link
Author

meglory commented Dec 15, 2017

嗯，我重启试试

@meglory
Copy link
Author

meglory commented Dec 15, 2017

您好，我重启了下supernode，然后换了一个在线文件下载，还是报这个错

@chenchaobing
Copy link
Contributor

[node]
address=127.0.0.1,127.0.0.1 这个address 配置成127.0.0.1 ， 另外能否发一下dfget执行的命令

@meglory
Copy link
Author

meglory commented Dec 15, 2017

按你说的改了，还是报错呢，命名：./dfget --url "https://download.jetbrains.8686c.com/go/goland-2017.3.dmg" --output /Users/mgh/Desktop/df1/df1.dmg

@chenchaobing
Copy link
Contributor

我刚才本地试验了一下，我这边执行你发的那个命令，可以成功下载。

你能否执行 docker exec -it ${CONTAINER ID} /bin/bash ， 进入docker 容器。
看一下 /home/admin/supernode/logs/app.log 这里面的日志。

另外，能否提供更详细一些的客户端的日志 ~/.small-dragonfly/logs/dfclient.log

@meglory
Copy link
Author

meglory commented Dec 15, 2017

您好，这个是容器中的/usr/local/tomcat/logs/app.log内容
2017-12-15 13:16:04.871 [localhost-startStop-1] INFO com.alibaba.dragonfly.supernode.common.util.MonitorService - available processors count is 2
2017-12-15 13:16:05.058 [Thread-2] ERROR com.alibaba.dragonfly.supernode.common.util.MonitorService - process fields:null error
java.io.IOException: Cannot run program "tsar": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) ~[na:1.8.0_151]
at java.lang.Runtime.exec(Runtime.java:620) ~[na:1.8.0_151]
at java.lang.Runtime.exec(Runtime.java:450) ~[na:1.8.0_151]
at java.lang.Runtime.exec(Runtime.java:347) ~[na:1.8.0_151]
at com.alibaba.dragonfly.supernode.common.util.MonitorService$1.run(MonitorService.java:56) ~[MonitorService$1.class:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method) ~[na:1.8.0_151]
at java.lang.UNIXProcess.(UNIXProcess.java:247) ~[na:1.8.0_151]
at java.lang.ProcessImpl.start(ProcessImpl.java:134) ~[na:1.8.0_151]
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ~[na:1.8.0_151]
... 5 common frames omitted
2017-12-15 13:16:06.693 [localhost-startStop-1] INFO com.alibaba.dragonfly.supernode.common.util.MonitorService - available processors count is 2
2017-12-15 13:16:06.740 [Thread-3] ERROR com.alibaba.dragonfly.supernode.common.util.MonitorService - process fields:null error
java.io.IOException: Cannot run program "tsar": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) ~[na:1.8.0_151]
at java.lang.Runtime.exec(Runtime.java:620) ~[na:1.8.0_151]
at java.lang.Runtime.exec(Runtime.java:450) ~[na:1.8.0_151]
at java.lang.Runtime.exec(Runtime.java:347) ~[na:1.8.0_151]
at com.alibaba.dragonfly.supernode.common.util.MonitorService$1.run(MonitorService.java:56) ~[MonitorService$1.class:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method) ~[na:1.8.0_151]
at java.lang.UNIXProcess.(UNIXProcess.java:247) ~[na:1.8.0_151]
at java.lang.ProcessImpl.start(ProcessImpl.java:134) ~[na:1.8.0_151]
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ~[na:1.8.0_151]
... 5 common frames omitted
2017-12-15 13:16:12.916 [http-bio-8080-exec-1] INFO com.alibaba.dragonfly.supernode.repository.TaskRepository - get file length:203093454 from http client about taskId:bf54d079e8d19bde5808f939215a02a140f7812803171556c5935b5c796bfdf7
2017-12-15 13:16:12.921 [http-bio-8080-exec-2] INFO com.alibaba.dragonfly.supernode.service.impl.CdnManagerImpl - do trigger cdn start for taskId:bf54d079e8d19bde5808f939215a02a140f7812803171556c5935b5c796bfdf7,httpLen:203093454
2017-12-15 13:16:13.014 [Thread-5] INFO com.alibaba.dragonfly.supernode.common.util.NetConfigNotification - current rate limiter count is 1
2017-12-15 13:16:13.044 [Thread-5] INFO com.alibaba.dragonfly.supernode.service.impl.CdnManagerImpl - do trigger cdn success for taskId:bf54d079e8d19bde5808f939215a02a140f7812803171556c5935b5c796bfdf7
2017-12-15 13:16:13.186 [pool-1-thread-1] INFO com.alibaba.dragonfly.supernode.service.cdn.Downloader - taskId:bf54d079e8d19bde5808f939215a02a140f7812803171556c5935b5c796bfdf7 fileUrl:https://download.jetbrains.8686c.com/go/goland-2017.3.dmg on downloader
2017-12-15 13:16:18.298 [pool-3-thread-1] INFO com.alibaba.dragonfly.supernode.common.util.NetConfigNotification - current rate limiter count is 2
2017-12-15 13:18:53.528 [pool-3-thread-2] WARN com.alibaba.dragonfly.supernode.service.cdn.SuperWriter - taskId:bf54d079e8d19bde5808f939215a02a140f7812803171556c5935b5c796bfdf7 get piece timeout
2017-12-15 13:19:02.832 [pool-3-thread-3] WARN com.alibaba.dragonfly.supernode.service.cdn.SuperWriter - taskId:bf54d079e8d19bde5808f939215a02a140f7812803171556c5935b5c796bfdf7 get piece timeout
2017-12-15 13:19:07.405 [pool-3-thread-4] WARN com.alibaba.dragonfly.supernode.service.cdn.SuperWriter - taskId:bf54d079e8d19bde5808f939215a02a140f7812803171556c5935b5c796bfdf7 get piece timeout
2017-12-15 13:19:16.308 [pool-3-thread-1] WARN com.alibaba.dragonfly.supernode.service.cdn.SuperWriter - taskId:bf54d079e8d19bde5808f939215a02a140f7812803171556c5935b5c796bfdf7 get piece timeout
2017-12-15 13:19:16.310 [Thread-6] INFO com.alibaba.dragonfly.supernode.service.impl.CdnReporterImpl - taskId:bf54d079e8d19bde5808f939215a02a140f7812803171556c5935b5c796bfdf7 fileLength:null status:FAIL from:writer
2017-12-15 13:19:17.241 [pool-1-thread-1] ERROR com.alibaba.dragonfly.supernode.service.cdn.Downloader - downloader read error for taskId:bf54d079e8d19bde5808f939215a02a140f7812803171556c5935b5c796bfdf7,fileUrl:https://download.jetbrains.8686c.com/go/goland-2017.3.dmg
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[na:1.8.0_151]
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[na:1.8.0_151]
at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:350) ~[na:1.8.0_151]
at com.alibaba.dragonfly.supernode.service.cdn.Downloader.call(Downloader.java:119) [Downloader.class:na]
at com.alibaba.dragonfly.supernode.service.cdn.Downloader.call(Downloader.java:46) [Downloader.class:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_151]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_151]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
2017-12-15 13:19:17.242 [pool-1-thread-1] INFO com.alibaba.dragonfly.supernode.service.impl.CdnReporterImpl - taskId:bf54d079e8d19bde5808f939215a02a140f7812803171556c5935b5c796bfdf7 fileLength:null status:FAIL from:downloader

@meglory
Copy link
Author

meglory commented Dec 15, 2017

这个是我本地的，dfclient log

[2017-12-15 21:16:12,887] INFO sign:91559-1513343772.884 lineno:51 : cmd params:Namespace(callsystem=None, console=False, dfdaemon=False, filter=None, header=None, identifier=None, locallimit=None, md5=None, node=None, notbs=False, output='/Users/mgh/Desktop/df2/df2.dmg', pattern=None, showbar=False, timeout=None, totallimit=None, url='https://download.jetbrains.8686c.com/go/goland-2017.3.dmg', version=False),python version:sys.version_info(major=2, minor=7, micro=13, releaselevel='final', serial=0)
[2017-12-15 21:16:12,887] INFO sign:91559-1513343772.884 lineno:107 : target file path:/Users/mgh/Desktop/df2/df2.dmg
[2017-12-15 21:16:12,890] INFO sign:91559-1513343772.884 lineno:133 : sysLang:UTF-8
[2017-12-15 21:16:12,890] INFO sign:91559-1513343772.884 lineno:137 : taskFileName:df2.dmg-91559-1513343772.884 and cid:127.0.0.1-91559-1513343772.884
[2017-12-15 21:16:12,890] INFO sign:91559-1513343772.884 lineno:140 : task url:https://download.jetbrains.8686c.com/go/goland-2017.3.dmg
[2017-12-15 21:16:12,902] INFO sign:91559-1513343772.884 lineno:60 : local http result:df2.dmg-91559-1513343772.884@0.0.1 for path:/check/ and cost:0.010
[2017-12-15 21:16:12,902] INFO sign:91559-1513343772.884 lineno:64 : do register to 127.0.0.1,remainder:['127.0.0.1']
[2017-12-15 21:16:13,363] INFO sign:91550-1513343771.178 lineno:93 : do register result:{u'code': 200, u'data': {u'pieceSize': 4194304, u'fileLength': 203093454, u'taskId': u'bf54d079e8d19bde5808f939215a02a140f7812803171556c5935b5c796bfdf7'}} and cost 1.066
[2017-12-15 21:16:13,364] INFO sign:91559-1513343772.884 lineno:93 : do register result:{u'code': 200, u'data': {u'pieceSize': 4194304, u'fileLength': 203093454, u'taskId': u'bf54d079e8d19bde5808f939215a02a140f7812803171556c5935b5c796bfdf7'}} and cost 0.461
[2017-12-15 21:16:13,385] INFO sign:91559-1513343772.884 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 0.758 ...
[2017-12-15 21:16:13,394] INFO sign:91550-1513343771.178 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 0.784 ...
[2017-12-15 21:16:14,155] INFO sign:91559-1513343772.884 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 1.626 ...
[2017-12-15 21:16:14,184] INFO sign:91550-1513343771.178 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 0.880 ...
[2017-12-15 21:16:15,074] INFO sign:91550-1513343771.178 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 1.735 ...
[2017-12-15 21:16:15,789] INFO sign:91559-1513343772.884 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 1.246 ...
[2017-12-15 21:16:16,818] INFO sign:91550-1513343771.178 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 1.300 ...
[2017-12-15 21:16:17,044] INFO sign:91559-1513343772.884 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 1.829 ...
[2017-12-15 21:16:18,128] INFO sign:91550-1513343771.178 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 1.179 ...
[2017-12-15 21:16:18,890] INFO sign:91559-1513343772.884 lineno:60 : local http result:20971520 for path:/rate/ and cost:0.002
[2017-12-15 21:16:18,907] ERROR sign:91559-1513343772.884 lineno:332 : piece range:0-4194303 error,realMd5:70461da8b94c6ca5d2fda3260c5a8c3b,expectedMd5:f41c57536b60d72c51190af835420f42,dstIp:127.0.0.1,total:162
[2017-12-15 21:16:18,929] ERROR sign:91559-1513343772.884 lineno:332 : piece range:0-4194303 error,realMd5:70461da8b94c6ca5d2fda3260c5a8c3b,expectedMd5:f41c57536b60d72c51190af835420f42,dstIp:127.0.0.1,total:162
[2017-12-15 21:16:18,950] ERROR sign:91559-1513343772.884 lineno:332 : piece range:0-4194303 error,realMd5:70461da8b94c6ca5d2fda3260c5a8c3b,expectedMd5:f41c57536b60d72c51190af835420f42,dstIp:127.0.0.1,total:162
[2017-12-15 21:16:18,969] ERROR sign:91559-1513343772.884 lineno:332 : piece range:0-4194303 error,realMd5:70461da8b94c6ca5d2fda3260c5a8c3b,expectedMd5:f41c57536b60d72c51190af835420f42,dstIp:127.0.0.1,total:162

@chenchaobing
Copy link
Contributor

看日志，是超级节点下载失败了导致的

@chenchaobing
Copy link
Contributor

你试着下载国内的资源呢，比如下载个小图片试试，我怀疑跟你网络有关

@meglory
Copy link
Author

meglory commented Dec 16, 2017

hi，您好，我试了在国内网站下载一个8M的文件，还是会报错呢，方便加个微信聊一下嘛^_^

@meglory
Copy link
Author

meglory commented Dec 22, 2017

@chenchaobing hi, 请问这个问题要怎么解决呢？

@meglory
Copy link
Author

meglory commented Jan 7, 2018

已经确认是由于本地有修改过配置，然后打镜像导致的。

@meglory meglory closed this as completed Jan 7, 2018
@lowzj lowzj added the invalid label Mar 21, 2018
@BlueBlue-Lee
Copy link

我也遇到这个问题，请问楼主是怎么解决的

@lggeor
Copy link

lggeor commented Jan 7, 2019

已经确认是由于本地有修改过配置，然后打镜像导致的。

遇到同样问题，是镜像本身的问题吗，我这里测试也发现是pull某些特定镜像时会发生。但是哪种类型的镜像会发生这个问题呢？

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@lowzj @lggeor @meglory @BlueBlue-Lee @chenchaobing