如果Github的项目太大，直接git clone可能下载到半路就中断了，导致反反复复下载不了。 这个时候应该咋办呢？ 本人写了一个支持Github特定文件夹内容下载的项目：GitDown，github地址为：https://github.com/jmhIcoding/GitDown
目前已经公开的类似开源项目，经常会出现下载文件丢失的问题，或者代码年久失效。 我这个项目会自动记录下载失败的文件，对于未成功下载的文件会反复尝试下载。 2021年10月1日运行是完全正常的。
pip3 install gitdowngitdown -h
usage: gitdown [-h] --repo REPO --directory DIRECTORY --dst_root DST_ROOT
[--thread_num THREAD_NUM]
Download github resposity directory.
optional arguments:
-h, --help show this help message and exit
--repo REPO Something like
https://github.com/jmhIcoding/social_webpage
--directory DIRECTORY
Something like pcaps/. Note that do not start with "/"
and end with "/", use "" (empty) for root directory.
--dst_root DST_ROOT Indicate where do you want to store the downloaded
files and sub-dirs from github.
--thread_num THREAD_NUM
The thread number for download!
一共有四个参数。
- repo: 这个是给出仓库的url路径， 注意：仓库名的格式类似于https://github.com/jmhIcoding/social_webpage，里面有用户名和仓库名。
- directory：给出要从这个仓库里面待下载的文件夹，注意：目录名不能以“/” 开始和结束。 项目支持递归下载给定文件夹下的所有内容。
- dst_root: 给出要将下载好的内容放置在何处
- thread_num: 多线程下载的线程数，默认为 5
首次运行需要提供自己的github access token， 因为项目运行需要能够访问你自己的github项目，因此需要此token，获取方式见：https://blog.csdn.net/jmh1996/article/details/109327334
Please input your github access token now, first time:1234如果输入的token无效会报错， 因此请按照教程自己申请github token。
Please input your github access token now, first time:1234
Traceback (most recent call last):
File "E:\GitDown\src\config.py", line 12, in <module>
git.get_repo('helloworld9992939329392939294333435452dhfhdfjdfdgfgdfgfd___dfdf')
File "C:\Program Files\Python36\lib\site-packages\github\MainClass.py", line 334, in get_repo
"GET", "%s%s" % (url_base, full_name_or_id)
File "C:\Program Files\Python36\lib\site-packages\github\Requester.py", line 319, in requestJsonAndCheck
verb, url, parameters, headers, input, self.__customConnection(url)
File "C:\Program Files\Python36\lib\site-packages\github\Requester.py", line 342, in __check
raise self.__createException(status, responseHeaders, output)
github.GithubException.BadCredentialsException: 401 {"message": "Bad credentials", "documentation_url": "https://docs.github.com/rest"}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 3, in <module>
from src.parser_git_dir import *
File "E:\GitDown\src\__init__.py", line 2, in <module>
import config
File "E:\GitDown\src\config.py", line 15, in <module>
raise ValueError('The token you input is invalid!!!')
ValueError: The token you input is invalid!!!命令：
gitdown --repo=https://github.com/jmhIcoding/social_webpage --directory=pcaps/weibo运行过程：
Download pcaps/weibo/shuqi/2021101/1633039566.pcap now!
85%|▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ | 3133/3685 [26:02<09:38, 1.05s/it]Download pcaps/weibo/shuqi/2021101/1633039591.pcap now!
85%|▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ | 3134/3685 [26:03<09:28, 1.03s/it]Download pcaps/weibo/shuqi/2021101/1633039615.pcap now!
85%|▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ | 3135/3685 [26:03<07:17, 1.26it/s]Download pcaps/weibo/shuqi/2021101/1633039639.pcap now!
85%|▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ | 3136/3685 [26:05<11:16, 1.23s/it]Download pcaps/weibo/shuqi/2021101/1633039663.pcap now!
85%|▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ | 3137/3685 [26:06<09:52, 1.08s/it]Download pcaps/weibo/shuqi/2021101/1633039687.pcap now!
85%|▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ | 3138/3685 [26:07<08:20, 1.09it/s]Download pcaps/weibo/shuqi/2021101/1633039711.pcap now!
85%|▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ | 3139/3685 [26:08<08:58, 1.01it/s]Download pcaps/weibo/shuqi/2021101/1633039735.pcap now!
85%|▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ | 3140/3685 [26:09<08:13, 1.11it/s]Download pcaps/weibo/shuqi/2021101/1633039759.pcap now!
85%|▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ | 3141/3685 [26:09<07:23, 1.23it/s]Download pcaps/weibo/shuqi/2021101/1633039783.pcap now!
85%|▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ | 3142/3685 [26:10<07:26, 1.22it/s]Download pcaps/weibo/shuqi/2021101/1633039807.pcap now!
85%|▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ | 3143/3685 [26:10<06:24, 1.41it/s]Download pcaps/weibo/shuqi/2021101/1633039831.pcap now!
85%|▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ | 3144/3685 [26:12<07:22, 1.22it/s]Download pcaps/weibo/shuqi/2021101/1633039856.pcap now!
85%|▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ | 3145/3685 [26:12<06:55, 1.30it/s]Download pcaps/weibo/shuqi/2021101/1633039881.pcap now!
85%|▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ | 3146/3685 [26:13<05:36, 1.60it/s]Download pcaps/weibo/shuqi/2021101/1633039906.pcap now!
下载的结果：
