-
Notifications
You must be signed in to change notification settings - Fork 270
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Don't re-download taxonomy files if already there #5
Comments
I think the right approach there is to have separate dlflag files for each file, or actually store a checkpoint value somewhere so that downloads start after the last complete step. I don't know the Aspera URL (and haven't worked with Aspera much myself, as might be evident). |
I love the rsync part though - nice touch. I realised after filing that you use touch-files for the other stuff. I guess it's just a matter of extending it to those 4 files. I suspect most people in EU / USA get fast FTP to NCBI. |
OK, I went and just did the touch file thing with the others in 98ac708 - there's a few other wgets in other scripts that are going to need to be wrapped in the same fashion, but I'll handle that in another issue. |
I was doing
kraken-build --db XXX --download-taxonomy
and the 4th FTP stalled, so i restarted it, and it started re-downloading everything from scratch (it takes 10 hours to get them from NCBI over FTP to my uni, they throttle FTP)https://github.com/DerrickWood/kraken2/blob/master/scripts/download_taxonomy.sh#L24-L27
wget
option to avoid re-download?Or only download if
if [ ! -r $FILE ]; then wget ... ; fi
?ascp
if it is installed?The text was updated successfully, but these errors were encountered: