Deduplicate file downloads

[hannes-ucsc]

[hannes-ucsc]

Instead of downloading files into per-bundle directories, download them into an .hca directory of the current directory. Use the sha256 of the file as the file name. Download only of the file does not already exist. If file is missing, download into a uniquely named temporary file under .hca. Once download is complete, check-sum temporary file, then rename file atomically to final file name. We don't care if a file was modified after download. Hard-link file into per-bundle dir, using the user-friendly filename from the bundle manifest.

The resulting directory layout will be the same as in the current version of the CLI except that files will be hard-links.

Later we'll add --layout to hca download-manifest (defaults to bundle) and implement --layout sample. The primary files will always be in .hca and --layout just affects the names of the hard links really.

[hannes-ucsc]

@mikebaumann proposed that we don't touch the existing manifest download functionality and instead add new experimental download-manifest-v2 CLI command.

[diekhans]

Can this be accomplished by introspection of the columns in the manifest rather than a command line option? Alternatively have a command line options to pick behavior rather than specifies a version? A user having to think `do I have v1 or v2 of the manifest' would very error prone and confusing, especially if the version number isn't part of the manifest. Hannes Schmidt <notifications@github.com> writes:

…

@mikebaumann proposed that we don't touch the existing manifest download functionality and instead add new experimental download-manifest-v2 CLI command. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.*

[hannes-ucsc]

@diekhans there is only one version of the manifest. There are two version of the command. The …-v2 version would replace the main version of the command once we are reasonably sure that users accept it.

[diekhans]

One version of the manifest is even better. So how about naming the option someting to do with the fuctionality rather than just v2? Something meaningful to the user?

…

-----Original Message----- From: Hannes Schmidt <notifications@github.com> To: HumanCellAtlas/dcp-cli <dcp-cli@noreply.github.com> Cc: Mark Diekhans <markd@ucsc.edu>, Mention <mention@noreply.github.com> Sent: Wed, 27 Feb 2019 9:44 pm Subject: Re: [HumanCellAtlas/dcp-cli] Deduplicate file downloads (#234) @diekhans the is only one version of the manifest. There are two version of the command. The `…-v2` version would replace the main version of the command once we are reasonably sure that users accept it.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #234 (comment)

[hannes-ucsc]

We may want to consolidate download and download-manifest:

download-bundle => download --bundle UUID --version VERSION --replica REPLICA
download-manifest => download --layout bundles --manifest PATH --replica REPLICA
download-manifest-v2 => download --layout raw --manifest PATH --replica REPLICA