OpenNeuro CLI - not skipping uploaded files

[franklin-feingold]

we are seeing that files uploaded on OpenNeuro are not being skipped in the upload process - this appears to maybe a regression. I can replicate this along with other users

[nellh]

The CLI is missing a parameter required to return the full tree of files since the file tree optimizations were applied, this should be simple to re-enable. It does work for top level files (since those are always returned).

[franklin-feingold]

I heard back from a user using Node.js (v12.16.1), npm (v6.14.2), and the cli (v3.13.0) and found that only the top-level files (mentioned previously) are being skipped. Files that are present on the web UI are being retransferred (the user is running against #1513 too)

[snastase]

I heard back from a user using Node.js (v12.16.1), npm (v6.14.2), and the cli (v3.13.0) and found that only the top-level files (mentioned previously) are being skipped. Files that are present on the web UI are being retransferred (the user is running against #1513 too)

I tried rolling back to openneuro-cli v3.12.9, but it seems to reproduce the same behavior—skips the top-level text files then starts wholesale re-uploading all stimuli and all subjects (most of which are unchanged and already present online).

[snastase]

Some good(?) news: When running this with v3.12.9—although it did not properly recognize existing files—it actually completed the upload of all 345 subjects without any ServerParseError (#1513). However, I'm not yet seeing any new subjects/files reflected on the web UI (after a couple hours).

[nellh]

This was a dataset specific issue with ds002345 originally caused by poor storage performance. We recently deployed a new storage backend that should avoid that going forward. We went back and fixed the metadata issues this caused in this dataset and you should be able to start new uploads now that correctly skip existing files.