-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Trouble Adding S3 URLs from NDA manifest #1
Comments
Desired to help figure reason out. Ref: ReproNim/datalad-nda#1
unrelated but you don't need the error at the end of STDERR[INFO] -> Adding URLs
Traceback (most recent call last):
File "/home/faird/shared/code/external/utilities/datalad-nda/scripts/datalad-nda", line 442, in <module>
main()
File "/home/umii/hendr522/SW/miniconda3/envs/datalad/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
return self.main(*args, **kwargs)
File "/home/umii/hendr522/SW/miniconda3/envs/datalad/lib/python3.8/site-packages/click/core.py", line 1062, in main
rv = self.invoke(ctx)
File "/home/umii/hendr522/SW/miniconda3/envs/datalad/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/umii/hendr522/SW/miniconda3/envs/datalad/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/umii/hendr522/SW/miniconda3/envs/datalad/lib/python3.8/site-packages/click/core.py", line 763, in invoke
return __callback(*args, **kwargs)
File "/home/faird/shared/code/external/utilities/datalad-nda/scripts/datalad-nda", line 250, in add2datalad
out = ds.addurls(
File "/home/umii/hendr522/SW/miniconda3/envs/datalad/lib/python3.8/site-packages/datalad/distribution/dataset.py", line 503, in apply_func
return f(**kwargs)
File "/home/umii/hendr522/SW/miniconda3/envs/datalad/lib/python3.8/site-packages/datalad/interface/utils.py", line 486, in eval_func
return return_func(generator_func)(*args, **kwargs)
File "/home/umii/hendr522/SW/miniconda3/envs/datalad/lib/python3.8/site-packages/datalad/interface/utils.py", line 474, in return_func
results = list(results)
File "/home/umii/hendr522/SW/miniconda3/envs/datalad/lib/python3.8/site-packages/datalad/interface/utils.py", line 459, in generator_func
raise IncompleteResultsError(
datalad.support.exceptions.IncompleteResultsError: Command did not complete successfully. 1 failed:
[{'action': 'addurls',
'message': 'First positional argument should be mapping '
'[addurls.py:format:79]',
'path': '/spaces/ngdr/workspaces/hendr522/ABCD/datalad2.0-ABCD-BIDS',
'status': 'error',
'type': 'dataset'}]
filed datalad/datalad#5850 to help make that message a bit more informative. overall, please double check that your .txt is a .tsv, i.e. tab separated (that is what I had) since ATM we just hardcode that assumption: https://github.com/ReproNim/datalad-nda/blob/master/scripts/datalad-nda#L78 |
although probably it is not that -- since it would have then just failed here https://github.com/ReproNim/datalad-nda/blob/master/scripts/datalad-nda#L92 instead of nicely getting those out... I should try to redo a sample run on what I had to see if may be something changed in datalad which rendered it not working. |
Let me know if you need anything from me. I'm using an updated manifest.txt from when you originally built the ABCD-BIDS datalad dataset, so maybe that is part of it? If it is useful, I am happy to send you the manifest file that I am using via BOX or something. I imagine you have an active ABCD DUC, right? |
Sorry about delay, I will try to get try this today, and let you know if we need to sync up on manifests ;-) |
I have pushed fefeb27 which should address that exception you (and I) were getting with 0.14.4 -- we have managed to change something on datalad end which stopped doing automagic treatment of pathlib's Path's as str where desired. But the main problem came later since I seem can't get access to any of those prior S3 urls -- may be they turned off that access approach already? but may be it would somehow work for you? |
FWIW, better upgrade to 0.14.7 datalad, just in case ;) |
Thanks for the push, I ran it with this and datalad version 0.14.6 (because that is the most recent conda had). Here is the information I have gotten from STDERR: Is that looking like you expect? If so, can you give me some advice on the resources a job like this will need to run to completion? Currently I have specified 10 CPUs, 20gb RAM, and 24 hours. |
Did it annex any files? Could you please take a head (eg 100 rows) of the file and run till completion, check that files are annexed and could be retrieved if you clone the result and remove origin remote (so it just doesn't fetch from where you already fetched to) Resources - hard to tell, depends if you use --fast mode (no checksums in annex keys, suboptimal), bandwidth etc. Iirc a full run took like a week for me and that is with iirc 10 parallel across subdatasets jobs and a really good bandwidth. Do your manifests contain md5 checksums? If so, I might better finally add support (recent git annex has needed functionality already) to just trust those, and then avoiding most if not all traffic - we would just populate datasets without actually having data. Will only query NDA / s3 for file sizes afaik |
I'm not totally sure what file that you are wanting me to take a "head" of, but it doesn't look that it was able to at least annex one file within the "sourcedata" folder named "submission-21948.csv" which appears to be a manifest as a CSV file. It does not look like the manifest I used has md5 checksums, the file headers are:
Here is the command that I used for the most recent execution: |
head -n 100 < /spaces/ngdr/workspaces/hendr522/ABCD/data/datalad/abcd316520200818/datastructure_manifest.txt > /spaces/ngdr/workspaces/hendr522/ABCD/data/datalad/abcd316520200818/datastructure_manifest-100.txt
/home/faird/shared/code/external/utilities/datalad-nda/scripts/datalad-nda add2datalad
-i /spaces/ngdr/workspaces/hendr522/ABCD/data/datalad/abcd316520200818/datastructure_manifest-100.txt
-d /spaces/ngdr/workspaces/hendr522/ABCD/data/datalad/ABCD-BIDS-3165-test100 -J 10 --fast --drop-after |
checked mine -- the same... I wonder what was some other file I had tried to work with. oh well -- I guess no "speedy" way for now (unless we see a checksum to be added to manifest by NDA - they might be computing them upon upload to ensure data integrity etc) |
Okay, what I am hearing you say is that it appears to be working, but it is going to take a lot longer than anticipated because it needs to create the md5 checksums. Maybe I'll do a little research into how Tobias Kadelka was able to pre-generate the md5 checksums via: https://github.com/TobiasKadelka/build_hcp and http://handbook.datalad.org/en/inm7/usecases/HCP_dataset.html |
Just thinking out loud, I could ask the NDA help desk if they distribute md5 checksums for releases. The other thing that I could do is that I have the entire ABCD-BIDS collection 3165 as a read-only folder at Minnesota. I could crawl all of the files and pull out the md5 checksums for each file. What do you think? |
nope -- I am waiting on you to say either that 100-lines head worked out nicely and files were annexed (when I tried) I failed to get access to those listed in the manifest I had
we could ask or check their API either they provide them already may be... if they do -- we could then more "insistingly" ask to include them in distributed manifests ;)
It is a "danger zone" since assumes that that folder has it 100% identical to what is in NDA and no evil software/hardware/human bug caused divergence. |
After running what you suggested here is what the datalad dataset looks like: derivatives: derivatives/abcd-hcp-pipeline: derivatives/abcd-hcp-pipeline/sub-XXXXXXXXX: derivatives/abcd-hcp-pipeline/sub-XXXXXXXXX: sourcedata: |
without any data file under any of those directories? if so -- means that it failed to fetch any file, most likely "good old direct S3 access" is no longer possible. |
Right, no data files underneath any of the directories. It appears that the this only file that is underneath any of the given directories is "submission-22640.csv". So what's next if "good old direct S3 access" is no longer possible? By the way, how does your script go about authenticating users? I've been following along with the tutorial at: http://handbook.datalad.org/en/latest/usecases/HCP_dataset.html#data-retrieval-and-interacting-with-the-repository. There it suggests that I explicitly add something to the ".config/providers/nda-s3.cfg" for authentication, but it seems like you are going about a totally different strategy. |
datalad comes with this https://github.com/datalad/datalad/blob/master/datalad/downloaders/configs/nda.cfg (I don't spot any "suggests" you mention present in the handbook) so it used nda-s3 type of credential defined here https://github.com/datalad/datalad/blob/master/datalad/downloaders/credentials.py#L353 which the dance to get the token to access S3... datalad should ask you for user/password, store those, and then mint a new token whenever needed You know -- I have tried now again -- and it seemed to work out as it all should have. May be things are back in "old normal"??? please try again, and if it doesn't work (how? any errors?) -- we would indeed need to work out based on the same manifest. My invocation was
on a sample of top 100 rows in manifest I had, and I had datalad 0.14.7 (FWIW: we do have now updated datalad and git-annex in conda-forge... would be interested to discover if that matter)... before that - you could try just plain direct |
I've tried this, but it seems to just hang. Based on our current discussion I get: And based on our discussion that I started with Michael Hanke, on the HCP-D data: In both cases it does not proceed any further. |
eh, I bet it is due to datalad/datalad#5099 ... I will look into at least making the situation more obvious. edit: datalad/datalad#5884 hopefully would mitigate this "halting" issue by first of all being more informative Original intention is to lock for querying credentials, and "theoretically" it should not hang unless there is another datalad process on the same box asking for credentials Please rerun with
|
Deleting the locked file definitely did the trick. The only problem I am now having is that despite entering the correct NDA username and password Here are the messages sent to the debugger:
|
quick workaround (BTW please share output of |
DataLad 0.14.7 WTF (configuration, credentials, datalad, dependencies, environment, extensions, git-annex, location, metadata_extractors, metadata_indexers, python, system)WTFconfiguration <SENSITIVE, report disabled by configuration>credentials
datalad
dependencies
environment
extensionsgit-annex
location
metadata_extractors
metadata_indexerspython
system
|
removed verbatim decoration in WTF -- looks how it looks now ;) so -- did you succeed with credentials workaround? |
Hello,
I am attempting to use the
datalad-nda
CLI in order to produce a datalad dataset based on the 09 2020 3165 ABCD release found on NDA. I have the manifest as an uncompressed text file, so I came up with the following command:/home/faird/shared/code/external/utilities/datalad-nda/scripts/datalad-nda add2datalad \ -i <(cat /spaces/ngdr/workspaces/hendr522/ABCD/datalad-ABCD-BIDS/abcd316520200818/datastructure_manifest.txt) \ -d /spaces/ngdr/workspaces/hendr522/ABCD/datalad2.0-ABCD-BIDS -J 10 --fast --drop-after
. I've attached the messages sent to STDOUT and STDERR in separate filesdatalad2.0-nda_ABCD-BIDS_5371074_STDOUT.txt
datalad2.0-nda_ABCD-BIDS_5371074_STDERR.txt
I am using the most recent version of the
datalad-nda
CLI (I pulled the repo maybe a week ago) and am running datalad version 0.14.4The text was updated successfully, but these errors were encountered: