-
Notifications
You must be signed in to change notification settings - Fork 18
New endpoint that returns count, mb size, and nodes for each file type #934
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov Report
@@ Coverage Diff @@
## master #934 +/- ##
==========================================
+ Coverage 90.14% 90.16% +0.02%
==========================================
Files 48 48
Lines 6462 6516 +54
==========================================
+ Hits 5825 5875 +50
- Misses 637 641 +4
Continue to review full report at Codecov.
|
Looks good, but we don't actually need the node list returned with the summary results. Are you providing the node list to make the eventual download request more efficient? |
|
||
try: | ||
result = config.db.command('aggregate', cont_name, pipeline=pipeline) | ||
except Exception as e: # pylint: disable=broad-except |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The string version of the failure would get returned to the API user. In situations like this it's better to internally log the error (log.warning(e)
) and then self.abort(500, 'Failure to load summary')
or something like that so the user doesn't get sent a mongo error.
api/download.py
Outdated
containers = ['acquisitions'] | ||
elif level == 'analyses': | ||
cont_query['analyses'] = {'_id': req['_id']} | ||
containers = containers[-1:] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should make sure we have tests if we're going to support file summaries for analysis downloads.
api/download.py
Outdated
acquisition_ids = [a['_id'] for a in acquisitions] | ||
parent_ids = [req['_id']] + acquisition_ids | ||
|
||
# # Grab analyses and their ids |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We won't want want analyses included in project/session/acquisition download summary (because they aren't included in the actual download), so it's okay to remove this.
That was the idea, it seemed like an easy place to generate it because we are already aggregating through the files anyways. |
Unfortunately that list might be 1000s of acquisitions long for large projects, and the user might choose multiple filters, meaning a frontend join of multiple lists of acquisitions - which would lose the gain in efficiency. |
2dbaecc
to
25ae9a7
Compare
1ab411c
to
d6116b7
Compare
Thanks for making the requested changes, LGTM! |
Notes
POST /download/summary
Review Checklist