Tune upload perf, better cleanup logic

[nl0]

Description

1. Limit hashing concurrency to prevent file access errors
2. Optimize upload code to reduce memory pressure (avoid creating and storing a closure for every upload)
3. Fall back to server-side hashing when it can't be done client-side

TODO

• Changelog entry

[codecov]

Codecov Report

Merging #2532 (c7a4447) into master (6889d4b) will decrease coverage by 0.01%.
The diff coverage is 0.00%.

❗ Current head c7a4447 differs from pull request most recent head 4fdf92a. Consider uploading reports for the commit 4fdf92a to get more accurate results

@@            Coverage Diff             @@
##           master    #2532      +/-   ##
==========================================
- Coverage   43.75%   43.74%   -0.02%     
==========================================
  Files         504      504              
  Lines       24257    24266       +9     
  Branches     3275     3276       +1     
==========================================
  Hits        10614    10614              
- Misses      12816    12825       +9     
  Partials      827      827              

Flag	Coverage Δ
api-python	90.26% <ø> (ø)
catalog	14.67% <0.00%> (-0.01%)	⬇️
lambda	93.95% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...app/containers/Bucket/PackageDialog/FilesInput.tsx	0.00% <0.00%> (ø)
.../containers/Bucket/PackageDialog/PackageDialog.tsx	24.26% <0.00%> (+0.07%)	⬆️
...og/app/containers/Bucket/PackageDialog/Uploads.tsx	0.00% <0.00%> (ø)
catalog/app/utils/ResourceCache.js	19.14% <ø> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6889d4b...4fdf92a. Read the comment docs.

[akarve]

I will test tomorrow and circle back

[akarve]

I am currently testing this with a 6.9GB upload on Chrome / Mac OS / 2016 MacBook Pro.

1. OK for separate PR: We should bump the warning size to idk 10GB (I got a warning for being over 1GB)
2. The UI thread stalled for me as I was typing in the name field (after dragging and dropping my files) — about a 2.5 sec pause but then it was OK.
3. My uplink is 21.5 Mbps and I seem to be ~ about saturating it; which is good. I'd say I'll finish 6.9GB in about 35 min total wall time.
4. OK for separate PR: The progress optically appears to stall when the upload goes over 1GB but it's only because the number changes more slowly :) We should either always show more decimal places of precision above 1GB and/or we should show indeterminate progress
5. OK for separate PR; it would be really nice to have some concurrency knobs (if meaningful) in the catalog settings so that different machines/users have some control and we can learn from the field how to optimize performance

[akarve]

😬

[nl0]

The UI thread stalled for me as I was typing in the name field (after dragging and dropping my files) — about a 2.5 sec pause but then it was OK.

it was processing the dropped files (synchronously), so it's kinda expected, tho not very nice, i guess

[nl0]

OK for separate PR; it would be really nice to have some concurrency knobs (if meaningful) in the catalog settings so that different machines/users have some control and we can learn from the field how to optimize performance

@akarve i'm not really sure how to approach this, ping me in slack for a discussion if you feel this is important