Error download large LAR files

[meissadia]

Similar issue:
cfpb/hmda-data-browser#62
cfpb/hmda-data-browser#83

[meissadia]

Downloading

• Chrome fails after 20 minutes, 150MB (unclear error)
• Safari fails after 40 minutes, 1.43GB (memory error)

Alternate Fetch methods

• Anchor method used for Data Browser fails due to our need for authentication
• Streams method fails (premature done signal)

Need to save the stream?

• https://github.com/jimmywarting/StreamSaver.js
• Need to dig more deeply into how this works
• Might not be useful in light of the premature done signal

[meissadia]

Tested again on 12/29

• Anchor method failed with 'File not found'.
• Basic fetch terminated prematurely without error.
• Stream method to prematurely get a done signal.

[meissadia]

12/30

• Doubled nginx config limits

  client_body_buffer_size  32k;
  client_header_buffer_size 2k;
  client_max_body_size 10m;
  large_client_header_buffers 4 16k;
  client_body_timeout 120s;
  client_header_timeout 120s;
  send_timeout 120s;
  

• Used blob format for API response

I'm still seeing the same early termination for large files.

I also see the same behavior via Postman. When doing a "Send and Download", I have not been able to download more than 230MB.

[meissadia]

1/22

Testing via CURL

curl -H "Authorization: Bearer <token>" \
https://<dev>/v2/filing/institutions/B90YWS6AFX2LGWOXJ1LD/filings/2020/submissions/723/edits/csv?format=csv \
--output ~/Downloads/edits_via_curl.test

Result:

curl: (18) transfer closed with outstanding read data remaining

Some suggestions that did not resolve the error:

• Add option --keepalive-time 2 ❌
• Add header Accept-Encoding: gzip, deflate ❌

curl -H "Authorization: Bearer <token>" \
-H 'Accept-encoding: gzip, deflate' \
--keepalive-time 2 \
--output ~/Downloads/edits_via_curl.test \
 https://<dev>/v2/filing/institutions/B90YWS6AFX2LGWOXJ1LD/filings/2020/submissions/723/edits/csv?format=csv 

Other avenues to explore:

• Server sending wrong Content-Length header?