Fix Octet Streaming of Datastore Dump #7839
Conversation
- Fixed view method for datastore dump to actually stream data in Flask response. - Fixed possible issue in sql return.
- Added change log file. - Fixed flake8 syntax reporting. - Fixed typing reporting.
|
Love to see this @JVickery-TBS Datastore dump for large datasets has been broken since 2.9 and really needs this fix. |
- Changed member dump view to not use datastore writer. - Updated typing for datastore dump inner methods.
- Fix more typing issues.
- Even more typing fixes.
- Please let this be the last fix.
|
Okay typing should be good now. And added a fix to |
- Removed `u` strings in new code. - Changed resource id validator to datastore search.
- Reworked new code into `dump_to`.
- Removed unused `Invalid` import.
- Reworked the `dump_to` method to work for the view method and the cli method. - Removed `bom` flag from cli command and handled condition based on format.
- Fixed unbound for `bom`.
- Made `dump_to` always return a generator. - Removed unnecessary params from `dump_to`. - Brought back BOM flag in cli dump command.
|
@JVickery-TBS some related tests are failing: I think circle ci working needs to be fixed on your end, perhaps authorizing GitHub again? |
|
@amercader sadly it was a recent change to CircleCI. Users need to have write permissions to the github repo, and also be added to the project in CircleCI to initialize a CircleCI build/workflow. I had to talk to CircleCI support about it as we recently moved to CircleCI. And of course the week we started using it was the week they made this change. They said it was for security purposes so that anyone cannot just initialize a CircleCI build. But thanks for running the workflow, I will update these tests to work with the new code |
- Removed byte strings in StringIO buffer usage.
|
@amercader Okay the tests should hopefully work now. At least they worked locally for me. Seems like github is just taking a while to update this PR with the new push. But hopefully it updates sometime soon |
- Use `BytesIO` instead of `StringIO`.
- Added `utf-8` encoding for byte writing.
- Added `utf-8` encoding for byte writing.
- Added `utf-8` encoding for byte writing.
- Added `utf-8` encoding for byte writing.
- Reverted some `str` to `bytes` changes.
- Typing, - Indents.
- Reverted some `str` to `bytes` changes.
- More attempts at byte writing for datastore dump.
- More attempts at byte writing for datastore dump.
- Moved output buffers into the datastore writer methods. - Made write record methods return bytes. - Added end file method for returning final bytes of the files.
- Lint fixes. - Typing fixes.
- Removed method param comments.
|
@amercader Okay Ian came to my rescue and this should all be working now. Had to move the StringIO/BytesIO stuff into the writers as the XML one wanted Bytes and the others String. So you will no longer need to supply a file-like object to the writer methods, they will make one for you and yield the bytes in write records |
|
Thanks @JVickery-TBS , and it looks you sorted out your circleci issues! As this is quite a big change, would you mind submitting a PR with the same changes against dev-v2.10 to make backporting easier? Many thanks |
fix(views): datastore ext dump octet streaming;
Fixes
Datastore dump does not actually stream the data. The view method has now be re-written to properly stream the data via Flask Response.
Also includes small condition for possible issue in the return of some sql stuff.
Features:
Please [X] all the boxes above that apply