Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Not using boto3 deliberately. This does mean we have low-level code, but... - To actually output JSON, we do have to do a little bit of faffing with bytes anyway, since S3 Select does not output valid JSON: it outputs JSON objects contatanated together with a delimeter. - S3 Select [at least, minios's implementation used for testing], appears to output unicode escape sequences for certain characters that don't need it, i.e. '\u0026' instead of just '&'. Not using boto3 means we can address issues like this in as a performant way as possible [even if we don't do much optimisation now, we are free to later]. - boto3 does not always support all of AWS, specifically with S3. For example boto/botocore#996 has been open for 4 years (to the day!) at this point, so we should not be thwarted or have to workaround any limitation of boto3: I suspect its architecture is not optimized for low-level/streaming access, which is exactly the sort of thing that is suspected to be useful in this project. - Am pro keeping the option to use asyncio open, at least for now in this early stage of the project, and boto3 doesn't appear to support it out of the box. In its current form, to move to asyncio shouldn't be a massive project since the dependencies are a web server and a web client [both of which would likely be aiohttp].
- Loading branch information
Showing
2 changed files
with
356 additions
and
5 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.