Remove S3 dependency from chunking process

[wpfl-dbt]

Context

In order to regularly test the AI in the system, we need:

• To import and use the chunking functions at the highest level we can
• Avoid mocking/simulating unnecessary dependencies

For these reasons this PR removes S3 from the chunking process, instead passing in a bytestream. In production this just means downloading from S3 higher up the process. For testing, this means we can use the chunking code directly without needing MinIO/S3 in the picture.

Changes proposed in this pull request

• Refactors chunking so S3 passes the opened file into it
• Removes dead redbox.parsing code and tests
• Updates quickupload.ipynb so it better reflects code in the worker

Guidance to review

• Check you're happy with the refactor
• Confirm all unit tests pass

Things to check

• I have added any new ENV vars in all deployed environments
• I have tested any code added or changed
• I have run integration tests

[jamesrichards4]

Love the idea, just one comment on whether we can avoid downloading the file. We should think about cleaning them up if we've pulled them to the worker

[jamesrichards4]

can we pass the 'Body' field of get_object to the documentloader to avoid this download? It's also a byte stream so that should work?

[wpfl-dbt]

I think I'm maybe using the incorrect terminology somewhere -- it lacks a .seek() method and throws an error. For unstructured I guess it specifically needs to be an io.BytesIO stream, not just any stream of bytes?

[wpfl-dbt]

Yeah I've read a few bits and bobs. There's ways to make what comes back from boto3 seekable but they look like they'd introduce fragility for not much gain, imo.

• This uses a private attribute
• This uses a bunch of fragile-looking code/decode

[jamesrichards4]

ah ok yeah in which case let's pull it. We should think about cleaning these files up at some point but maybe don't need to worry in this release. Restaring/scaling the workers will clear out all files anyway so should be ok for a bit