Feature/redbox 411 documents refactor

[jamesrichards4]

Context

To start using Documents rather than custom chunks. The intention is to replace all use of chunks in the 'AI' code in a second PR following this one

Changes proposed in this pull request

Update the retriever and storagehandler to be able to use chunk and document style stored chunks. This means documents ingested either way are fine.

Worker rewritten to be LCEL based. This should allow moving to Azure embeddings more easily and is in line with the rest of the codebase

Guidance to review

The storage handler currently handles File objects and can track status. To drop the storage handler chunk access we need to find a way to update the API about status. Moving the storage handler to being a file handler only and having the worker update the status makes this quite clean but other ideas welcome on how we make the most of these changes

There are several places logic is duplicated, this has been left as it will be replaced by dropping the storagehandler version of this logic once we can drop chunk storage (and use documents only). This will come when they expire in all environments.

This will be followed by several PRs:

• One to complete the use of documents and make the retriever use documents from Elastic and pass them through to SourceDocuments to the frontend, this will remove all references to Chunks. The storagehandler logic will also be updated here.
• Move to Azure Embeddings in ingest and API to remove the need for torch, sentencetransformers etc.
• Dropping duplicate logic for fields on stored chunks and use just the document structure in the 'AI' side code. These can be mapped to SourceDocuments and other API objects in defined runnables

Relevant links

Things to check

• I have added any new ENV vars in all deployed environments
• I have tested any code added or changed
• I have run integration tests

[gecBurton]

To drop the storage handler chunk access we need to find a way to update the API about status.

It might help to know that we do not need the same level of granularity that we currently have. Users have told us that they arent interested in the uploaded/chunking/embedding/complete status but only want to know if the file is ready/not-ready cc @KevinEtchells

[gecBurton]

nice. this "inclusive" approach looks like the best way to start this major refactor.

[gecBurton]

I have understood the approach as:

1. all new text-chunks will be encoded as Documents
2. on reading text-chunks they will be cast to Chunks regardless of how they we encoded

In which case it all looks sensible to me.

I presume the next stage is to change step (2) to return Documents? Are we thinking that:

1. we should maintain backwards compatibility for text-chunks encoded as Chunks?
2. we should write one off migration script
3. we should accept a breaking change for July

[jamesrichards4]

I'll handle the conflicts after #630 goes in. We should merge that first