Async Support for LLMChainExtractor #3587
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implemented
acompress_documents
and changed syntax forcompress_documents
slightly to make sync/async functions consistent.LLMChainExtractor
as implemented in #2915 for use in theContextualCompressionRetriever
lacked an async method. As compression of retrieved documents is a highly parallelizable task, this was a major performance bottleneck in my tests.This implementation is consistent to the implementation in the somewhat similar
MapReduceDocumentsChain
chain, see https://github.com/hwchase17/langchain/blob/85dae78548ed0c11db06e9154c7eb4236a1ee246/langchain/chains/combine_documents/map_reduce.py#L131 .In my own tests (standalone as well as in a compression pipeline) inputs/outputs are unchanged and the async-speedup is significant.