feat(upload): async improved #2544
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dosubot
bot
added
the
size:L
|
Someone is attempting to deploy a commit to the Quivr-app Team on Vercel. A member of the Team first needs to authorize it. |
dosubot
bot
added
the
area: backend
|
Thanks a lot ! I'll review it and let you know if there is anything |
|
Thanks a lot! It works great except for when you upload URLs ;) I'll fix that |
Signed-off-by: aminediro <aminediro@github.com>
The list_files_array in the QuivrRAG class is updated to include file URLs in addition to file names. This change allows for better handling and display of files in the application.
|
StanGirard
reviewed
Jun 4, 2024
|
|
||
| def __init__(self, **data): | ||
| super().__init__(**data) | ||
| data["file_sha1"] = compute_sha1_from_content(data["bytes_content"]) |
StanGirard
added a commit
that referenced
this pull request
Jun 5, 2024
🤖 I have created a release *beep* *boop* --- ## 0.0.259 (2024-06-04) ## What's Changed * feat(upload): async improved by @AmineDiro in #2544 ## New Contributors * @AmineDiro made their first contribution in #2544 **Full Changelog**: v0.0.258...v0.0.259 --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Hey,
Here's a breakdown of what I've done:
langchainloader classes, we couldn't use memory buffers for the loaders. Now, with the changes made, we only open a single temporary file for eachprocess_file_and_notify, cutting down on excessive file opening, read syscalls, and memory buffer usage. This could cause stability issues when ingesting and processing large volumes of documents. Unfortunately, there is still reopening of temporary files in some code paths but this can be improved further in later work.UploadFileclass from File: TheUploadFile( a FastAPI abstraction over a SpooledTemporaryFile for multipart upload) was redundant in ourFilesetup since we already downloaded the file from remote storage and read it into memory + wrote the file into a temp file. By removing this abstraction, we streamline our code and eliminate unnecessary complexity.asyncfunction Adjustments: I've removed the async labeling from functions where it wasn't truly asynchronous. For instance, callingfilter_filefor processing files isn't genuinely async, ass async file reading isn't actually asynchronous—it uses a threadpool for reading the file . Given that we're already leveragingceleryfor parallelism (one worker per core), we need to ensure that reading and processing occur in the same thread, or at least minimize thread spawning. Additionally, since the rest of the code isn't inherently asynchronous, our bottleneck lies in CPU operations rather than asynchronous processing.These changes aim to improve performance and streamline our codebase.
Let me know if you have any questions or suggestions for further improvements!
Checklist before requesting a review