Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stream documents #4544

Merged
merged 4 commits into from May 17, 2024
Merged

Stream documents #4544

merged 4 commits into from May 17, 2024

Conversation

irevoire
Copy link
Member

@irevoire irevoire commented Mar 28, 2024

Pull Request

Related issue

Fixes #4383

Perf

2M hackernews:

main:
Time to retrieve: 7s
RAM consumption: 2+GiB

stream:
Time to retrieve: 4.7s
RAM consumption: Too small

@irevoire irevoire added performance Related to the performance in term of search/indexation speed or RAM/CPU/Disk consumption CPU/RAM usage labels Mar 28, 2024
@irevoire irevoire marked this pull request as ready for review April 18, 2024 13:54
@irevoire irevoire added this to the v1.9.0 milestone Apr 18, 2024
Copy link
Member

@Kerollmops Kerollmops left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice job!
bors merge

meili-bors bot added a commit that referenced this pull request May 16, 2024
4544: Stream documents r=Kerollmops a=irevoire

# Pull Request

## Related issue
Fixes #4383


### Perf
2M hackernews:

main:
Time to retrieve: 7s
RAM consumption: 2+GiB

stream:
Time to retrieve: 4.7s
RAM consumption: Too small

Co-authored-by: Tamo <tamo@meilisearch.com>
Copy link
Contributor

meili-bors bot commented May 16, 2024

Build failed:

@curquiza
Copy link
Member

bors merge

Copy link
Contributor

meili-bors bot commented May 17, 2024

@meili-bors meili-bors bot merged commit 59ecf1c into main May 17, 2024
10 checks passed
@meili-bors meili-bors bot deleted the stream-documents branch May 17, 2024 15:17
@irevoire irevoire added the breaking change The related changes are breaking for the users label May 20, 2024
@irevoire
Copy link
Member Author

Reverting this PR since we realized it was actually introducing an unexpected breaking change in meilisearch sorry

meili-bors bot added a commit that referenced this pull request May 20, 2024
4644: Revert "Stream documents" and keep heed+arroy to the latest verion r=Kerollmops a=irevoire

Reverts #4544

Fixes #4641

I didn’t realize that some http clients were not handling chunked http requests like you would expect (if you ask the body, it gives you the body), which made the previous PR breaking.

There is no way to provide a good fix to the issue we initially wanted to fix without breaking meilisearch and that’s not planned for now.

Co-authored-by: Tamo <irevoire@protonmail.ch>
Co-authored-by: Tamo <tamo@meilisearch.com>
meili-bors bot added a commit that referenced this pull request May 20, 2024
4644: Revert "Stream documents" and keep heed+arroy to the latest verion r=curquiza a=irevoire

Reverts #4544

Fixes #4641

I didn’t realize that some http clients were not handling chunked http requests like you would expect (if you ask the body, it gives you the body), which made the previous PR breaking.

There is no way to provide a good fix to the issue we initially wanted to fix without breaking meilisearch and that’s not planned for now.

Co-authored-by: Tamo <irevoire@protonmail.ch>
Co-authored-by: Tamo <tamo@meilisearch.com>
meili-bors bot added a commit that referenced this pull request May 20, 2024
4644: Revert "Stream documents" and keep heed+arroy to the latest verion r=irevoire a=irevoire

Reverts #4544

Fixes #4641

I didn’t realize that some http clients were not handling chunked http requests like you would expect (if you ask the body, it gives you the body), which made the previous PR breaking.

There is no way to provide a good fix to the issue we initially wanted to fix without breaking meilisearch and that’s not planned for now.

Co-authored-by: Tamo <irevoire@protonmail.ch>
Co-authored-by: Tamo <tamo@meilisearch.com>
meili-bors bot added a commit that referenced this pull request May 20, 2024
4644: Revert "Stream documents" and keep heed+arroy to the latest verion r=curquiza a=irevoire

Reverts #4544

Fixes #4641

I didn’t realize that some http clients were not handling chunked http requests like you would expect (if you ask the body, it gives you the body), which made the previous PR breaking.

There is no way to provide a good fix to the issue we initially wanted to fix without breaking meilisearch and that’s not planned for now.

Co-authored-by: Tamo <irevoire@protonmail.ch>
Co-authored-by: Tamo <tamo@meilisearch.com>
meili-bors bot added a commit that referenced this pull request May 21, 2024
4644: Revert "Stream documents" and keep heed+arroy to the latest verion r=curquiza a=irevoire

Reverts #4544

Fixes #4641

I didn’t realize that some http clients were not handling chunked http requests like you would expect (if you ask the body, it gives you the body), which made the previous PR breaking.

There is no way to provide a good fix to the issue we initially wanted to fix without breaking meilisearch and that’s not planned for now.

Co-authored-by: Tamo <irevoire@protonmail.ch>
Co-authored-by: Tamo <tamo@meilisearch.com>
meili-bors bot added a commit that referenced this pull request May 21, 2024
4644: Revert "Stream documents" and keep heed+arroy to the latest verion r=Kerollmops a=irevoire

Reverts #4544

Fixes #4641

I didn’t realize that some http clients were not handling chunked http requests like you would expect (if you ask the body, it gives you the body), which made the previous PR breaking.

There is no way to provide a good fix to the issue we initially wanted to fix without breaking meilisearch and that’s not planned for now.

Co-authored-by: Tamo <irevoire@protonmail.ch>
Co-authored-by: Tamo <tamo@meilisearch.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking change The related changes are breaking for the users CPU/RAM usage performance Related to the performance in term of search/indexation speed or RAM/CPU/Disk consumption
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Stop allocating the documents we return in RAM
3 participants