Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Faster document validation #5163

Merged
merged 3 commits into from
Jun 27, 2024
Merged

Faster document validation #5163

merged 3 commits into from
Jun 27, 2024

Conversation

fulmicoton
Copy link
Contributor

@fulmicoton fulmicoton commented Jun 26, 2024

This PR just removes the need to build documents when validating
documents.

@fulmicoton fulmicoton force-pushed the faster-document-validation branch 5 times, most recently from 4a6a6a3 to 8af87ed Compare June 26, 2024 02:05
@fulmicoton fulmicoton marked this pull request as ready for review June 26, 2024 02:06
@fulmicoton fulmicoton requested a review from guilload June 26, 2024 02:33
Copy link

github-actions bot commented Jun 26, 2024

On SSD:

Average search latency is 1.02x that of the reference (lower is better).
Ref run id: 2328, ref commit: ae2227a
Link

On GCS:

Average search latency is 1.02x that of the reference (lower is better).
Ref run id: 2306, ref commit: 1302863
Link

This PR just removes the need to build documents when validating
documents.
Copy link
Member

@guilload guilload left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The PR could use some unit tests, though.

quickwit/quickwit-ingest/src/ingest_v2/doc_mapper.rs Outdated Show resolved Hide resolved
quickwit/quickwit-doc-mapper/src/doc_mapper.rs Outdated Show resolved Hide resolved
fulmicoton and others added 2 commits June 27, 2024 15:08
Co-authored-by: Adrien Guillo <adrien@quickwit.io>
@fulmicoton fulmicoton merged commit 6ac0ef8 into main Jun 27, 2024
4 checks passed
@fulmicoton fulmicoton deleted the faster-document-validation branch June 27, 2024 08:20
fulmicoton added a commit that referenced this pull request Jul 2, 2024
* Faster document validation

This PR just removes the need to build documents when validating
documents.

* Apply suggestions from code review

Co-authored-by: Adrien Guillo <adrien@quickwit.io>

* minor change

---------

Co-authored-by: Adrien Guillo <adrien@quickwit.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants