Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add boolean fields #1382

Merged
merged 18 commits into from
Jun 13, 2022
Merged

Add boolean fields #1382

merged 18 commits into from
Jun 13, 2022

Conversation

boraarslan
Copy link
Contributor

The first draft to add boolean field support. There is a very high chance I missed some stuff.

I will update the docs after the review (in case of something needs to change)

@codecov-commenter
Copy link

Codecov Report

Merging #1382 (c979d84) into main (5e599d9) will decrease coverage by 0.07%.
The diff coverage is 83.00%.

❗ Current head c979d84 differs from pull request most recent head 9c01d42. Consider uploading reports for the commit 9c01d42 to get more accurate results

@@            Coverage Diff             @@
##             main    #1382      +/-   ##
==========================================
- Coverage   94.26%   94.19%   -0.08%     
==========================================
  Files         235      235              
  Lines       42896    43186     +290     
==========================================
+ Hits        40437    40680     +243     
- Misses       2459     2506      +47     
Impacted Files Coverage Δ
src/postings/per_field_postings_writer.rs 94.11% <ø> (ø)
src/schema/document.rs 94.11% <0.00%> (-1.89%) ⬇️
src/schema/flags.rs 56.25% <ø> (ø)
src/schema/value.rs 57.08% <21.05%> (-3.20%) ⬇️
src/schema/field_type.rs 80.57% <44.00%> (-3.39%) ⬇️
src/indexer/segment_writer.rs 95.69% <63.63%> (-0.65%) ⬇️
src/schema/term.rs 84.49% <85.00%> (+0.04%) ⬆️
src/query/query_parser/query_parser.rs 94.87% <92.10%> (-0.10%) ⬇️
src/fastfield/mod.rs 94.52% <92.91%> (-0.18%) ⬇️
src/fastfield/writer.rs 90.17% <100.00%> (+0.04%) ⬆️
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5e599d9...9c01d42. Read the comment docs.

@PSeitz
Copy link
Contributor

PSeitz commented Jun 3, 2022

Nice job. Checkout test_operation_strategy and extend it to cover boolean fields. It covers different scenarios with randomized input.

@fulmicoton
Copy link
Collaborator

@PSeitz @boraarslan Should we merge?

@PSeitz Do we have a mechanism to use the field type to change the fastfield codec logic? It would be nice to force a bitset codec actually... Maybe even drop the DynamicReader for booleans.

@PSeitz
Copy link
Contributor

PSeitz commented Jun 7, 2022

@PSeitz Do we have a mechanism to use the field type to change the fastfield codec logic? It would be nice to force a bitset codec actually... Maybe even drop the DynamicReader for booleans.

Callingnew_u64_fast_field instead create_auto_detect_u64_fast_field_with_idx always uses Bitpacked encoding. (I think we need to rename that)
But if we have large blocks of true or false, the codec estimator could offer better compression.

@fulmicoton
Copy link
Collaborator

@boraarslan @PSeitz Any update?

@PSeitz
Copy link
Contributor

PSeitz commented Jun 10, 2022

LGTM, setting the codec can imo be done in another PR

@boraarslan
Copy link
Contributor Author

I agree with @PSeitz but If codec change is a must have I can look into that in this PR.

@fulmicoton
Copy link
Collaborator

@PSeitz Can you have approve & merge the PR?

@PSeitz PSeitz merged commit 88054aa into quickwit-oss:main Jun 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants