Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use iterator in aggregation stages #2480

Merged
merged 18 commits into from Apr 26, 2023

Conversation

chilagrow
Copy link
Contributor

@chilagrow chilagrow commented Apr 19, 2023

Description

Closes #1889.

  • $group needs to accumulate all values in the memory so it reads all document.
  • $unwind is a bit tricky to handle array iterator, it reads all document.

Readiness checklist

  • I added/updated unit tests.
  • I added/updated integration/compatibility tests.
  • I added/updated comments and checked rendering.
  • I made spot refactorings.
  • I updated user documentation.
  • I ran task all, and it passed.
  • I ensured that PR title is good enough for the changelog.
  • (for maintainers only) I set Reviewers (@FerretDB/core), Labels, Project and project's Sprint fields.
  • I marked all done items in this checklist.

@chilagrow chilagrow self-assigned this Apr 19, 2023
@codecov
Copy link

codecov bot commented Apr 19, 2023

Codecov Report

Merging #2480 (734151c) into main (ce28270) will decrease coverage by 0.33%.
The diff coverage is 66.98%.

❗ Current head 734151c differs from pull request most recent head 66cb98d. Consider uploading reports for the commit 66cb98d to get more accurate results

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2480      +/-   ##
==========================================
- Coverage   65.10%   64.78%   -0.33%     
==========================================
  Files         399      400       +1     
  Lines       19747    19769      +22     
==========================================
- Hits        12857    12808      -49     
- Misses       5968     6043      +75     
+ Partials      922      918       -4     
Impacted Files Coverage Δ
...rnal/handlers/common/aggregations/stages/stages.go 20.00% <ø> (ø)
internal/handlers/common/limit_iterator.go 82.75% <0.00%> (-2.96%) ⬇️
internal/handlers/tigris/msg_aggregate.go 0.00% <0.00%> (ø)
...l/handlers/common/aggregations/stages/collstats.go 83.92% <25.00%> (-12.15%) ⬇️
...ernal/handlers/common/aggregations/stages/group.go 87.71% <57.14%> (-1.51%) ⬇️
...rnal/handlers/common/aggregations/stages/unwind.go 82.47% <62.50%> (-2.31%) ⬇️
internal/handlers/pg/msg_aggregate.go 83.18% <77.77%> (+1.78%) ⬆️
internal/handlers/common/count_iterator.go 91.17% <91.17%> (ø)
...ernal/handlers/common/aggregations/stages/count.go 100.00% <100.00%> (ø)
...ernal/handlers/common/aggregations/stages/limit.go 81.25% <100.00%> (+11.25%) ⬆️
... and 4 more

... and 4 files with indirect coverage changes

Flag Coverage Δ
integration 58.00% <66.98%> (+<0.01%) ⬆️
mongodb 5.11% <0.00%> (-0.01%) ⬇️
pg 57.92% <66.98%> (+<0.01%) ⬆️
unit 25.28% <0.00%> (-0.39%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

@mergify
Copy link
Contributor

mergify bot commented Apr 19, 2023

@chilagrow this pull request has merge conflicts.

@mergify mergify bot added the conflict PRs that have merge conflicts label Apr 19, 2023
@chilagrow chilagrow force-pushed the issue-1889-aggregation-iterator branch from d9e5ff1 to 5363b18 Compare April 19, 2023 09:29
@mergify mergify bot removed the conflict PRs that have merge conflicts label Apr 19, 2023
@chilagrow chilagrow added the code/chore Code maintenance improvements label Apr 20, 2023
@chilagrow chilagrow changed the title Aggregation uses iterator Aggregation stages uses iterator Apr 20, 2023
@chilagrow chilagrow marked this pull request as ready for review April 20, 2023 08:03
@chilagrow chilagrow requested a review from a team as a code owner April 20, 2023 08:03
@chilagrow chilagrow enabled auto-merge (squash) April 20, 2023 08:04
@chilagrow chilagrow requested review from a team, w84thesun and noisersup April 20, 2023 08:04
Copy link
Member

@rumyantseva rumyantseva left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the implementation (it looks similar to how FilterIterator works), but I see quite a lot of things fail, so this PR needs some changes to make the linters and tests pass etc.

@chilagrow
Copy link
Contributor Author

Thanks @w84thesun for fixing the linter and merging main 🤗

@chilagrow
Copy link
Contributor Author

chilagrow commented Apr 25, 2023

I like the implementation (it looks similar to how FilterIterator works), but I see quite a lot of things fail, so this PR needs some changes to make the linters and tests pass etc.

@rumyantseva thanks for noticing this, I think linter/failing tests were due to not being up to date with main branch and few conflicting change. Should be fine now, thanks to @w84thesun 🙏

Copy link
Contributor

@w84thesun w84thesun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Copy link
Member

@rumyantseva rumyantseva left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@noisersup noisersup left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@chilagrow chilagrow merged commit ca200f5 into FerretDB:main Apr 26, 2023
18 of 20 checks passed
@rumyantseva rumyantseva changed the title Aggregation stages uses iterator Use iterator in aggregation stages Apr 26, 2023
@AlekSi AlekSi added this to the v1.1.0 milestone May 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
code/chore Code maintenance improvements
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

Use iterators in aggregation pipeline stages
5 participants