REP-6766 Use a batch cursor to parse the change stream #153
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
FGasper
changed the title
FGasper
added a commit
that referenced
this pull request
Oct 31, 2025
This alters document comparison to use a batch cursor instead of the Go driver’s cursor. The advantage here is that, instead of creating giant slices of documents (which are copied from the driver), we’ll read the raw documents directly from the server response buffer. This minimizes GC pressure. Pprof analysis shows that before this change, slices.Clone() allocated more heap than any other functions. With this change, those allocations are gone. PR #153 will make the change stream use the same batch cursor.
tdq45gj
approved these changes
Oct 31, 2025
Collaborator
tdq45gj
left a comment
tdq45gj
left a comment
There was a problem hiding this comment.
Left two small suggestions. LGTM.
| err = csr.readAndHandleOneChangeEventBatch(ctx, ri, cs, sess) | ||
| err = csr.readAndHandleOneChangeEventBatch(ctx, ri, csCursor) | ||
|
|
||
| if err := csCursor.GetNext(ctx); err != nil { |
Collaborator
There was a problem hiding this comment.
I would prefer a different variable name for the GetNext error, especially since we haven't finished with the original err.
| SetMaxAwaitTime(maxChangeStreamAwaitTime) | ||
| ) (*cursor.BatchCursor, bson.Timestamp, error) { | ||
|
|
||
| changeStreamStage := bson.D{ |
Collaborator
There was a problem hiding this comment.
We could use the BSON builder.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This applies the new batch cursor from PR #148 to the change stream.