-
Notifications
You must be signed in to change notification settings - Fork 11
adds snapshot iterator #22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
db10d48 to
d045009
Compare
2344550 to
f4fd084
Compare
ae77df4 to
53be2c2
Compare
…s into dylan/snapshot-iterator
lovromazgon
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A couple of nitpicks, but the main things are the open transaction and the unchecked context in Next, after this we should be good 👍 The error related issues will be addressed in #27 so I skipped those.
source/logrepl/snapshot_test.go
Outdated
| rows, err := tx.Query(context.Background(), query) | ||
| is.NoErr(err) | ||
|
|
||
| var name *string | ||
| is.True(rows.Next()) | ||
| err = rows.Scan(&name) | ||
| is.NoErr(err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nitpick: we can simplify using QueryRow, we know there will be only one row. Also name doesn't have to be a pointer, and we can use ctx as the context.
| rows, err := tx.Query(context.Background(), query) | |
| is.NoErr(err) | |
| var name *string | |
| is.True(rows.Next()) | |
| err = rows.Scan(&name) | |
| is.NoErr(err) | |
| row := tx.QueryRow(ctx, query) | |
| var name string | |
| err = row.Scan(&name) | |
| is.NoErr(err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We actually can't use QueryRow here, because it messes with our transaction when we call Scan, which handles its own connection cleanup. Will update to match the rest, though 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried it out and it works for me 🤔
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If this indeed doesn't work we should open an issue in the driver repo because this doesn't do anything out of the ordinary, just a query within a transaction.
EDIT: I realized you are talking about the closing of the rows - that is implicit in here yes, but that doesn't mean that the connection or transaction is cleaned up.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So, I tested this before I wrote my first comment to verify that behavior still persisted, and it broke exactly as I expected it to the first time... but after you said you tested it and it worked for you, I went back and tested it again, and... it worked? 🤔 I guess I'll accept a passing test 😅
lovromazgon
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM 👍 Left two more comments, but approving already!
source/logrepl/snapshot.go
Outdated
|
|
||
| err = s.loadRows(ctx) | ||
| if err != nil { | ||
| if rollErr := s.tx.Rollback(ctx); err != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| if rollErr := s.tx.Rollback(ctx); err != nil { | |
| if rollErr := s.tx.Rollback(ctx); rollErr != nil { |
source/logrepl/snapshot.go
Outdated
| sdk.Logger(ctx).Err(err).Msg("load rows failed") | ||
| return nil, fmt.Errorf("rollback failed: %w", rollErr) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nitpick: I'd rather turn the logic around and just log the rollback error (same as we do in line 107)
| sdk.Logger(ctx).Err(err).Msg("load rows failed") | |
| return nil, fmt.Errorf("rollback failed: %w", rollErr) | |
| sdk.Logger(ctx).Err(rollErr).Msg("rollback failed") |
Description
Adds the snapshot iterator to the
logreplpackage.This is the first step towards adding a snapshot that works seamlessly with CDC mode.
We define a
SnapshotIteratorthat takes a config which allows for aSnapshotNameto be passed and iterates overthe rows from the configured table at that snapshot's point in time.
We necessarily secure a transaction with
READ REPEATABLEfrom the database for our iterator to read.Related to #15
Quick checks: