-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pre-Work for adding Checkpointing #3163
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
@@ -154,6 +154,7 @@ private static RecordWriter recordWriterFunction(final Map<String, Index> indexN | |||
// Tools like DBT do not apply. Therefore, we need to try to write data in the most usable format | |||
// possible that does not require alteration. | |||
final String json = Jsons.serialize(recordStream |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rename variable: records
instead of recordStream
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
f08ba2e
to
27559b8
Compare
What
This PR handles cleaning up some stuff that has to be done before checkpointing can happening. Doing it here to avoid adding noise to the meat of the change.
AirbyteMessage
s instead of justAirbyteRecordMessage
s.RecordWriter
iface operate on lists of records (instead of streams). In practice the records were represented as lists on both sides of the interface already anyway. This interface it operating on batches that are being written to the db, so they need to be able to fit into memory anyway.