Skip to content

Resumable Full Refresh sync for mssql - #37451

Merged
Rodi Reich Zilberman (rodireich) merged 69 commits into
masterfrom
rodi/5th-7016
May 7, 2024
Merged

Rodi Reich Zilberman (rodireich) merged 69 commits into
masterfrom
rodi/5th-7016

Conversation

@rodireich

@rodireich Rodi Reich Zilberman (rodireich) commented Apr 19, 2024 •

Copy link
Copy Markdown
Contributor

The implementation for Mssql references MySql's implementation: #36932

@vercel

vercel Bot commented Apr 19, 2024 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
airbyte-docs ⬜️ Ignored (Inspect) Visit Preview May 7, 2024 5:49am

@rodireich
Rodi Reich Zilberman (rodireich) changed the base branch from master to xiaohan/cdk-rfr-interface April 23, 2024 22:25
@rodireich
Rodi Reich Zilberman (rodireich) marked this pull request as ready for review April 30, 2024 23:49
@rodireich Rodi Reich Zilberman (rodireich) changed the title RFR for mssql Resumable Full Refresh sync for mssql Apr 30, 2024
)
abstract class JdbcSourceAcceptanceTest<S : Source, T : TestDatabase<*, T, *>> {
@JvmField protected var testdb: T = createTestDatabase()
@JvmField protected var testdb: T? = null

@rodireich Rodi Reich Zilberman (rodireich) May 1, 2024 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: This line created a new test db every time an instance was created, leaving it open, while test themselves opened another test db in setup.
This left test db's that were never torn down

bgThread.stop = true;
}
super.close();
MssqlDebeziumStateUtil.disposeInitialState();

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: Because concurrent mssql tests are reusing threads in a fixed thread pool, it is necessary to dispose of cached state so the next test using this thread will start fresh.
Cached initial state is tied to a thread (ThreadLocal<>)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pretty clean! I'm halfway through the PR and will finish later.

? new CdcState().withState(initialDebeziumState)
: stateManager.getCdcStateManager().getCdcState();
cdcStreamsForInitialOrderedColumnLoad(stateManager.getCdcStateManager(), catalog, savedOffsetStillPresentOnServer);
final CdcState stateToBeUsed = getCdcState(database, catalog, stateManager, savedOffsetStillPresentOnServer);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should rename this to initialState or initialStateToBeUsed

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

final Long syncCheckpointRecords = config.get(SYNC_CHECKPOINT_RECORDS_PROPERTY) != null ? config.get(SYNC_CHECKPOINT_RECORDS_PROPERTY).asLong()
: DebeziumIteratorConstants.SYNC_CHECKPOINT_RECORDS;

initialLoadStateManager.setStreamStateForIncrementalRunSupplier(streamStateForIncrementalRunSupplier);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realized this will be a bug, I reproduced it on mysql and postgres. I think mssql is fine because we set streamStateForIncrementalRunSupplier the same for both full refresh and incremental refresh - so just FYI

Since we share initialLoadStateManager, and we setup the iterators before actually iterating through them, the full refresh setup will overwrite the ones used in incremental iterators.

In postgres and mysql my solution was to provide a optional streamStateForIncrementalRunSupplier and in state manager it will default to emptyObject. So I think we achieved the same thing but just with different approach.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's take this offline. I'd like to understand how mssql is different.


@TestInstance(Lifecycle.PER_METHOD)
@Execution(ExecutionMode.CONCURRENT)
@Execution(ExecutionMode.SAME_THREAD)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this still required?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops. forgot to change back. thanks

Comment thread docs/integrations/sources/mssql.md Outdated
|:--------|:-----------| :---------------------------------------------------------------------------------------------------------------- |:------------------------------------------------------------------------------------------------------------------------------------------------|
| 4.0.15 | 2024-04-22 | [37541](https://github.com/airbytehq/airbyte/pull/37541) | Adopt latest CDK. reduce excessive logs. |
|:--------|:-----------|:------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------|
| 4.0.16 | 2024-04-30 | [37451](https://github.com/airbytehq/airbyte/pull/37451) | Resumable full refresh read of tables. |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: this needs to be a minor update! and also we need to figure out how to highlight the minimum platform requirement in this doc.

this.pairToOrderedColLoadStatus = MssqlInitialLoadStateManager.initPairToOrderedColumnLoadStatusMap(initialLoadStreams.pairToInitialLoadStatus());
}

public MssqlInitialLoadStreamStateManager(final ConfiguredAirbyteCatalog catalog,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can delete this now

@rodireich
Rodi Reich Zilberman (rodireich) changed the base branch from xiaohan/cdk-rfr-interface to master May 3, 2024 22:47
throw new RuntimeException(String.format("Stream %s was not provided with an appropriate cursor", stream.getStream().getName()));
}
final CursorBasedStatus cursorBasedStatus = new CursorBasedStatus();
final Optional<String> maybeCursorField = Optional.ofNullable(cursorInfoOptional.get().getCursorField());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stephane Geneix (@stephane-airbyte) actually this change by Rodi Reich Zilberman (@rodireich) should have fixed the issue that we were discussing here

#39143 (comment)

Noticing the use of Optional and ifPresent function below which were not checked in the previous version. So I will close that issue.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes. This is indeed fixing it. Thank you

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the discussion that led to this investigation!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/connectors Connector related issues area/documentation Improvements or additions to documentation CDK Connector Development Kit connectors/source/mssql

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants