Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DBZ-7007 Use DebeziumEngine API instead of EmbeddedEngine in the testsuite #4960

Merged
merged 11 commits into from
Nov 6, 2023
Merged

DBZ-7007 Use DebeziumEngine API instead of EmbeddedEngine in the testsuite #4960

merged 11 commits into from
Nov 6, 2023

Conversation

vjuranek
Copy link
Member

@vjuranek vjuranek marked this pull request as draft October 25, 2023 12:27
@vjuranek
Copy link
Member Author

MySQL 5.7 error is unrelated:

2023-10-25T11:44:01.4623251Z [ERROR] Errors: 
2023-10-25T11:44:01.4630678Z [ERROR] io.debezium.connector.mysql.MySqlParserIT.parseTableWithInVisibleColumns
2023-10-25T11:44:01.4632888Z [ERROR]   Run 1: MySqlParserIT.beforeEach:53 <C2><BB> Runtime Couldn't obtain stable version for image debezium/example-mysql
2023-10-25T11:44:01.4635184Z [ERROR]   Run 2: MySqlParserIT.afterEach:73 NullPointer Cannot invoke "org.testcontainers.containers.MySQLContainer.stop()" because "this.mySQLContainer" is null
2023-10-25T11:44:01.4636887Z [INFO] 
2023-10-25T11:44:01.4637470Z [INFO] 
2023-10-25T11:44:01.4641042Z [ERROR] Tests run: 317, Failures: 0, Errors: 1, Skipped: 10

DB2 failures are unrelated (Db2ConnectorIT.restartInTheMiddleOfTx and other known well-known failures)

Server error is unrelated and should be fixed by debezium/debezium-server#44

However, Vitess error may be related.

@vjuranek
Copy link
Member Author

vjuranek commented Nov 2, 2023

It seems also Vitess failure is not related but a bug in the test - filed DBZ-7104 for it.

@vjuranek vjuranek marked this pull request as ready for review November 2, 2023 09:03
@vjuranek vjuranek changed the title DBZ-7007 Use DebeziumEngine instead of EmbeddedEngine in the testsuite DBZ-7007 Use DebeziumEngine API instead of EmbeddedEngine in the testsuite Nov 2, 2023
@vjuranek
Copy link
Member Author

vjuranek commented Nov 3, 2023

DB2 failures are unrelated:

 Error:  Failures: 
Error:    Db2ConnectorIT.restartInTheMiddleOfTx:698->restartInTheMiddleOfTx:603 
Expected size:<30> but was:<0> in:
<[]>
Error:    Db2ConnectorIT.restartInTheMiddleOfTxAfterCompletedTx:692->restartInTheMiddleOfTx:603 
Expected size:<30> but was:<0> in:
<[]>
Error:    Db2ConnectorIT.restartInTheMiddleOfTxAfterSnapshot:686->restartInTheMiddleOfTx:603 
Expected size:<30> but was:<0> in:
<[]>
Error:    IncrementalSnapshotIT>AbstractIncrementalSnapshotTest.snapshotWithAdditionalConditionWithRestart:802->AbstractSnapshotTest.consumeMixedWithIncrementalSnapshot:233->AbstractSnapshotTest.consumeMixedWithIncrementalSnapshot:182->AbstractSnapshotTest.consumeMixedWithIncrementalSnapshot:200 [Too many no data record results, 0 < 1000] 
Expecting:
 <6>
to be less than or equal to:
 <5> 
Error:    IncrementalSnapshotIT>AbstractIncrementalSnapshotTest.snapshotWithAdditionalConditionWithSurrogateKey:847->AbstractConnectorTest.assertNoRecordsToConsume:984 expected:<[tru]e> but was:<[fals]e>
[INFO] 
Error:  Tests run: 61, Failures: 5, Errors: 0, Skipped: 2

jpechane
jpechane reviewed Nov 6, 2023
View reviewed changes
Copy link
Contributor

@jpechane jpechane left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vjuranek LGTM, thanks! I left one comment WRT renaming of the isRunning attribute. When done please feel free to merge PR yourself.

debezium-embedded/src/test/java/io/debezium/embedded/AbstractConnectorTest.java Outdated Show resolved Hide resolved
@vjuranek
DBZ-7007 Use DebeziumEngine API instead of EmbeddedEngine in postgres…
cc0e3ce 
… tests
@vjuranek
DBZ-7007 Move embedded engine config into separate interface
2e6f47a 
This config will be re-used by possible other implementations of
DebeiumEngine API in the embedded package. As DebeziumEngine API
can have completely different implementations and thus also config,
the class is called `EmbeddedEngineConfig` as it's assumed to be used
only by embedded engine "family" of implementations.

To keep backward compatibility, the config options are extracted into
an interface and `EmbeddedEngine` implements this interface, thus
allowing to use these options in custom classes without any need for the
code changes.
@vjuranek
DBZ-7007 Use property based configuration in the emebdded tests
46b094e
@vjuranek
DBZ-7007 Switch to DebeziuEngine in EmbeddedEngineTest
f7fb314
@vjuranek
DBZ-7007 Switch to DebeziuEngine in ConnectorOutputTest
7de8684
@vjuranek
DBZ-7007 Use close() instead of EmbeddedEngine#stop() method
4504008
@vjuranek
DBZ-6213 Remove unneeded code from the testsuite
4752478 
After fixing start method to really start the connector, this wait loop
is not needed any more.
@vjuranek
DBZ-7007 Replace engine.await() with Awaitility condition
c735ae1 
In the second case replace infinite loop by one minute wait. If this is
not sufficient for any case, we can increase the timeout in the future.
@vjuranek
DBZ-7007 Remove engine.isRunning() from testsuite
31c34c8
@vjuranek
DBZ-7007 Introduce TestingDebeziumEngine and switch to it in Abstract…
91f39d3 
…ConnectorTest

Postgres `RecordsStreamProducerIT` reliaes on
EmebeddedEngine.runWithTask(). As this method effectively expose
engine's internal task and tests do the asserts against the state
of the task, it's hard to replace it. If we want to keep the tests,
the most simple approach seems to expose engine task in similar way
how EmbeddedEngine does that.

Add interface for testing Debezium engine, which would define minimal
set of methods which needs to be exposed by the implementing classes to
be able to run the testsuite against the Debezium engine. The number of
such methods should be as low as possible. Implementing classes would
typically act as proxies to actual `DebeziumEngine` implementations.

Add `TestingDebeziumEngine` implementation for `EmbeddedEngine` and
switch to `TestingDebeziumEngine` in `AbstractConnectorTest`.
@vjuranek
DBZ-7007 Use better name for auxiliary variable detecing if engine is…
b3f0024 
… running
@jpechane jpechane merged commit 11d2ff0 into debezium:main Nov 6, 2023
28 of 30 checks passed
@jpechane
Copy link
Contributor

jpechane commented Nov 6, 2023

@vjuranek Applied, thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jpechane jpechane jpechane left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@vjuranek @jpechane