Debezium MongoDB Connector native support

[JiriOndrusek]

No description provided.

[JiriOndrusek]

Please, assign to me.

[JiriOndrusek]

Issue is blocked by the fact that debezium is using driver 3.x, but quarkus-mongodb-client uses 4.x.
We have to wait for the debezium to upgrade their driver - https://issues.redhat.com/browse/DBZ-2138

[ppalaga]

@JiriOndrusek do you have a proper test and is it passing the JVM mode at least?

[JiriOndrusek]

@ppalaga I have branch with tests.
Tests can be seen here: https://github.com/JiriOndrusek/camel-quarkus/tree/1190_debezium_mongodb/integration-tests/debezium/src/test/java/org/apache/camel/quarkus/component/debezium/it/mongodb

All tests for debezium-* extensions are gathered together to one integration-test module.

I can review it, make sure it passes jvm and then create PR with mongodb-tests with @disabled. WDYT?

edited
Sorry for not answering directly.: yes, tests are prepared and in the time of its creation, tests in jvm were successful

[ppalaga]

I vote for adding the tests to the existing JVM module https://github.com/apache/camel-quarkus/tree/master/extensions-jvm/debezium-mongodb/integration-test
They can well be enabled there and increase our confidence that the extension works well in the JVM mode. WDYT?

[JiriOndrusek]

I see added value with these approach. My only concern is about the fact, that all debezium integration tests are in one module and they are using common code (abstract parents).
Refactoring mongodb test into JVM module would bring redundant code (I would need to copy several classes from https://github.com/apache/camel-quarkus/tree/master/integration-tests/debezium/src/test/java/org/apache/camel/quarkus/component/debezium/common/it and possibly code from https://github.com/apache/camel-quarkus/blob/master/integration-tests/debezium/src/main/java/org/apache/camel/quarkus/component/debezium/common/it/AbstractDebeziumResource.java). But because we plan to remove this jvm test and use this 'common' module (once native is not blocked), it is not probably big problem. Do you agree?

[ppalaga]

Oh, I see. OK, let's add them to the common Debezium test module and let's not create the native test class so that they run only in the JVM mode?

[JiriOndrusek]

ok, sound good. I'll prepare it

[JiriOndrusek]

I've started to refactor of mongodb test for jvm only. I discovered that mongodb tests are not complete in my branch (I've had probably discovered that extension was blocked before finishing them) - I'll finish them.

[JiriOndrusek]

After investigation it seems that it is not possible to have working jvm tests for debezium mongodb. Here are reasons:

• Camel-quarkus uses mongodb version from quarkus. Which brings driver of version 4.x - d07e2a5#diff-2cdf2d35793fae7cd8f70171d04de916f2a6c295b19f099a0aea50589be615b4R442
• Connection to mongodb from component is provided by io.debezium:debezium-connector-mongodb:jar:1.3.0.Final, which will support driver 4.x from version 1.4 (https://issues.redhat.com/browse/DBZ-2138)

Moreover current debezium mongodb extension does not work in JVM, because it uses newer mongodb driver.
Solution could be to exclude all new mongodb dependencies and use old ones. But because versions of such dependencies are restricted by bom. Solution, where there are more versions of mongodb driver in bom, should not be possible. (In other words, making jvm debeziom-mongodbwork, would fail all extension using mongdb)

@ppalaga What do you think? I'd suggest to wait with completing extension to release of io.debezium:debezium-connector-mongodb:jar:1.4..x. It would be also nice to remove debezium-mongodb from extension-jvm, because it can not work.

[ppalaga]

It would be nice to check that Debezium master (or whichever branch they are going to release their next 1.4.x from) works for us. If so, let's wait for DBZ 1.4. WDYT?

[JiriOndrusek]

I'll try to build some debezium-1.4.snapshot and prepare extension to the branch.

In time debezium 1.4 is out, it should be enough to change version in pom and whole extension should be ready.

[jamesnetherton]

In time debezium 1.4 is out, it should be enough to change version in pom and whole extension should be ready.

We should bump the Debezium version on the Camel side too. Which may actually be a prerequisite to getting this working. I remember in the past we had issues where code was compiled against the Mongo 3.x drivers but 4.x was used at runtime.

[JiriOndrusek]

Yes, upgrade of debezium in camel sounds like a correct step. I'll create jira ticket for it and link here.

[JiriOndrusek]

I'm trying to prepare JVM tests for debezium-mongodb. But I've found a problem in debezium-connector, which prevents me to finish test using test container (and therefore random port). I've reported issue as https://issues.redhat.com/browse/DBZ-2831.

After the fix, I'll continue with preparation of tests using 1.4.0-SNAPSOT.

[ppalaga]

Thanks @JiriOndrusek, DBZ-2831 being an issue with quite a serious impact, could you please file a proxy issue on our side too?

[JiriOndrusek]

Ok, issue is created - #2060

[JiriOndrusek]

@ppalaga I've created jvm only test in extension-jvm/integration-test in my repo - https://github.com/JiriOndrusek/camel-quarkus/tree/debezium-mongodb-test-jvm-attempt-2

Test is using 1.4.0-SNAPSHOT of debezium-connector - it can not be merged as a part of jvm component so far.

If mongodb container's exposed port is set to 27017 (default one) - code , test is successful.

I'll look into an option to make it work with 1.3.0-final version of debezium-connector to be able to marge it into git...

[ppalaga]

'll look into an option to make it work with 1.3.0-final version of debezium-connector

Do not hesitate to give up if it turns out to be too much work. DBZ 1.4.x might come soon enough.