Provide a mongodb connector for Syndesis

[heiko-braun]

See also https://issues.jboss.org/browse/ENTESB-11499

As a citizen user I want to be able to connect to MongoDB datastore. I will provide my own connection configuration strings such as:

mongodb://[username:password@]host1[:port1][,...hostN[:portN]]][/[database][?options]]

(See https://docs.mongodb.com/manual/reference/connection-string/)

Acceptance Criteria

• CRUD support

Stretch items

• Adhoc queries (regex, range query)
• Javascript queries
• Aggregation (Map-reduce)
• Force ACID Tranactions

--There is an initial draft of the user doc for this feature.
--The writer acknowledges receiving enough information from engineering to write the initial draft of the user doc for this feature.

[rareddy]

From Teiid side the connection properties, I will be interested in capturing will be

mongodb.host=localhost
mongodb.port=27017
mongodb.database=sampledb
mongodb.username=admin
mongodb.password=admin
mongodb.authenticationDatabase=admin

These are in sync with Spring Boot properties

[oscerd]

In camel they are different

[rareddy]

I am not concerned if the names are different, but if I can map properties one to one or extract above that would suffice the data virtualization need.

[squakez]

First basic structure and unit testing are showing that upstream component is working fine. However there may be a problem due to the fact that a connection bean is required on the CamelContext. Will provide further analysis.

[lburgazzoli]

This is going to be an issue, I think that we need to enhance the the component upstream to allow more flexibility for configuring the mongodb connection

On Fri, 7 Jun 2019 at 17:21, Pasquale Congiusti ***@***.***> wrote: First basic structure and unit testing are showing that upstream component is working fine. However there may be a problem due to the fact that a connection bean is required on the CamelContext. Will provide further analysis. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#5337?email_source=notifications&email_token=AAOIJBL2T6ICIH5IJ6HEVMLPZJ4HLA5CNFSM4HMCOYM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXGETMI#issuecomment-499927473>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAOIJBLJGES7KF6756ZRI6LPZJ4HLANCNFSM4HMCOYMQ> .

-- -- Luca Burgazzoli

[squakez]

I've created this issue to enhance the upstream component: #5773

[squakez]

@rareddy, about the connection configuration you're mentioning the presence of mongodb.database this would be more typical of an integration configuration (within collection). The user may use the same connection to connect to different databases. Can we keep this out of the list of connection properties? the rest is already configured.

[rareddy]

@squakez you mean the Collection name? I am not sure I follow use the same configuration to connect to different databases?

[asmigala]

The current implementation requires that the collection used is a capped collection .

This has several limitations:

1. capped collections have a specific size and when this is exceeded, the oldest document in the collection is removed
2. updates to documents stored in a capped collection cannot change the size of the document, i.e. {"_id": 1, "foo": "bar"} cannot be changed to {"_id": 1, "foo": "barX"} because the new document would be longer
3. documents cannot be deleted from a capped collection

All of this severly limits the usability of this connector, to the point where I do not think we can claim general mongodb support.

@gaughan @squakez

[squakez]

As just discussed in chat, likely capped collection must be used for consumer only. Produced should be used and tested with normal collections. It would be a matter of highlighting in the connector documentation.

[asmigala]

Ok, so per my current understanding

Operation	Capped Collection	Regular Collection
create	✔️	✔️
read	✔️	❌
update	❌	✔️
delete	❌	✔️

[rareddy]

@squakez is there a limitation to use the capped collection for consumer?

[gaughan]

If we know potential breaking changes are needed to complete or add to the matrix above, the connector should be TP until we can refactor.

[squakez]

As mentioned in the requirement document: Integration with Mongo as a consumer: the upstream component feature is limited by the fact that a Mongo collection must be a capped collection with a feature named “Tailable cursor consumer”. We will start with this solution and evolve to Change Streams if we get it required.

The change stream feature is something added lately to the upstream component, and, unfortunately it's only available in camel 3. I made some quick test to see how easy would be to backport to 2.x but, unfortunately there are breaking changes that may require some effort to have it working.

I can give priority to the refactor of the change stream feature to have it working in camel 2.x, though I cannot estimate the effort needed.

[squakez]

I've run some experiment to see the complexity of backporting the feature to from actual 3.0 to 2.x. It seems it's viable, so I'll keep working on that direction and implement the change stream to have the connector GA. I've created an ad-hoc subtask: #6436

[heiko-braun]

Migrated to https://issues.jboss.org/browse/ENTESB-11499

[rareddy]

@heiko-braun @squakez Just FYI, the DV support can NOT be added with out additional support for the default mongodb.database property. This will akin providing the "schema" in the "sql" connector.