Add MongoDB with Panache extension #3650

loicmathieu · 2019-08-22T13:50:51Z

This PR adds a new extension: MongoDB with Panache.

This is a Panache implementation for MongoDB inspired by the one for Hibernate.

It provides the following:

Entity and Repository model close to the one that exist for Hibernate with Panache.
Support mapping customization via @BsonId, @BsonIgnore and @BsonProperty.
Offers a @MongoEntity annotation to customize the name of the collection and the database.
Offers the exact same methods that Hibernate with Panache, and support queries in two flavors, native queries and PanacheQL queries:
- Native queries : Person.find("{'firstname':?1}", firstname) or Person.find("{lastname'::lastname}", lastname)
- PanacheQL queries: Person.find("firstname", firstname) or Person.find("firstname = ?1 and lastname = ?2", ...) or Person.find("firstname =:firstname 1 and lastname = :lastname", ...) , you can find more query example on the guide with the list of operators supported by PanacheQL.
- field name replacement based on @BsonProperty are only supported on PanacheQL queries not on native one.
Type conversion is supported via the Pojo Codec that is used under the hood. Special support is provided for dates on the queries: you pass a date, it transforms it to a Mongo ISODate().
Projection is possible via the project() method on PanacheQuery : Person.find("firstname", firstname).project("lastname)
The default id is of type ObjectId, the name of the field is 'id' and can be customized via @BsonId. I create serializer/deserializer to String to facilitate it's usage with resteasy but didn't include them by default.
It offers low level access to the Mongo API with:
- find/list/stream/delete/count methods that accept a BSON Document query
- getMongoCollection() and getMongoDatabase() to access the underlying database and collection.

There is still some works to be done:

Decide if we keep the insert/update methods or only the persist methods.
Find a way to simplify the usage of getMongoCollection and getMongoDatabase that both needs to pass the entity class as parameter.
Include by default the ObjectId serializer/deserializer when resteasy-jsonb is in the classpath
Provides Jackson serializer/deserializer for ObjectId and include them whenresteasy-jacson is in the classpath
Optional: improve the usage of PanacheQL: support more operators and generate exception for unsupported queries.

loicmathieu · 2019-08-22T13:53:38Z

@emmanuelbernard, @FroMage : thanks the works we done together on the design (and for the paser )
@cescoffier thanks for your work on MongoDB client that enables this etension

gsmet · 2019-08-22T14:18:13Z

Maybe it should be squashed? Or at least reduced to a number of important semantic commits?

loicmathieu · 2019-08-22T15:07:09Z

@gmset: of course, but as it's a big PR I propose to wait for a first round of feedback/updates before doing the cosmetic changes as I assume I will have to make a lot of changes anyway ;).

machi1990

@loicmathieu really great PR, I mostly read and reviewed the guide doc and I have a couple of rephrasing suggestions. Thanks.

docs/src/main/asciidoc/mongodb-panache-guide.adoc

machi1990 · 2019-08-23T18:33:29Z

docs/src/main/asciidoc/mongodb-panache-guide.adoc

+
+MongoDB with Panacge relies on compile-time bytecode enhancements to your entities. 
+If you define your entities in the same project where you build your Quarkus application, everything will work fine. 
+If the entities come from external projects or jars, you can make sure that your jar is treated like a Quarkus application library by adding an empty `META-INF/beans.xml` file.


If this #3545 lands, we may need a link to it here.

So the cited PR has landed, maybe we can add a reference to How to Generate a Jandex Index section?

Yes please do that.

Done. If the phrasing is OK I can update the hibernate guide with the same one.

@gsmet is the new phrasing OK? Should I update the hibernate guide with it ?

...ongodb-panache/runtime/src/main/java/io/quarkus/mongodb/panache/runtime/MongoOperations.java

loicmathieu · 2019-08-27T09:25:26Z

@machi1990 can you review your conversations and close them if everything is OK for you

geoand · 2019-08-27T10:19:26Z

I think the PR needs to be rebased onto master to ensure the proper CI configuration for the native tests (now that #3719 has been merged)

docs/src/main/asciidoc/mongodb-panache-guide.adoc

loicmathieu · 2019-08-27T12:17:27Z

@geoand I rebase onto master, wait & see :)

geoand · 2019-08-27T12:18:17Z

Cool!

gsmet

I haven't reviewed the code but I have made a first pass on the documentation.

docs/src/main/asciidoc/mongodb-panache-guide.adoc

gsmet · 2019-08-28T10:44:52Z

docs/src/main/asciidoc/mongodb-panache-guide.adoc

+        return JsonbBuilder.create(config);
+    }
+
+}


It seems a bit unfortunate that you have to manually add this for the default id behavior.

Agreed, it's on the TODO list on the description of the PR :)

But the discussion about how to know if an extension is included needs to more forward because today there is no way to know if JsonB is included (not all extension provides a Capabilities - in fact only Arc and Narayana, so therer is currently no way to know that an extension is included).

Well, just add a capability if you need one. You can do that in separate PRs, we will merge them.

OK, will do

In fact, it's more complex that I first thought ...

Moreover the PR #3553, if merged, will change the way serializers/deserializers will be managed.
For the moment, I will update the TODO list with to do it later

@gsmet both JSON-B and Jackson serializer/deserializer are automatically registered if resteasy-jsonb or resteasy-jackson capability is present.

When changing one of these two library we need to carefully review this as it may creates incompabilities (for example an impact analyses needs to be done for PR #3553, and if we add some customization to Jackson ObjectMapper we need to port them to panache-mongo or change the way the ObjectMapper is created).

Fo this to work, I had to create a resteasy-common-spi module.

gsmet · 2019-08-28T10:45:54Z

docs/src/main/asciidoc/mongodb-panache-guide.adoc

+we would never do something like that for regular objects in the Object Oriented architecture, where state and methods are in the same class. Moreover, this requires two classes per entity, and requires injection of the DAO or Repository where you need to do entity operations, which breaks your edit flow and requires you to get out of the code you're writing to set up an injection point before coming back to use it.
+- MongoDB queries are super powerful, but overly verbose for common operations, requiring you to write queries even
+when you don't need all the parts.
+- MongoDB queries are JSON based, so you will need some String manipulation or using the `Document` type and it will needs a lot of boilerplate code.


I don't think the first reasons fly for MongoDB. They are more related to traditional JPA development than to the MongoDB API.

I can rephrase to

Even if MongoDB queries are not too verbose, PanacheQL go one step further by allowing to not write any queries for basic use cases.

This is a reference to find("name", name) shortcut query.

docs/src/main/asciidoc/mongodb-panache-guide.adoc

gsmet · 2019-08-28T10:47:18Z

docs/src/main/asciidoc/mongodb-panache-guide.adoc

+
+MongoDB with Panacge relies on compile-time bytecode enhancements to your entities. 
+If you define your entities in the same project where you build your Quarkus application, everything will work fine. 
+If the entities come from external projects or jars, you can make sure that your jar is treated like a Quarkus application library by adding an empty `META-INF/beans.xml` file.


Yes please do that.

FroMage

Sorry my review took so long, but that was a lot of code. Overall you know this is a great PR and we want to merge it ASAP. I added change requests, but those are pretty minor points.

Thanks a lot!

build-parent/pom.xml

devtools/common/src/main/filtered/extensions.json

docs/src/main/asciidoc/index.adoc

FroMage · 2019-09-05T14:10:26Z

docs/src/main/asciidoc/index.adoc

@@ -47,6 +47,7 @@ include::quarkus-intro.adoc[tag=intro]
 * link:kogito-guide.html[Using Kogito (business automation with processes and rules)]
 * link:oauth2-guide.html[Using OAuth2 RBAC]
 * link:tika-guide.html[Using Apache Tika]
+* link:mongodb-panache-guid.adoc[Simplified MongoDB with Panache]


I was going to suggest to put it next to the mongo guide, but it's not in the list :( Can you add it just before yours?
Also we need to figure out how to call the guide, because the current one is mongo-guide.adoc and yours uses mongodb. Given that the extension is called mongodb I suggest we rename the current guide. Agreed @cescoffier ?

I add it but I didn't rename it as it implies changing the links on the website and some coordination for the release. @gsmet is it worth it to rename the guide file name mongo-guide.adoc to mongodb-guide.adoc ?

docs/src/main/asciidoc/mongodb-panache-guide.adoc

...b-panache/runtime/src/main/java/io/quarkus/mongodb/panache/runtime/PanacheQlQueryBinder.java

extensions/panache/pom.xml

...ation-tests/mongodb-panache/src/main/java/io/quarkus/it/mongodb/panache/book/BookEntity.java

...sts/mongodb-panache/src/main/java/io/quarkus/it/mongodb/panache/book/BookEntityResource.java

...ployment/src/main/java/io/quarkus/mongodb/panache/deployment/PanacheMongoEntityEnhancer.java

loicmathieu · 2019-09-06T14:53:54Z

Thanks @FroMage for your review.

For me, there is two points that needs to be solved before being able to merge:

Is the current project() implementation that returns a PanacheQuery OK ?
Do we keep persist()/insert()/update(), or persist()/update() or ???. What I understood from @emmanuelbernard is that persist() sould be enought with an upsert semantic.

I need guidance for these two points, my opinion is to keep only persist() with it's current implementation (so removing all the insert and update methods) and to keep the current project() as is. We can later propose a project(Class) method that will allow to pass a projection interface on it.

I have stilll some tests to do (jackson and documentation among others), but for the other open points I prefere to open followup issues (as answered in my comments) when this PR will be merged as it's already a big PR opened since a long time :)

docs/src/main/asciidoc/mongo-guide.adoc

emmanuelbernard · 2019-09-18T01:22:05Z

I can see Stef hates the term upsert but that’s an industry accepted one. persistOrUpdate is cool for me as well and I guess we could implement it on data store that don’t support proper upserts.

…

On Tue 17 Sep 2019 at 01:43, Stéphane Épardaud ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In extensions/panache/mongodb-panache/runtime/src/main/java/io/quarkus/mongodb/panache/PanacheMongoEntityBase.java <#3650 (comment)>: > + * @see PanacheMongoEntity + */ +public abstract class PanacheMongoEntityBase { + + // Operations + + /** + * Insert this entity in the database. + * This will set it's ID field if not already set. + * + * @see #insert(Iterable) + * @see #insert(Stream) + * @see #insert(Object, Object...) + * TODO delete and keep only persist ? + */ + public void insert() { OK. I think that at some point we have to accept that there's an idiomatic way to do things when it comes to SQL people that differs from Mongo users. SQL people will be used to insert/update being separated, especially if coming from ORM (even though the update is implicit), and Mongo users will complain if they use upsert all the time and we don't provide it. I guess we can accept that we can please everyone and provide: - persist (inserts and new entity, will complain if the entity is already persisted or if the ID already exists in the DB) - update (updates an already-persisted entity to the DB, will complain if the entity is not in the DB) - persistOrUpdate that does an upsert. We can't use save next to persist because it's just a guess how they differ without reading the docs. upsert() is kind of lame because we're using persist and not insert, and upsist is gabbleduck. saveOrUpdate would be admitting that we should have called persist save ;) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3650?email_source=notifications&email_token=AACJNWEV27IELCNMWNRQRIDQKCKCXA5CNFSM4IOVIEIKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCE5WJSA#discussion_r325047627>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACJNWFHMP7VLM4UPVGZACTQKCKCXANCNFSM4IOVIEIA> .

FroMage · 2019-09-18T08:35:16Z

I can see Stef hates the term upsert but that’s an industry accepted one. persistOrUpdate is cool for me as well and I guess we could implement it on data store that don’t support proper upserts.

I don't hate upsert, I said it shows too much how our persist should have been called insert and it's too late to change that.

FroMage · 2019-09-18T08:38:28Z

@loicmathieu you're good to go on my end.

The native test failure doesn't seem related. Did you rebase on the latest master?

loicmathieu · 2019-09-18T09:57:40Z

@FroMage thanks a lot :)
I rebased onto master, let's wait for the CI.

loicmathieu · 2019-09-18T14:22:12Z

@gsmet features are OK and @FroMage approve it. So it's ready to go :)

FroMage · 2019-09-18T15:25:33Z

Hah, it passed CI.

gsmet

I added a bunch of comments and questions.

I would appreciate additional commit(s) for the updates, then I think we should squash everything into only one atomic commit.

gsmet · 2019-09-18T15:51:53Z

...-tests/mongodb-panache/src/test/java/io/quarkus/it/mongodb/panache/NativeBookResourceIT.java

+
+import io.quarkus.test.junit.SubstrateTest;
+
+@DisabledOnOs(OS.WINDOWS)


You need a comment describing why it's not run on Windows.

It nows run again on Windows

gsmet · 2019-09-18T15:52:54Z

...tion-tests/mongodb-panache/src/test/java/io/quarkus/it/mongodb/panache/BookResourceTest.java

+
+    @Container
+    private static final GenericContainer MONGO = new FixedHostPortGenericContainer("mongo:4.0")
+            .withFixedExposedPort(27018, 27017);


Is there a reason why we use test containers? We already use them for the other MongoDB extension?

I use test containers because mongodb-embedded that was use in the mongodb-client extension causes issue on my laptop (need to run the test as root). Other extension would also want to use test containers because it's very conveniant (Vault extension), there is an issue opened about it.
IMHO leveraging test containers can be very conveniant for Quarkus integration tests (for example to test different DBs).
But now that I finish the implementation, I can switch back to embeded MongoDB as it is used by mongodb-client extension.

I go back to flapddodle for embedded mongo to be able to run in Windows

gsmet · 2019-09-18T15:53:11Z

...tion-tests/mongodb-panache/src/test/java/io/quarkus/it/mongodb/panache/BookResourceTest.java

+import io.restassured.parsing.Parser;
+import io.restassured.response.Response;
+
+@DisabledOnOs(OS.WINDOWS)


We need a comment explaining why Windows is not supported.

It now runs again on Windows

gsmet · 2019-09-18T15:53:46Z

integration-tests/mongodb-panache/src/main/resources/application.properties

+quarkus.mongodb.write-concern.journal=false
+quarkus.mongodb.database=books
+
+quarkus.log.category."io.quarkus.mongodb.panache.runtime".level=DEBUG


I would remove that to avoid having the build being too verbose.

Log DEBUG on panache mongo only allow to trace the queries send to MongoDB. For the current integration test it adds 10 lines to the buid.
I found it very convenient, is it OK to keep it ?

As a general rule, I would prefer to have it commented out but I suppose we can live with 10 more lines :).

gsmet · 2019-09-18T15:54:28Z

integration-tests/mongodb-panache/pom.xml

+                </configuration>
+            </plugin>
+            <plugin>
+                <groupId>${project.groupId}</groupId>


You need to make that one io.quarkus now or you will have a warning.

gsmet · 2019-09-18T16:25:39Z

docs/src/main/asciidoc/mongodb-panache-guide.adoc

+
+== Transactions
+
+MongoDB offers ACID transactions since version 4.0. MongoDB with Panache doesn't provide support for them.


Maybe make it a warning?

done
I hope it can be done soon as Narayana should be able to provide support for it

docs/src/main/asciidoc/mongodb-panache-guide.adoc

loicmathieu · 2019-09-23T11:20:01Z

@gsmet I finally found the issue with this PR, I made a bad merge ...
Now the tests passes.
If eveything is OK on your side I can squash everything on one commit after the CI pass and ... you will be able to merge it :)

gsmet · 2019-09-24T11:59:30Z

@loicmathieu yes please squash everything and, unfortunately, we have a conflict now.

I'll try to merge it as soon as CI is green.

loicmathieu · 2019-09-24T13:18:58Z

@gsmet I rebased, fixed the merge issue (conflicted build capabilities), launched the test and squashed everything.
When the CI will be green again we should be good to go ;)

machi1990 · 2019-09-24T13:50:03Z

@gsmet I rebased, fixed the merge issue (conflicted build capabilities), launched the test and squashed everything.
When the CI will be green again we should be good to go ;)

Excited to see this in. Thanks @loicmathieu.

loicmathieu · 2019-09-24T14:00:51Z

@machi1990 as it is a big PR, testing and feedback will be appreciate :)

loicmathieu · 2019-09-24T16:00:28Z

@gsmet: all green !!!

gsmet · 2019-09-24T16:05:04Z

Merged! Congrats to you and Stéphane for this big achievement.

loicmathieu · 2019-09-24T16:07:28Z

Thanks for all you guys that worked with me on this one: @FroMage, @emmanuelbernard and @gsmet it's been a long and very interresting journey :)

cescoffier requested review from gsmet and FroMage August 22, 2019 14:37

loicmathieu force-pushed the features/mongo_panache branch 3 times, most recently from 63c7748 to 6602c01 Compare August 23, 2019 14:27

machi1990 reviewed Aug 23, 2019

View reviewed changes

loicmathieu force-pushed the features/mongo_panache branch 3 times, most recently from 67317cf to 8db1717 Compare August 26, 2019 09:01

machi1990 reviewed Aug 27, 2019

View reviewed changes

docs/src/main/asciidoc/mongodb-panache-guide.adoc Outdated Show resolved Hide resolved

loicmathieu force-pushed the features/mongo_panache branch from 8db1717 to 0dbc935 Compare August 27, 2019 12:13

loicmathieu force-pushed the features/mongo_panache branch from bc7ae00 to c8edaad Compare August 28, 2019 09:27

gsmet requested changes Aug 28, 2019

View reviewed changes

loicmathieu force-pushed the features/mongo_panache branch 4 times, most recently from 81d9123 to de8b33d Compare September 2, 2019 13:10

FroMage requested changes Sep 5, 2019

View reviewed changes

FroMage reviewed Sep 6, 2019

View reviewed changes

...ployment/src/main/java/io/quarkus/mongodb/panache/deployment/PanacheMongoEntityEnhancer.java Show resolved Hide resolved

loicmathieu force-pushed the features/mongo_panache branch 2 times, most recently from 2f6d94a to a753d02 Compare September 6, 2019 14:34

FroMage reviewed Sep 9, 2019

View reviewed changes

docs/src/main/asciidoc/mongo-guide.adoc Outdated Show resolved Hide resolved

FroMage approved these changes Sep 18, 2019

View reviewed changes

loicmathieu force-pushed the features/mongo_panache branch from 703528b to d48d173 Compare September 18, 2019 09:55

gsmet requested changes Sep 18, 2019

View reviewed changes

gsmet added the release/noteworthy-feature label Sep 18, 2019

loicmathieu force-pushed the features/mongo_panache branch 3 times, most recently from e0c8442 to 4bd0d3b Compare September 23, 2019 11:16

loicmathieu force-pushed the features/mongo_panache branch from d561d4a to 787db9e Compare September 23, 2019 14:01

gsmet added this to the 0.24.0 milestone Sep 24, 2019

feat: implements MongoDB with Panache

1946498

loicmathieu force-pushed the features/mongo_panache branch from 787db9e to 1946498 Compare September 24, 2019 13:17

gsmet approved these changes Sep 24, 2019

View reviewed changes

gsmet merged commit b347c6d into quarkusio:master Sep 24, 2019

loicmathieu mentioned this pull request Sep 25, 2019

Followup on MongoDB with Panache #4193

Closed

9 tasks

gsmet changed the title ~~Extension: MongoDB with Panache~~ Add MongoDB with Panache extension Oct 10, 2019

loicmathieu deleted the features/mongo_panache branch December 17, 2019 12:57


		import io.quarkus.test.junit.SubstrateTest;

		@DisabledOnOs(OS.WINDOWS)


		== Transactions

		MongoDB offers ACID transactions since version 4.0. MongoDB with Panache doesn't provide support for them.

Add MongoDB with Panache extension #3650

Add MongoDB with Panache extension #3650

Conversation

loicmathieu commented Aug 22, 2019 • edited Loading

loicmathieu commented Aug 22, 2019

gsmet commented Aug 22, 2019

loicmathieu commented Aug 22, 2019

machi1990 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

loicmathieu commented Aug 27, 2019

geoand commented Aug 27, 2019

loicmathieu commented Aug 27, 2019

geoand commented Aug 27, 2019

gsmet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

loicmathieu Aug 28, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FroMage left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

loicmathieu commented Sep 6, 2019

emmanuelbernard commented Sep 18, 2019 via email

FroMage commented Sep 18, 2019

FroMage commented Sep 18, 2019

loicmathieu commented Sep 18, 2019

loicmathieu commented Sep 18, 2019

FroMage commented Sep 18, 2019

gsmet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

loicmathieu commented Sep 23, 2019 • edited Loading

gsmet commented Sep 24, 2019

loicmathieu commented Sep 24, 2019

machi1990 commented Sep 24, 2019

loicmathieu commented Sep 24, 2019

loicmathieu commented Sep 24, 2019

gsmet commented Sep 24, 2019

loicmathieu commented Sep 24, 2019

loicmathieu commented Aug 22, 2019 •

edited

Loading

loicmathieu Aug 28, 2019 •

edited

Loading

loicmathieu commented Sep 23, 2019 •

edited

Loading