Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add MongoDB with Panache extension #3650

Merged
merged 1 commit into from
Sep 24, 2019

Conversation

loicmathieu
Copy link
Contributor

@loicmathieu loicmathieu commented Aug 22, 2019

This PR adds a new extension: MongoDB with Panache.

This is a Panache implementation for MongoDB inspired by the one for Hibernate.

It provides the following:

  • Entity and Repository model close to the one that exist for Hibernate with Panache.
  • Support mapping customization via @BsonId, @BsonIgnore and @BsonProperty.
  • Offers a @MongoEntity annotation to customize the name of the collection and the database.
  • Offers the exact same methods that Hibernate with Panache, and support queries in two flavors, native queries and PanacheQL queries:
    • Native queries : Person.find("{'firstname':?1}", firstname) or Person.find("{lastname'::lastname}", lastname)
    • PanacheQL queries: Person.find("firstname", firstname) or Person.find("firstname = ?1 and lastname = ?2", ...) or Person.find("firstname =:firstname 1 and lastname = :lastname", ...) , you can find more query example on the guide with the list of operators supported by PanacheQL.
    • field name replacement based on @BsonProperty are only supported on PanacheQL queries not on native one.
  • Type conversion is supported via the Pojo Codec that is used under the hood. Special support is provided for dates on the queries: you pass a date, it transforms it to a Mongo ISODate().
  • Projection is possible via the project() method on PanacheQuery : Person.find("firstname", firstname).project("lastname)
  • The default id is of type ObjectId, the name of the field is 'id' and can be customized via @BsonId. I create serializer/deserializer to String to facilitate it's usage with resteasy but didn't include them by default.
  • It offers low level access to the Mongo API with:
    • find/list/stream/delete/count methods that accept a BSON Document query
    • getMongoCollection() and getMongoDatabase() to access the underlying database and collection.

There is still some works to be done:

  • Decide if we keep the insert/update methods or only the persist methods.
  • Find a way to simplify the usage of getMongoCollection and getMongoDatabase that both needs to pass the entity class as parameter.
  • Include by default the ObjectId serializer/deserializer when resteasy-jsonb is in the classpath
  • Provides Jackson serializer/deserializer for ObjectId and include them whenresteasy-jacson is in the classpath
  • Optional: improve the usage of PanacheQL: support more operators and generate exception for unsupported queries.

@loicmathieu
Copy link
Contributor Author

@emmanuelbernard, @FroMage : thanks the works we done together on the design (and for the paser )
@cescoffier thanks for your work on MongoDB client that enables this etension

@gsmet
Copy link
Member

gsmet commented Aug 22, 2019

Maybe it should be squashed? Or at least reduced to a number of important semantic commits?

@cescoffier cescoffier requested review from gsmet and FroMage August 22, 2019 14:37
@loicmathieu
Copy link
Contributor Author

@gmset: of course, but as it's a big PR I propose to wait for a first round of feedback/updates before doing the cosmetic changes as I assume I will have to make a lot of changes anyway ;).

@loicmathieu loicmathieu force-pushed the features/mongo_panache branch 3 times, most recently from 63c7748 to 6602c01 Compare August 23, 2019 14:27
Copy link
Member

@machi1990 machi1990 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@loicmathieu really great PR, I mostly read and reviewed the guide doc and I have a couple of rephrasing suggestions. Thanks.

docs/src/main/asciidoc/mongodb-panache-guide.adoc Outdated Show resolved Hide resolved
docs/src/main/asciidoc/mongodb-panache-guide.adoc Outdated Show resolved Hide resolved
docs/src/main/asciidoc/mongodb-panache-guide.adoc Outdated Show resolved Hide resolved
docs/src/main/asciidoc/mongodb-panache-guide.adoc Outdated Show resolved Hide resolved
docs/src/main/asciidoc/mongodb-panache-guide.adoc Outdated Show resolved Hide resolved
docs/src/main/asciidoc/mongodb-panache-guide.adoc Outdated Show resolved Hide resolved

MongoDB with Panacge relies on compile-time bytecode enhancements to your entities.
If you define your entities in the same project where you build your Quarkus application, everything will work fine.
If the entities come from external projects or jars, you can make sure that your jar is treated like a Quarkus application library by adding an empty `META-INF/beans.xml` file.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this #3545 lands, we may need a link to it here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the cited PR has landed, maybe we can add a reference to How to Generate a Jandex Index section?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes please do that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. If the phrasing is OK I can update the hibernate guide with the same one.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gsmet is the new phrasing OK? Should I update the hibernate guide with it ?

@loicmathieu loicmathieu force-pushed the features/mongo_panache branch 3 times, most recently from 67317cf to 8db1717 Compare August 26, 2019 09:01
@loicmathieu
Copy link
Contributor Author

@machi1990 can you review your conversations and close them if everything is OK for you

@geoand
Copy link
Contributor

geoand commented Aug 27, 2019

I think the PR needs to be rebased onto master to ensure the proper CI configuration for the native tests (now that #3719 has been merged)

@loicmathieu loicmathieu force-pushed the features/mongo_panache branch from 8db1717 to 0dbc935 Compare August 27, 2019 12:13
@loicmathieu
Copy link
Contributor Author

@geoand I rebase onto master, wait & see :)

@geoand
Copy link
Contributor

geoand commented Aug 27, 2019

Cool!

@loicmathieu loicmathieu force-pushed the features/mongo_panache branch from bc7ae00 to c8edaad Compare August 28, 2019 09:27
Copy link
Member

@gsmet gsmet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't reviewed the code but I have made a first pass on the documentation.

docs/src/main/asciidoc/mongodb-panache-guide.adoc Outdated Show resolved Hide resolved
docs/src/main/asciidoc/mongodb-panache-guide.adoc Outdated Show resolved Hide resolved
docs/src/main/asciidoc/mongodb-panache-guide.adoc Outdated Show resolved Hide resolved
docs/src/main/asciidoc/mongodb-panache-guide.adoc Outdated Show resolved Hide resolved
docs/src/main/asciidoc/mongodb-panache-guide.adoc Outdated Show resolved Hide resolved
docs/src/main/asciidoc/mongodb-panache-guide.adoc Outdated Show resolved Hide resolved
return JsonbBuilder.create(config);
}

}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems a bit unfortunate that you have to manually add this for the default id behavior.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, it's on the TODO list on the description of the PR :)

But the discussion about how to know if an extension is included needs to more forward because today there is no way to know if JsonB is included (not all extension provides a Capabilities - in fact only Arc and Narayana, so therer is currently no way to know that an extension is included).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, just add a capability if you need one. You can do that in separate PRs, we will merge them.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, will do

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In fact, it's more complex that I first thought ...

Moreover the PR #3553, if merged, will change the way serializers/deserializers will be managed.
For the moment, I will update the TODO list with to do it later

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gsmet both JSON-B and Jackson serializer/deserializer are automatically registered if resteasy-jsonb or resteasy-jackson capability is present.

When changing one of these two library we need to carefully review this as it may creates incompabilities (for example an impact analyses needs to be done for PR #3553, and if we add some customization to Jackson ObjectMapper we need to port them to panache-mongo or change the way the ObjectMapper is created).

Fo this to work, I had to create a resteasy-common-spi module.

we would never do something like that for regular objects in the Object Oriented architecture, where state and methods are in the same class. Moreover, this requires two classes per entity, and requires injection of the DAO or Repository where you need to do entity operations, which breaks your edit flow and requires you to get out of the code you're writing to set up an injection point before coming back to use it.
- MongoDB queries are super powerful, but overly verbose for common operations, requiring you to write queries even
when you don't need all the parts.
- MongoDB queries are JSON based, so you will need some String manipulation or using the `Document` type and it will needs a lot of boilerplate code.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the first reasons fly for MongoDB. They are more related to traditional JPA development than to the MongoDB API.

Copy link
Contributor Author

@loicmathieu loicmathieu Aug 28, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can rephrase to

Even if MongoDB queries are not too verbose, PanacheQL go one step further by allowing to not write any queries for basic use cases.

This is a reference to find("name", name) shortcut query.

docs/src/main/asciidoc/mongodb-panache-guide.adoc Outdated Show resolved Hide resolved

MongoDB with Panacge relies on compile-time bytecode enhancements to your entities.
If you define your entities in the same project where you build your Quarkus application, everything will work fine.
If the entities come from external projects or jars, you can make sure that your jar is treated like a Quarkus application library by adding an empty `META-INF/beans.xml` file.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes please do that.

@loicmathieu loicmathieu force-pushed the features/mongo_panache branch 4 times, most recently from 81d9123 to de8b33d Compare September 2, 2019 13:10
Copy link
Member

@FroMage FroMage left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry my review took so long, but that was a lot of code. Overall you know this is a great PR and we want to merge it ASAP. I added change requests, but those are pretty minor points.

Thanks a lot!

build-parent/pom.xml Outdated Show resolved Hide resolved
devtools/common/src/main/filtered/extensions.json Outdated Show resolved Hide resolved
docs/src/main/asciidoc/index.adoc Outdated Show resolved Hide resolved
@@ -47,6 +47,7 @@ include::quarkus-intro.adoc[tag=intro]
* link:kogito-guide.html[Using Kogito (business automation with processes and rules)]
* link:oauth2-guide.html[Using OAuth2 RBAC]
* link:tika-guide.html[Using Apache Tika]
* link:mongodb-panache-guid.adoc[Simplified MongoDB with Panache]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was going to suggest to put it next to the mongo guide, but it's not in the list :( Can you add it just before yours?
Also we need to figure out how to call the guide, because the current one is mongo-guide.adoc and yours uses mongodb. Given that the extension is called mongodb I suggest we rename the current guide. Agreed @cescoffier ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I add it but I didn't rename it as it implies changing the links on the website and some coordination for the release. @gsmet is it worth it to rename the guide file name mongo-guide.adoc to mongodb-guide.adoc ?

docs/src/main/asciidoc/mongodb-panache-guide.adoc Outdated Show resolved Hide resolved
extensions/panache/pom.xml Show resolved Hide resolved
@loicmathieu loicmathieu force-pushed the features/mongo_panache branch 2 times, most recently from 2f6d94a to a753d02 Compare September 6, 2019 14:34
@loicmathieu
Copy link
Contributor Author

Thanks @FroMage for your review.

For me, there is two points that needs to be solved before being able to merge:

  • Is the current project() implementation that returns a PanacheQuery OK ?
  • Do we keep persist()/insert()/update(), or persist()/update() or ???. What I understood from @emmanuelbernard is that persist() sould be enought with an upsert semantic.

I need guidance for these two points, my opinion is to keep only persist() with it's current implementation (so removing all the insert and update methods) and to keep the current project() as is. We can later propose a project(Class) method that will allow to pass a projection interface on it.

I have stilll some tests to do (jackson and documentation among others), but for the other open points I prefere to open followup issues (as answered in my comments) when this PR will be merged as it's already a big PR opened since a long time :)

@emmanuelbernard
Copy link
Member

emmanuelbernard commented Sep 18, 2019 via email

@FroMage
Copy link
Member

FroMage commented Sep 18, 2019

I can see Stef hates the term upsert but that’s an industry accepted one. persistOrUpdate is cool for me as well and I guess we could implement it on data store that don’t support proper upserts.

I don't hate upsert, I said it shows too much how our persist should have been called insert and it's too late to change that.

@FroMage
Copy link
Member

FroMage commented Sep 18, 2019

@loicmathieu you're good to go on my end.

The native test failure doesn't seem related. Did you rebase on the latest master?

@loicmathieu loicmathieu force-pushed the features/mongo_panache branch from 703528b to d48d173 Compare September 18, 2019 09:55
@loicmathieu
Copy link
Contributor Author

@FroMage thanks a lot :)
I rebased onto master, let's wait for the CI.

@loicmathieu
Copy link
Contributor Author

@gsmet features are OK and @FroMage approve it. So it's ready to go :)

@FroMage
Copy link
Member

FroMage commented Sep 18, 2019

Hah, it passed CI.

Copy link
Member

@gsmet gsmet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a bunch of comments and questions.

I would appreciate additional commit(s) for the updates, then I think we should squash everything into only one atomic commit.


import io.quarkus.test.junit.SubstrateTest;

@DisabledOnOs(OS.WINDOWS)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need a comment describing why it's not run on Windows.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It nows run again on Windows


@Container
private static final GenericContainer MONGO = new FixedHostPortGenericContainer("mongo:4.0")
.withFixedExposedPort(27018, 27017);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason why we use test containers? We already use them for the other MongoDB extension?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I use test containers because mongodb-embedded that was use in the mongodb-client extension causes issue on my laptop (need to run the test as root). Other extension would also want to use test containers because it's very conveniant (Vault extension), there is an issue opened about it.
IMHO leveraging test containers can be very conveniant for Quarkus integration tests (for example to test different DBs).
But now that I finish the implementation, I can switch back to embeded MongoDB as it is used by mongodb-client extension.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I go back to flapddodle for embedded mongo to be able to run in Windows

import io.restassured.parsing.Parser;
import io.restassured.response.Response;

@DisabledOnOs(OS.WINDOWS)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need a comment explaining why Windows is not supported.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It now runs again on Windows

quarkus.mongodb.write-concern.journal=false
quarkus.mongodb.database=books

quarkus.log.category."io.quarkus.mongodb.panache.runtime".level=DEBUG
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would remove that to avoid having the build being too verbose.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Log DEBUG on panache mongo only allow to trace the queries send to MongoDB. For the current integration test it adds 10 lines to the buid.
I found it very convenient, is it OK to keep it ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a general rule, I would prefer to have it commented out but I suppose we can live with 10 more lines :).

</configuration>
</plugin>
<plugin>
<groupId>${project.groupId}</groupId>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to make that one io.quarkus now or you will have a warning.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


== Transactions

MongoDB offers ACID transactions since version 4.0. MongoDB with Panache doesn't provide support for them.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe make it a warning?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
I hope it can be done soon as Narayana should be able to provide support for it

docs/src/main/asciidoc/mongodb-panache-guide.adoc Outdated Show resolved Hide resolved
docs/src/main/asciidoc/mongodb-panache-guide.adoc Outdated Show resolved Hide resolved
docs/src/main/asciidoc/mongodb-panache-guide.adoc Outdated Show resolved Hide resolved
@loicmathieu loicmathieu force-pushed the features/mongo_panache branch 3 times, most recently from e0c8442 to 4bd0d3b Compare September 23, 2019 11:16
@loicmathieu
Copy link
Contributor Author

loicmathieu commented Sep 23, 2019

@gsmet I finally found the issue with this PR, I made a bad merge ...
Now the tests passes.
If eveything is OK on your side I can squash everything on one commit after the CI pass and ... you will be able to merge it :)

@loicmathieu loicmathieu force-pushed the features/mongo_panache branch from d561d4a to 787db9e Compare September 23, 2019 14:01
@gsmet
Copy link
Member

gsmet commented Sep 24, 2019

@loicmathieu yes please squash everything and, unfortunately, we have a conflict now.

I'll try to merge it as soon as CI is green.

@gsmet gsmet added this to the 0.24.0 milestone Sep 24, 2019
@loicmathieu loicmathieu force-pushed the features/mongo_panache branch from 787db9e to 1946498 Compare September 24, 2019 13:17
@loicmathieu
Copy link
Contributor Author

@gsmet I rebased, fixed the merge issue (conflicted build capabilities), launched the test and squashed everything.
When the CI will be green again we should be good to go ;)

@machi1990
Copy link
Member

@gsmet I rebased, fixed the merge issue (conflicted build capabilities), launched the test and squashed everything.
When the CI will be green again we should be good to go ;)

Excited to see this in. Thanks @loicmathieu.

@loicmathieu
Copy link
Contributor Author

@machi1990 as it is a big PR, testing and feedback will be appreciate :)

@loicmathieu
Copy link
Contributor Author

@gsmet: all green !!!

@gsmet gsmet merged commit b347c6d into quarkusio:master Sep 24, 2019
@gsmet
Copy link
Member

gsmet commented Sep 24, 2019

Merged! Congrats to you and Stéphane for this big achievement.

@loicmathieu
Copy link
Contributor Author

Thanks for all you guys that worked with me on this one: @FroMage, @emmanuelbernard and @gsmet it's been a long and very interresting journey :)

@gsmet gsmet changed the title Extension: MongoDB with Panache Add MongoDB with Panache extension Oct 10, 2019
@loicmathieu loicmathieu deleted the features/mongo_panache branch December 17, 2019 12:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants