New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MongoDB Connector #1643

Closed
marktani opened this Issue Jan 16, 2018 · 59 comments

Comments

Projects
None yet
@marktani
Contributor

marktani commented Jan 16, 2018

This feature requests serves as a central place to discuss development and progress for the MongoDB connector. Please contribute in the comments below. I will update the top comment as consensus forms.

@alexandrakl

This comment has been minimized.

alexandrakl commented Jan 17, 2018

I would like to work on this! Where is a good place to start looking? How to wrap head around the task and break it into its constituent parts? Please provide some guidelines if possible, would be much appreciated! Thanks for your work on this project, it is very cool, reminds me of RESTHeart ;)

@cbioley

This comment has been minimized.

cbioley commented Jan 17, 2018

@ookangzheng Have a look at the project README file. It says "Prisma can be used for MySQL Databases out of the box" ;)

@sorenbs

This comment has been minimized.

Member

sorenbs commented Jan 17, 2018

Hi @alexandrakl it would be great to collaborate on this!

The first step is to map out the kind of queries that are particularly well supported by Mongo as well as queries currently supported by Prisma that does not fit the document model.

Based on this we can design the API for Document-flavoured GraphQL.

Feel free to dump your thoughts in this thread so we can get the conversation going :-)

@prisma prisma deleted a comment from lolcoolkat Jan 17, 2018

@prisma prisma deleted a comment from ForsakenHarmony Jan 17, 2018

@prisma prisma deleted a comment from ookangzheng Jan 17, 2018

@GilUpstream

This comment has been minimized.

GilUpstream commented Jan 20, 2018

@alexandrakl @sorenbs I'd love to work with you all on this.

@sorenbs

This comment has been minimized.

Member

sorenbs commented Feb 6, 2018

Here's an outline of some things we need to consider:

  • align with existing Mongo-based GraphQL APIs (vulcanJs)
  • active/passive (does schema migration even make sense?)
  • mutations (can we support the powerful nested mutations? relations?)
  • queries (What are the query limitations?)
  • embedded types and relations
  • subscriptions

Take a look at #1340 for the API spec for the MySQL conenctor

@GilUpstream @alexandrakl and anyone who want to contribute. It would be great if you could start mapping out some of these questions in this thread.

@sorenbs

This comment has been minimized.

Member

sorenbs commented Feb 12, 2018

Top level fields

For each type with an id field, the following fields are generated:

Types without an id field are called embedded types and are stored as nested documents.

The id field is represented as _id in the mongo document per normal mongo convention. Prisma does not support having both _id and id fields.

The type User is used as an example throughout this document

Query

  • user returns a single user
  • users returns a list of users
  • usersConnection returns a relay style Connection of users

In addition to the generated query fields, the following field is present:

  • node returns a single node by id

field name generation

The user field name is generated by taking the type name and lowercasing the first letter

The users and usersConnection field names are generated by taking the type name and lowercasing the first letter and then using the evo-inflector library

Mutation

  • createUser create a single user
  • updateUser update a single user
  • deleteUser delete a single user
  • upsertUser update a single user if exists, otherwise create
  • updateManyUsers update multiple users
  • deleteManyUsers delete multiple users

Note: The API has no explicit mutations for managing relations. Instead relations are managed through nested mutations #1280

MongoDB has no concept of foreign key constraints and cascading deletes. Prisma ensures that relations stays valid and that cascading deletes are enforced.

Examples:

A -> B
  • Deleting B will fail if the relation is required and there is no cascading delete
  • Deleting B will succeed if the relation is required and there is cascading delete
  • Deleting B will succeed if the relation is not required and there is no cascading delete. In this case the relation field will be set to null
  • Deleting B will (obviously) succeed if the relation is not required and there is cascading delete

Note: MongoDB has no concept of transactions that span more than one document. As mutations that involve relations or mutate many nodes, involve multiple documents, it is possible that the mutation will be only partially applied if the server is stopped before the mutation completes.

Details

type Query {
  user(where: UserWhereUniqueInput): User

  users(
    where: UserWhereInput
    orderBy: UserOrderByInput
    skip: Int
    before: String
    after: String
    first: Int
    last: Int
  ): [User!]!

  usersConnection(
    where: UserWhereInput
    orderBy: UserOrderByInput
    skip: Int
    before: String
    after: String
    first: Int
    last: Int
  ): UserConnection!
}

type Mutation {
  createUser(data: UserCreateInput!): User!
  updateUser(data: UserUpdateInput!, where: UserWhereUniqueInput!): User
  deleteUser(where: UserWhereUniqueInput!): User
  upsertUser(where: UserWhereUniqueInput!, create: UserCreateInput!, update: UserUpdateInput!): User

  updateManyUsers(data: UserUpdateInput!, where: UserWhereInput!): BatchPayload
  deleteManyUsers(where: UserWhereInput!): BatchPayload
}

query fields

where

Same filter as for the SQL connector, except relational filters only work for embedded types.
See #1890
All predicates map directly to mongo:

scalar
  • field: $eq
  • field_not: $ne
  • substring matching: contains, starts_with, ends_with: $regex
  • field_lt: $lt
  • field_lte: $lte
  • field_gt: $gt
  • field_gte: $gte
  • field_in: $id
  • field_not_in: $nin
relation to embedded type
  • field_every: $all
  • field_some: $elemMatch
  • field_none: $not $elemMatch

orderBy

Same enum as for SQL connector

orderBy: CATEGORY_ASC maps to sort: { category: 1 }

skip

Integer specifying the number of documents to skip

before

id of the document to start from, going backwards: max

after

id of the document to start from: min

first

Integer limiting the number of documents to return: limit

last

Integer limiting the number of documents to return: limit

Nested Mutations

Mongo supports arbitrarily nested documents but has no understanding of relations between documents.

Prisma supports two ways to model related data:

  • relations
  • embedded types.

relations is a great fit for relational databases and Prisma implements embedded types for SQL databases by automatically managing all the tables and foreign keys required.

In the same way, embedded types is a great fit for mongos document model. Prisma implements relations by performing multiple inserts/updates or queries under the hood.

When is a type embedded?

In Prisma a type is considered to be embedded when it has no id field. Types without an id field can only ever exist when referenced from an "owning" type. The mongo connector takes advantage of this fact by using mongos native capability to nest documents. This can result in improved performance compared to SQL.

Note: MongoDB has a storage limit of 16mb per document. This limits the aggregate size of a single node with embedded types. The SQL connector does not have this limit.

Note: The SQL connector is able to change a type from embedded to normal and back again very easily because they underlying storage mechanism is the same. This is not the case for the MongoDB connector as embedded types are embedded in the parent document.

handling of relations

todo: specify how mutations and queries are handled for embedded types

scalar lists

todo: specify how scalar lists are implemented. They should be embedded. Document how each operation maps to mongos primitives. Document storage limit

@juicycleff

This comment has been minimized.

juicycleff commented Feb 13, 2018

There needs the be a common api or interface so one can switch to different connectors.

@sorenbs

This comment has been minimized.

Member

sorenbs commented Feb 14, 2018

That's a great point @juicycleff

We plan to support database specific APIs as well a a Core API. Most of the database specific APIs will have significant overlap with the core API making it easy to switch between connectors. For Mongo and SQL specifically we will be able to have almost 100% overlap :-)

@Yogu

This comment has been minimized.

Yogu commented Mar 2, 2018

If you need some inspiration for a GraphQL type system of a document database, have a look at cruddl. It's a TypeScript library that serves a GraphQL API similar the one of prisma, but it uses ArangoDB as database (might also be interesting for #1645). It is built to embrace the possibilities of document databasees like embedding. We invented four kinds of object types to support different use cases of embedding and relating objects - like when ids should be auto-generated and whether omitted fields in update mutations should remove them or leave them unchanged.

Considering the similarities between cruddl and prisma, we would look forward to work towards a common "standard" GraphQL API for document databases.

@DavyBello

This comment has been minimized.

DavyBello commented Mar 8, 2018

Have a look at graphql-compose it's a toolkit for construction of complex GraphQL schemas. Provides bunch of plugins for type generation. Allows to build your own plugins/helper-functions for type construction and modification and graphql-compose-mongoose a Mongoose model converter to GraphQL types with resolvers. They also have type generator plugins for elasticsearch and aws

@duytai

This comment has been minimized.

duytai commented Mar 13, 2018

@sorenbs in my opinion, we should consider a model without embedded type to remove the complexity of mongodb connector. As far as I know:

  1. Current generated Prisma api does not support embedded type
  2. We just implement connector to support current Prisma API
@sorenbs

This comment has been minimized.

Member

sorenbs commented Mar 13, 2018

@duytai i agree.

The term embedded type is used here only to describe a certain way to lay out your data. Prismas SQL connector has no understanding of that term and neither will the Mongo connector. The final api will be almost identical apart from less powerful filters.

@DakshMiglani

This comment has been minimized.

DakshMiglani commented Mar 20, 2018

any updates on this?

@ojongerius

This comment has been minimized.

ojongerius commented Apr 6, 2018

Is someone working on this at the moment?

@marktani

This comment has been minimized.

Contributor

marktani commented Apr 6, 2018

Yes, we are 🙂

We plan to start a preview for the MongoDB connector later this month. Please reach out to me in Slack (@nilan) if you're interested in participating.

@lfades

This comment has been minimized.

lfades commented Apr 11, 2018

An update for this note:

Note: MongoDB has no concept of transactions that span more than one document. As mutations that involve relations or mutate many nodes, involve multiple documents, it is possible that the mutation will be only partially applied if the server is stopped before the mutation completes.

Transactions will be supported in 4.0

@lfades

This comment has been minimized.

lfades commented Apr 11, 2018

todo: specify how mutations and queries are handled for embedded types

In Mongodb joins/relations are not recommended for cases where the related documents don't change often, better said, we have a lot more of queries than mutations, for example, suppose that our User has a books fields which can be a one-to-many relation or just an embedded field with the books instead, A Book will probably never change so we don't need a one-to-many relation here.

I believe embedded types can be handled in the same way Prisma handles relations, but every embedded document should contain an id (and the operations for update/remove/insert will be using different mongodb operators). In most cases embedded documents will contain an id, or something that will allow them to be identified inside the array (like the index), if the embedded document is just an object and not an array, all operations will be for that document.

@nhuesmann

This comment has been minimized.

nhuesmann commented Apr 17, 2018

@sorenbs Have you considered including any mongoose-specific abstractions like virtuals and references/reference population (i.e. using references as an alternative to embedded documents)? I've really liked utilizing mongoose's populate operator to keep document sizes to a predictable limit, avoid arrays growing without bound, and normalize the data as much as possible. Virtuals/getters and setters are more of a nice-to-have but have still proven to be handy in certain situations, acting like Redux selectors.

I know the purpose of the mongodb connector isn't to be a mongoosejs connector, however I anticipate a large number of developers having likely interacted with mongoosejs as much if not more than the mongodb API directly, so having some familiar features might make the adoption process easier.

Also, do you plan to support GeoJSON object types for geospatial queries? They've been indispensable in a few projects and I'd really hope for them to be supported.

@heysailor

This comment has been minimized.

heysailor commented Apr 18, 2018

+1 for GeoJSON. My primary use case for mongo is to enable efficient geospatial queries. These complex queries can be done with mongo directly, but the ability to resolve GeoJSON data through Prisma would be very handy.

@do4gr

This comment has been minimized.

Member

do4gr commented Oct 5, 2018

We just merged Join-Relations (relations between models in different collections) into the Mongo Connector preview. If you want to try it out, I’ve updated the preview page. As always: feedback is very welcome. https://github.com/prisma/Mongo-Connector-Preview/blob/master/README.md (ed

@do4gr

This comment has been minimized.

Member

do4gr commented Oct 11, 2018

This is a proposal for the directives to be introduced for the Mongo connector.

We are already using connector specific directives in the Postgres Passive connector. https://www.prisma.io/docs/data-model-and-migrations/introspection-mapping-to-existing-db-soi1/#overview

Postgres Directive Active Passive
@pgTable(name: "X") opt opt can be provided to rename tables
@pgColumn(column: "X") opt opt can be provided to rename columns
@pgRelation(column: "X") opt req has to be provided to identify fields with relation ids
@pgRelationTable(table: "X") - req identifies a relation table
Mongo Directive
@mongoCollection(name: "X") opt opt can be provided to rename collections
@mongoField(field:"X") opt opt can be provided to rename fields
@mongoRelation req req explicitly determine the side that stores inline relation ids
@mongoRelation(field:"X") opt opt can be provided to rename inline relations and embedded types

The field argument on @mongoRelation is optional. If it is not provided Prisma will use the exact fieldName from the data model to look for as a field in the collection.

Active Connector

Bi-Directional Join Relation

type Parent{
    name: String
    child: Child @mongoRelation    # required! this side has inline ids, autogenerated field names
}

type Child{
    name: String
    parent: Parent
}

Uni-Directional Join Relation

type Parent{
    name: String
    child: Child     # optional. no directive necessary since there is only one relationField
}

type Child{
    name: String
 }

Embedded Type Relation

type Parent{
    name: String
    child: Child     # optional. embedded under fieldname
}

type Child @embedded {
    name: String
}

Embedded Type To Other Non-Embedded Type Relation

type Parent{
    name: String
    child: Child     # optional. stored under fieldname
}

type Friend{
    name: String
}

type Child @embedded {
    name: String
    friend: Friend  # optional. stored under fieldname
}

Passive Connector

Bi-Directional Join Relation

type Parent{
    name: String
    child: Child @mongoRelation    # required! field which stores ids
}

type Child{
    name: String
    parent: Parent
}

Uni-Directional Join Relation

type Parent{
    name: String
    child: Child    # optional. field which stores ids
}

type Child{
    name: String
 }

Embedded Type Relation

type Parent{
    name: String
    child: Child      # optional. field that stores inline ids
}

type Child @embedded {
    name: String
}

Embedded Type To Other Non-Embedded Type Relation

type Parent{
    name: String
    child: Child     # optional. field that stores inline ids
}

type Friend{
    name: String
}

type Child @embedded {
    name: String
    friend: Friend   # optional. field that stores inline ids
}

Open Questions:

  • Always use name instead of field/column/name for the optional directive argument?
  • Is distinction between Mongo / Postgres directives necessary?
  • Are relation tables in Mongo a thing?

@prisma prisma deleted a comment from iamrommel Oct 11, 2018

@prisma prisma deleted a comment from bmayen Oct 11, 2018

@prisma prisma deleted a comment from lfades Oct 11, 2018

@prisma prisma deleted a comment from iamrommel Oct 11, 2018

@prisma prisma deleted a comment from jay-jlm Oct 11, 2018

@prisma prisma deleted a comment from iamrommel Oct 11, 2018

@sorenbs

This comment has been minimized.

Member

sorenbs commented Oct 11, 2018

While I appreciate the passion displayed by the community in the comments here, this thread is a place to discuss design decisions for the upcoming MongoDB connector. I have deleted a number of comments that were off-topic. If you are interested in these other topics, feel free to reach out to me in slack or open a conversation in the forum.

@prisma prisma unlocked this conversation Oct 11, 2018

@agolomoodysaada

This comment has been minimized.

agolomoodysaada commented Oct 11, 2018

Take a look at mongodb references. Equivalent to foreign keys in pgsql.

It was also previously mentioned that multi-document transactions is supported by mongo 4.0.

@sorenbs

This comment has been minimized.

Member

sorenbs commented Oct 11, 2018

Thanks @agolomoodysaada!

We will most likely support multi-document transactions in the future. Keep in mind that this is a feature that currently does not support sharding, and even when implemented in MongoDB 4.2 will incur a performance penalty that might be too high for many applications.

References similarly does not support sharding, and I don't know of any public plans to support it in the future.

@sorenbs

This comment has been minimized.

Member

sorenbs commented Oct 17, 2018

As we are getting closer to the first beta release of the Mongo connector, we are looking for early adopters to take it for a spin.

If you are planning to build an application using Prisma and MongoDB, now is a good time to get involved to make sure the MongoDB connector covers all your needs.

If you are interested, please ping @do4gr here or in the Prisma slack 🙏

@agolomoodysaada

This comment has been minimized.

agolomoodysaada commented Oct 17, 2018

@do4gr , I'm interested in testing it out... any docs I can follow?

@do4gr

This comment has been minimized.

Member

do4gr commented Oct 17, 2018

The documentation how to set up Prisma to use Mongo and how to use the new features can be found here: https://github.com/prisma/Mongo-Connector-Preview

@houmark

This comment has been minimized.

houmark commented Oct 18, 2018

Also interested in testing out the MongoDB connector.

@amilner42

This comment has been minimized.

amilner42 commented Nov 7, 2018

To those trying out this new connector, it would be great to leave links if your work is open source.

Hoping to try this myself soon.

@nhuesmann

This comment has been minimized.

nhuesmann commented Nov 20, 2018

@do4gr I'd like to test the MongoDB connector and hope to use it in production on a project I'm currently building.

@do4gr

This comment has been minimized.

Member

do4gr commented Nov 20, 2018

The current documentation how to use it can be found here: https://github.com/prisma/Mongo-Connector-Preview

We'll release full documentation in the next days.

@schickling

This comment has been minimized.

Member

schickling commented Dec 4, 2018

@vibhanshu909

This comment has been minimized.

vibhanshu909 commented Dec 5, 2018

I was waiting a long time for this, and I must say prisma is doing a great job, I was wondering if there is support for multi-document ACID transactions that were introduced in mongodb v4. I can see that the current documentation is using mongodb v3.6 not v4+. Thanks!

@do4gr

This comment has been minimized.

Member

do4gr commented Dec 7, 2018

Hey @vibhanshu909,

we haven't done extensive testing on the compatibility with Mongo 4.0. The core is structured in a way that we should be able to introduce transactionality with 4.0 fairly easily. The challenge is in properly testing this for correctness and performance afterwards. This is why we are still holding off on this. But I would expect us to enable this at some point.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment