Support sparse unique indexes

[janpio]

During the backend meeting it came up that we might want to test "sparse unique indexes". Backend team (probably @sorenbs) has more context.

[pantharshit00]

I need more context indeed for this. I want to know whether this refers to multi column indexes or many uniques index on fields of the same model.

[janpio]

My understanding:
"sparse unique indexes" are its own concept: https://docs.mongodb.com/manual/core/index-sparse/ So it is a unique index on a field, where many instances of NULL are allowed for example.

@sorenbs might be able to provide more context, he mentioned it.

[cotterjd]

Is sparse unique indexes still on the agenda? I would really like to see that feature in Prisma.

[HommeSauvage]

This is a very useful feature where a user can have either an email address or phone number as login and one of the 2 can be null. We still need a @unique index on these 2 fields. Currently, I add the sparse indexes manually, but it would be amazing if prisma supported them.

[mattslight]

I've also come up against this error. Sparse unique indexes in MongoDB enforce only non-null values to be unique which is a very handy feature. If i can help with this feature dev please let me know how

Screen.Recording.2022-09-15.at.11.10.09.PM.mov

Edit: If I try and push the schema without the unique clause I get an error telling me one-to-one relations must use unique fields. So it's a bit of a catch 22 situation.

Error: P1012

error: Error parsing attribute "@relation": A one-to-one relation must use unique fields on the defining side. Either add an `@unique` attribute to the field `driverId`, or change the relation to one-to-many.
  -->  schema.prisma:81
   | 
80 |   name     String  @unique
81 |   driver   User?   @relation(fields: [driverId], references: [id])
82 |   driverId String? @db.ObjectId

[mattslight]

suggest to add the feature using the keyword @sparse in the schema eg:

model Candidate {
  id               String    @id @default(auto()) @map("_id") @db.ObjectId
  firstName        String?
  lastName         String?
  email            String?   @unique @sparse
}

I could take a look at a PR if agreed? @janpio @sorenbs

[janpio]

We will want to design this feature in a bigger context, and unfortuantely we are currently not set up great to accept contributions with our distributed codebase in Prisma Client and Prisma Engines.

We are looking into adding some index options soon, so maybe this could be one of them.

Can a field that has a sparse index also have other indexes (unique or normal)?
Seems yes since Mongo 5: https://www.mongodb.com/docs/manual/core/index-sparse/#sparse-and-non-sparse-unique-indexes

What about this note from Mongo docs:

Changed in version 3.2: Starting in MongoDB 3.2, MongoDB provides the option to create partial indexes. Partial indexes offer a superset of the functionality of sparse indexes. If you are using MongoDB 3.2 or later, partial indexes should be preferred over sparse indexes.

? We have a request for partial indexes already: #3076

[mattslight]

Support for either would be great 😊 Implementing a clean method for defining a partial index in the prisma schema would be harder than simply adding a flag to define an index as @Spare (for partial it would also be needed to have a mechanism to specify the filter, and be more complex in your db push/pull commands). If I can help in any way please let me know.

…

On Wed, 28 Sep 2022 at 22:58, Jan Piotrowski ***@***.***> wrote: We will want to design this feature in a bigger context, and unfortuantely we are currently not set up great to accept contributions with our distributed codebase in Prisma Client and Prisma Engines. We are looking into adding some index options soon, so maybe this could be one of them. Can a field that has a sparse index also have other indexes (unique or normal)? Seems yes since Mongo 5: https://www.mongodb.com/docs/manual/core/index-sparse/#sparse-and-non-sparse-unique-indexes What about this note from Mongo docs: Changed in version 3.2: Starting in MongoDB 3.2, MongoDB provides the option to create partial indexes <https://www.mongodb.com/docs/manual/core/index-partial/#std-label-index-type-partial>. Partial indexes offer a superset of the functionality of sparse indexes. If you are using MongoDB 3.2 or later, partial indexes <https://www.mongodb.com/docs/manual/core/index-partial/#std-label-index-type-partial> should be preferred over sparse indexes. ? We have a request for partial indexes already: #3076 <#3076> — Reply to this email directly, view it on GitHub <#3419 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIGDZ6EUEY6U27MO7UGVLLWASINBANCNFSM4QJ4ODHQ> . You are receiving this because you commented.Message ID: ***@***.***>

[janpio]

Right now the biggest help would be in understanding exactly how sparse works.

From reading the docs you linked to it seems that it exists for both unique and non unique indexes, which points to an additional configuration type for @unique, @@unique and @@index. But at the same time you can have both a sparse and a non sparse variant at the same time - which would clash with how we currently define at least @unique. There your @sparse suggestions might be better - but of course that you would need to differentiate between unique and index in there also seems wrong.

Am I missing something else?

[mattslight]

This is where NoSQL and SQL differ slightly. In SQL models a row must fill all values, any values that are 'empty' are set to NULL. There is no concept of a missing/undefined value. There is either a value or its NULL. By contrast in NOSQL a value can: 1) have a value 2) be null 3) not be provided / not exist (undefined) A unique sparse index only enforces the index for the documents with the value. Partial is similar but only enforces the uniqueness for the documents where the field value matches the filter. Hope that helps? https://www.mongodb.com/docs/manual/core/index-sparse/ I think support for sparse would be relatively straightforward in prisma, simply parsing @Sparse in the schema for adapters that support it and creating a key in the db with the sparse flag set. DB Pull would be recognising a sparse key and writing it to schema. What do you think @janpio ?

…

On Thu, 29 Sep 2022 at 00:04, Jan Piotrowski ***@***.***> wrote: Right now the biggest help would be in understanding exactly how sparse works. From reading the docs you linked to it seems that it exists for both unique and non unique indexes, which points to an additional configuration type for @unique, @@unique and @@Index. But at the same time you can have both a sparse and a non sparse variant at the same time - which would clash with how we currently define at least @unique. There your @Sparse suggestions might be better - but of course that you would need to differentiate between unique and index in there also seems wrong. Am I missing something else? — Reply to this email directly, view it on GitHub <#3419 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIGDZYPHC4UAMV7FICZLM3WASQDVANCNFSM4QJ4ODHQ> . You are receiving this because you commented.Message ID: ***@***.***>

[janpio]

As sparse indexes can be either normal indexes or unique indexes, and both can exist on the same field, I do not think it is as simple. I am also not clear if this really justifies a full new attribute, instead of becoming a modifier - although it then also requires changing the rules for those so some use cases are possible.

[mattslight]

Interesting, how would a modifier work?

…

On Thu, 29 Sept 2022 at 21:49, Jan Piotrowski ***@***.***> wrote: As sparse indexes can be either normal indexes or unique indexes, and both can exist on the same field, I do not think it is as simple. I am also not clear if this really justifies a full new attribute, instead of becoming a modifier - although it then also requires changing the rules for those so some use cases are possible. — Reply to this email directly, view it on GitHub <#3419 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIGDZ65VPFBAUQ5FVCGUJDWAXJCHANCNFSM4QJ4ODHQ> . You are receiving this because you commented.Message ID: ***@***.***>

[janpio]

I meant something like @unique(sparse: true) or similar. That would make clearer how this is just a "variant" of a normal index/constraint. And the problem with this suggestion is, that one field only accepts one @unique value right now afaik.

[mattslight]

That's a good proposal, seems a better way to go. 👌🏻

…

On Thu, 29 Sep 2022 at 22:40, Jan Piotrowski ***@***.***> wrote: I meant something like @unique(sparse: true) or similar. That would make clearer how this is just a "variant" of a normal index/constraint. — Reply to this email directly, view it on GitHub <#3419 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIGDZ3AJNCBC5BEO26GP4LWAXPC3ANCNFSM4QJ4ODHQ> . You are receiving this because you commented.Message ID: ***@***.***>

[arthurfiorette]

Yup, a paremeter seems better than another @keyword. There's anything that we can do for now to create sparse indexes?

[janpio]

You can probably just create it manually on your collection and it should work. (Of course I have no idea what running db push would do though.)

What is a bit unknown and will need some research on our side, is how this influences non MongoDB databases. This issue is from a time when Prisma had no MongoDB support, and there was some idea that we would need to do something about it. (From the issue description it is clear that I was not the one knowing what they talk about here though.)

[mattslight]

Doing a quick test: If a sparse index is set in mongodb (but not unique) `db push` removes the index. If a unique sparse index is set in mongo AND defined as unique in the prisma schema then `db push` silently ignores `Db pull` obviously doesn’t pick up sparsity So in a nutshell, currently Prisma doesn’t support sparse indexes that are not unqiue but there is a workaround for unique sparse indexes in that if you set it manually using the `<Collection>_<field>_key` index naming convention it will preserve the index (since Prisma is currently blind to sparsity). I have a question with the @unique(sparse: true) proposal, how would you ensure “non-unique sparse indexes” are preserved? Note that sparsity is separate from uniqueness.

[janpio]

@@index([...], sparse: true) would also need to exist I guess.

[mattslight]

Sounds good 👍🏻

…

On Sat, 1 Oct 2022 at 02:18, Jan Piotrowski ***@***.***> wrote: @@Index([...], sparse: true) would also need to exist I guess. — Reply to this email directly, view it on GitHub <#3419 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIGDZ6UCNHFXTJ3HE2V4CLWA5RJRANCNFSM4QJ4ODHQ> . You are receiving this because you commented.Message ID: ***@***.***>

[janpio]

I learned that sparse indexes are sometimes (when you request more documents than contained in the index) only used by queries with a hint(). That poses a challenged for implementing this in Prisma Client. The Prisma Schema definition, Migration and Introspection should be fine and straightforward - but the Client has no concept of index hints at all yet and I am not sure how an API for that could/should look like. How common is that case? Do we even need that API or can we suggest to fall back to raw queries for that?

[gustawdaniel]

@janpio I have real case from app.

User can have property, eg.: telegram_id but it it optional. On the other hand I would to see error if second user will use the same telegram_id.

Now I have two options:
a) set index and then more than one user can't have telegram_id = null // this is unacceptable
b) remove index and then manually check if it is unique instead of use index // this is not optimal

For me it would be enough if I would be able to describe this use case in schema and unblock:

#17689

Even docs about recommended method of appending custom mongo indexes to prisma schema would be useful.

---

Btw I know rust, typescript and prisma, so I can try to help with this issue if you will give me some introduction.

[lucy2329]

Hey guys, is there any update on this?

[lucasrossell]

Also experiencing this issue, which is forcing us to create one-to-many relationships that realistically should be able to be one-to-one. something like @unique(sparse: true) would be much appreciated

[boredland]

I wanted to follow the instructions to create a optional 1:1 relation, but am on MongoDB. I'd say this just doesn't work on MongoDB due to the lack of the sparse index feature. Am I wrong?

[boredland]

Another problem I face: declaring a unique index across optional fields makes them non-optional, for example in the where clause of an upsert, so I can't even work around this.

[chloehjung15]

Any update on this? I want to create a optional 1:1 relation using MongoDB and I can't do that at the moment

[andrey-hohlov]

I just started doing my first steps with Prisma and stuck with issue that is already 5 year here. Very sad

[Nik-Novak]

I just started doing my first steps with Prisma and stuck with issue that is already 5 year here. Very sad

It seems @janpio is going to finally resolve this.

[gustavoggsb]

Do we have any updates on this?

[saoudi-h]

It literally took Mikro-ORM one line of code and one day from the issue to the PR to address this problem VS ...

mikro-orm/mikro-orm#484