Huge Int value in upsert causes: "PANIC: called Option::unwrap() on a None value in query-engine/core/src/query_document/parser.rs:250:87"

[elie222]

UPDATE: I can reproduce this error consistently.
It happens after sending quite high volume to Prisma. Nested upserts. It only starts to show after a few minutes.
Happening both locally and on our deployed servers.

Hi Prisma Team! My Prisma Client just crashed. This is the report:

Versions

Name	Version
Node	v14.15.3
OS	darwin
Prisma Client	3.0.2
Query Engine	0.1.0
Database	postgresql

What happened:
I was sending a high volume of upserts to Prisma. I can send the schema in private if needed.

Logs

 prisma:client:fetcher how you used Prisma Client in the issue. 
  prisma:client:fetcher 
  prisma:client:fetcher     at LibraryEngine.logger (project/db/node_modules/@prisma/client/runtime/index.js:25007:32)
  prisma:client:fetcher     at project/db/node_modules/@prisma/client/runtime/index.js:24977:36  
  prisma:client:libraryEngine sending request, this.libraryStarted: true  

Client Snippet

// PLEASE FILL YOUR CODE SNIPPET HERE

Schema

I can send the schema in private if needed.

// PLEASE ADD YOUR SCHEMA HERE IF POSSIBLE

Prisma Engine Query

{"X":{}}

[janpio]

Can you reproduce this consistently? Even more interesting, can you build a reproduction of it in an isolated project? I know we get these reports sometimes, but we could never reproduce them (reliably) ourselves and so are having a really hard time pinpointing what is going on.

[yassinebridi]

@janpio That is the issue, it's really hard to reproduce it, it happens sometimes, randomly, even if you have a workaround for one query, another query randomly does this.
#9795

[elie222]

I can reproduce it. If I send a lot of queries it will happen after a few minutes. Happy to give repo access to the Prisma team if it helps you debug it properly.

[elie222]

Any chance this will work better if I use queryRaw or it won't make any difference?
Other option is that I move this piece to another Postgres library.

[elie222]

Latest update:

• I'm setting a wait of 500ms between my upserts. I'm hopeful this has solved the issue. Possible I will find out it hasn't in an hour or two. 200ms gaps wasn't enough.
• Happy to share repo access in private if it helps fix this bug. Or even walk through with a team member.

[janpio]

• Happy to share repo access in private if it helps fix this bug.

That would be nice. Can you please invite @janpio and @pantharshit00? Thanks.

Any chance this will work better if I use queryRaw or it won't make any difference?

That might be interesting to test actually. "Just" send the same queries the current Prisma Client call creates via queryRaw and see if it also triggers this problem. Actually might help us pinpoint the problem here.

I am really happy you found a workaround, even though it adds half a second delay between each upsert - I hope that is acceptable to you for now. We will look into this of course, especially as there are also other, related issues. Hope we can figure this out soon.

[elie222]

So the workaround isn't perfect. Even with 500ms break it still happens but a lot less than before.

Was actually trying to reproduce the problem today locally but couldn't get it to crash locally (probably would have eventually if I waited long enough). I'll work on a demo repo and see if I can stress Prisma enough to resurface the bug on a consistent basis.

[janpio]

"Kinda consistent" is also fine, we just need something we can get to run in under half an hour with instructions on how to hammer it to probably cause the problem sooner or later.

[elie222]

In response to:
#9795 (comment)

What is your Prisma version (prisma -v)?

prisma                  : 3.4.2
@prisma/client          : 3.4.2
Current platform        : darwin
Query Engine (Node-API) : libquery-engine 57771c0558568c7d08bd34c7248af5244ae16bd9 (at node_modules/@prisma/engines/libquery_engine-darwin.dylib.node)
Migration Engine        : migration-engine-cli 57771c0558568c7d08bd34c7248af5244ae16bd9 (at node_modules/@prisma/engines/migration-engine-darwin)
Introspection Engine    : introspection-core 57771c0558568c7d08bd34c7248af5244ae16bd9 (at node_modules/@prisma/engines/introspection-engine-darwin)
Format Binary           : prisma-fmt 57771c0558568c7d08bd34c7248af5244ae16bd9 (at node_modules/@prisma/engines/prisma-fmt-darwin)
Default Engines Hash    : 57771c0558568c7d08bd34c7248af5244ae16bd9
Studio                  : 0.438.0

What OS? What Node version? What database and version?

MacOS but also on Linux in production.
v14.15.3
PostgreSQL 13

What is similar in the Prisma Client queries you are running?

Quite heavily nested but hard to pinpoint. A lot of upserts one after the other.

And most important: Does anyone have a (kinda) reliable reproduction of what is happening?

Yes, I can't reproduce in a fresh repo so will give access to our main one even though the set up takes more effort.

Will hopefully update with access in the next hour.

[elie222]

Well I'm struggling to reproduce now. Which I guess is a good thing 🤷‍♂️
Although I can see it in my logs 62 times in the past few days :( I can't get it back to happening every few minutes consistently like it was before.

[elie222]

Another update:
I can confirm it's still happening. But after around 500k upserts it happened only 4 times (although part of this is because we use a retry strategy and requeue in RabbitMQ). It's possible other changes on our end also improved things. One changed was updating Prisma from 3.0.2 to the latest 3.4.2 but there were others too.

If I can find a reproduction in the future I will share again. Always happy to add to the repo if you'd like to take a look around.

[pantharshit00]

Yes, if you can share a reproduction, that would be great. Otherwise, it would be very hard to triage.

[elie222]

This is still a major issue for us. If it helps get the ball rolling I can add to our GitHub repo and share the Sentry errors too.

I've tried to create simple and complex reproductions with no luck but we get a lot of errors for this issue consistently.

[elie222]

The code in question where the PANIC error can happen:

export async function upsertEvent(myId: number, event: Prisma.EventCreateInput) {
  try {
    return await prisma.event.upsert({
      where: { myId },
      create: event,
      update: event,
    })

event is quite deep and depends on multiple collections. I can share in private if helpful.

[pantharshit00]

@elie222 Share your reproduction to schemas@prisma.io privately. Then we can have a look and try to understand what is happening here.

[elie222]

@elie222 Share your reproduction to schemas@prisma.io privately. Then we can have a look and try to understand what is happening here.

Will do.

Another update:
This bug only happens with nested upserts. We removed the nesting from our upserts and it hasn't happened at all now.

[enzojavier-desimone]

Same issue here when taking data from an API and doing a createMany, I was casting some number string to integer and a value was greater than the supported by the Int schema. Hard to debug because the error do not give any information.

[elie222]

Same issue here when taking data from an API and doing a createMany, I was casting some number string to integer and a value was greater than the supported by the Int schema. Hard to debug because the error do not give any information.

Interesting. There's a chance that that was the issue for us too. We never found the actual cause of it just that unnesting fixed it.

[janpio]

Can you turn that into a reproduction @enzojavier-desimone?

[gleuch]

Same issue here when taking data from an API and doing a createMany, I was casting some number string to integer and a value was greater than the supported by the Int schema. Hard to debug because the error do not give any information.

Ditto. It took some debugging to narrow down to the specific input being passed into an upsert() call that causes this error. I'm able to trigger Option::unwrap() PANIC if an integer greater than Postgres's upper bound for Int is used in either create or update inputs options.

[janpio]

Same question to you, can you provide a minimal reproduction of this @gleuch? That would be amazing.
Then the others in this issue can also chime in if this could apply to their data as well.

[gleuch]

@janpio Example: https://github.com/gleuch/prisma-panic-example

[pantharshit00]

Thanks for the reproduction @gleuch. I can confirm this issue now. We need to properly handle these panics.

[janpio]

Everyone else who posted in this issue (@elie222 @yassinebridi @enzojavier-desimone), can you take a look at https://github.com/gleuch/prisma-panic-example/blob/main/index.js and confirm this could also be causing your issue?

[enzojavier-desimone]

yes, exactly same issue in my case

[elie222]

It’s possible that was causing the issue. I know that removing nesting fixed it for us though. Maybe it just made the error less obscure. But panic went away after unnesting. Best, Eliezer

…

On Tue, Mar 22 2022 at 3:45 PM, EJDS < ***@***.*** > wrote: yes, exactly same issue in my case — Reply to this email directly, view it on GitHub ( #10049 (comment) ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AAXSQXZVF3FHVRD44X2AGIDVBHFJNANCNFSM5HFLA6IA ). You are receiving this because you were mentioned. Message ID: <prisma/prisma/issues/10049/1075200380 @ github. com>

[dpetrick]

3.15 will ship a change that returns a structured error for over or underflowing integers instead of a panic.