Panic in Query Engine with SIGABRT signal (Debian Bookworm, engineType = binary)

[jove4015]

Hi Prisma Team! My Prisma Client just crashed. This is the report:

Versions

Name	Version
Node	v18.16.0
OS	linux-arm64-openssl-3.0.x
Prisma Client	4.14.1
Query Engine	query-engine d9a4c5988f480fa576d43970d5a23641aa77bc9c
Database	undefined

Logs

prisma:client prisma.$executeRawUnsafe(SET LOCAL request.tenantId = 'org_2Ejd4e1XnLEAeH0E1GCET4jeMNq', [])
prisma:client Prisma Client call:
prisma:client prisma.$executeRawUnsafe({
  query: 'SET LOCAL request.tenantId = \'org_2Ejd4e1XnLEAeH0E1GCET4jeMNq\'',
  parameters: {
    values: '[]',
    __prismaRawParameters__: true
  }
})
prisma:client Generated request:
prisma:client mutation {
  executeRaw(
    query: "SET LOCAL request.tenantId = 'org_2Ejd4e1XnLEAeH0E1GCET4jeMNq'"
    parameters: "[]"
  )
}

prisma:client prisma.$executeRawUnsafe(SET LOCAL request.userId = 'user_2Efn1vwma0vatqFH7kNL8GtVOPi', [])
prisma:client Prisma Client call:
prisma:client prisma.$executeRawUnsafe({
  query: 'SET LOCAL request.userId = \'user_2Efn1vwma0vatqFH7kNL8GtVOPi\'',
  parameters: {
    values: '[]',
    __prismaRawParameters__: true
  }
})
prisma:client Generated request:
prisma:client mutation {
  executeRaw(
    query: "SET LOCAL request.userId = 'user_2Efn1vwma0vatqFH7kNL8GtVOPi'"
    parameters: "[]"
  )
}

prisma:client Prisma Client call:
prisma:client prisma.facility.findMany({
  where: {
    memberships: {
      some: {
        user: {
          externalId: 'user_2Efn1vwma0vatqFH7kNL8GtVOPi'
        },
        deletedAt: null
      }
    }
  },
  include: {
    location: true
  }
})
prisma:client Generated request:
prisma:client query {
  findManyFacility(where: {
    memberships: {
      some: {
        user: {
          externalId: "user_2Efn1vwma0vatqFH7kNL8GtVOPi"
        }
        deletedAt: null
      }
    }
  }) {
    id
    locationId
    licenseNumber
    address
    stateId
    tenantId
    location {
      id
      name
      description
      size
      canopySize
      locationType
      tenantId
      deletedAt
      deletedById
    }
  }
}

prisma:engine stdout 
prisma:engine stdout 
prisma:engine stdout 
prisma:engine Search for Query Engine in /usr/src/app/node_modules/.prisma/client
plusX Execution permissions of /usr/src/app/node_modules/.prisma/client/query-engine-linux-arm64-openssl-3.0.x are fine
prisma:engine stdout 
prisma:engine stdout 
prisma:engine Client Version: 4.14.1
prisma:engine Engine Version: query-engine d9a4c5988f480fa576d43970d5a23641aa77bc9c
prisma:engine Active provider: postgresql
prisma:engine stdout 
prisma:engine stderr malloc(): unaligned tcache chunk detected
prisma:engine {"error":{"name":"SocketError","code":"UND_ERR_SOCKET","socket":{"localAddress":"127.0.0.1","localPort":53574,"remoteAddress":"127.0.0.1","remotePort":43861,"remoteFamily":"IPv4","bytesWritten":756,"bytesRead":0}}}
prisma:client:request_handler {"name":"SocketError","code":"UND_ERR_SOCKET","socket":{"localAddress":"127.0.0.1","localPort":53574,"remoteAddress":"127.0.0.1","remotePort":43861,"remoteFamily":"IPv4","bytesWritten":756,"bytesRead":0}}
prisma:client:request_handler {"name":"SocketError","code":"UND_ERR_SOCKET","socket":{"localAddress":"127.0.0.1","localPort":53574,"remoteAddress":"127.0.0.1","remotePort":43861,"remoteFamily":"IPv4","bytesWritten":756,"bytesRead":0}}
prisma:client:request_handler {"name":"SocketError","code":"UND_ERR_SOCKET","socket":{"localAddress":"127.0.0.1","localPort":53574,"remoteAddress":"127.0.0.1","remotePort":43861,"remoteFamily":"IPv4","bytesWritten":756,"bytesRead":0}}
prisma:engine {"error":{"name":"SocketError","code":"UND_ERR_SOCKET","socket":{"localAddress":"127.0.0.1","localPort":53580,"remoteAddress":"127.0.0.1","remotePort":43861,"remoteFamily":"IPv4","bytesWritten":5217,"bytesRead":0}}}
prisma:client:request_handler {"name":"SocketError","code":"UND_ERR_SOCKET","socket":{"localAddress":"127.0.0.1","localPort":53580,"remoteAddress":"127.0.0.1","remotePort":43861,"remoteFamily":"IPv4","bytesWritten":5217,"bytesRead":0}}
prisma:client:request_handler {"name":"SocketError","code":"UND_ERR_SOCKET","socket":{"localAddress":"127.0.0.1","localPort":53580,"remoteAddress":"127.0.0.1","remotePort":43861,"remoteFamily":"IPv4","bytesWritten":5217,"bytesRead":0}}
prisma:client:request_handler {"name":"SocketError","code":"UND_ERR_SOCKET","socket":{"localAddress":"127.0.0.1","localPort":53580,"remoteAddress":"127.0.0.1","remotePort":43861,"remoteFamily":"IPv4","bytesWritten":5217,"bytesRead":0}}
prisma:engine {"error":{"name":"SocketError","code":"UND_ERR_SOCKET","socket":{"localAddress":"127.0.0.1","localPort":53592,"remoteAddress":"127.0.0.1","remotePort":43861,"remoteFamily":"IPv4","bytesWritten":995,"bytesRead":0}}}
prisma:client:request_handler {"name":"SocketError","code":"UND_ERR_SOCKET","socket":{"localAddress":"127.0.0.1","localPort":53592,"remoteAddress":"127.0.0.1","remotePort":43861,"remoteFamily":"IPv4","bytesWritten":995,"bytesRead":0}}
prisma:client:request_handler {"name":"SocketError","code":"UND_ERR_SOCKET","socket":{"localAddress":"127.0.0.1","localPort":53592,"remoteAddress":"127.0.0.1","remotePort":43861,"remoteFamily":"IPv4","bytesWritten":995,"bytesRead":0}}
prisma:client:request_handler {"name":"SocketError","code":"UND_ERR_SOCKET","socket":{"localAddress":"127.0.0.1","localPort":53592,"remoteAddress":"127.0.0.1","remotePort":43861,"remoteFamily":"IPv4","bytesWritten":995,"bytesRead":0}}

Client Snippet

// You can see the queries causing this error in the dump above: 
prisma.$executeRawUnsafe(SET LOCAL request.userId = 'user_2Efn1vwma0vatqFH7kNL8GtVOPi', []) 
// We wrap every transaction with these in order to facilitate RLS, and have been doing this for a long time (since client extensions were introduced).

Schema

// PLEASE ADD YOUR SCHEMA HERE IF POSSIBLE

I can provide this by DM if required.

Prisma Engine Query

{"X":true}

This has been happening to us all day, is happening on our master branch which passed testing yesterday without issue, and seems to happen now on various old branches that all worked fine for us before.

[janpio]

Did you upgrade Prisma between when it worked and does not anymore?
Did anything else in your project change?
Maybe on your database?

[jove4015]

No, we have been running 4.14.1 the whole time. Obviously, we write code every day, so as is true for any software development team, yes, we have made changes. There were no migrations, changes to prisma configuration, or anything else that seems relevant or significant. Nothing in our database changed, aside from the fact that people used the database.

The problem comes and goes, which is even more frustrating. Three times yesterday I had to stop all work because Prisma was in a panic loop that just wouldn't stop no matter what I did. I will add too that this is happening across multiple people's computers in multiple time zones. The one common thread seems to be that this happens on ARM machines (macbooks). It does not seem to happen in our x86 CI environments.

[joe-giunti-kiefa]

@janpio Additionally, one strange thing that happened the same day with no changes being made is that our docker image started failing to build with this error which forced us to have to do apt-get install -y netcat-traditional instead.

At face value this didn't seem related to prisma, but we thought it is suspicious that this started happening as well. We are using node:18-slim as our base image so we were suspicious that maybe something in this base image has changed that is causing problems with prisma on ARM since this issue is only happening to us locally on our ARM based macbooks vs in our x86 CI and production environments.

[jove4015]

After some more investigation, Joe discovered that node:18-slim was actually updated from debian version bullseye to bookworm a couple days ago.

We were able to restore our app to normal behavior by changing our base image to node:18-bullseye-slim.

I think this indicates there may be some problem with Prisma on bookworm / ARM, but there is no huge urgency for us to upgrade this right now. Eventually it will become an issue if not addressed but we have at least an interim solution.

[janpio]

Nice sleuthing to pinpoint it to that Debian upgrade of the underlying container of node:18-slim. That messed with a bunch of system dependencies, that cause problem for Prisma indeed - but usually in a very different (and more obvious) way. We are currently investigating how to deal with that.

The error message you are getting comes from Undici, which indicates you might be using engineType = binary in your project - is that correct? If so, is there a remaining reason for that?

[jove4015]

I had to go check my notes - we switched to engine binary back in March when we had to upgrade to Node 18 in order to support another package that started requiring it. Previously we had downgraded to Node 16 in order to avoid this bug:

#10649

But staying on node 16 wasn't an option. Someone suggested switching the engine type to binary on that PR and that worked for us. I didn't even notice that the issue was closed; however when I was troubleshooting bookworm before I did try switching back to the library engine and it did not work (ie there was still crashing) - it may have had a different error message, I'm not sure anymore. Since that issue is closed though now I will give the library engine a fresh shot!

[yepMad]

You can reproduce this on "Node.js 18 running on 64bit Amazon Linux 2023/6.0.0" using ARM architecture. As a quick solution, we migrated to 64-Bits, it worked again.

[aqrln]

Given this line in logs (and the overall description and symptoms):

prisma:engine stderr malloc(): unaligned tcache chunk detected

this is the issue that was fixed in 5.3.0-dev.42 by prisma/engine-images#75 (and will be released in a stable 5.3.0 release in a week). Please check if it fixes the issue on your side. I'll close this issue as fixed since I'm pretty confident in it, please let us know if that's not the case and we'll reopen it then.