Error: Command was killed with SIGSEGV (Segmentation fault): /Users/.../node_modules/prisma2/query-engine-darwin --enable_raw_queries cli --dmmf

[janpio]

Redwood was released yesterday, and we have 2 reports of segfaults in Prisma tooling:

1. prisma2 migrate causes SIGSEGV segfault migrate#364
2. [Prisma Bug] Error while generating Prisma client in Example Blog redwoodjs/redwood#215

This issue is our representation of the second one above:

This project looks great! I'm hoping to get a demo going with TinaCMS added to it for content editing.

Unfortunately, running yarn rw db up threw an error while I was trying to setup the example-blog.

Migrate database up... [completed]
Generating the Prisma client... [started]
Error: Error: Command was killed with SIGSEGV (Segmentation fault): /Users/ncphillips/Code/ncphillips/example-blog/node_modules/prisma2/query-engine-darwin --enable_raw_queries cli --dmmf
Generating the Prisma client... [failed]
→ Command failed with exit code 1: prisma2 generate
Command failed with exit code 1: prisma2 generate
✨  Done in 1.66s.

I can reliably reproduce by following the Install Steps for the Example Blog, even from a fresh clone. The only thing I changed from teh repo is I switched the datasource provider to sqlite.

| OS | macOS 10.15.3 |
| node | 12.14.1 |
| yarn | 1.15.2 |

The Redwood team also supplied some context where in their code our tooling is called here:

• https://github.com/redwoodjs/redwood/blob/master/packages/cli/src/commands/dbCommands/up.js#L14-L18
• https://github.com/redwoodjs/redwood/blob/master/packages/cli/src/lib/index.js#L160-L167
• https://github.com/redwoodjs/redwood/blob/master/packages/cli/src/commands/dbCommands/generate.js#L29-L36

[janpio]

We could reproduce internally and are now investigating and working on a fix.

[janpio]

Persisting the error logs of our internal reproduction (that only worked once unfortunately):

Error: Error: Command was killed with SIGSEGV (Segmentation fault): /Users/matthias/repos/work/delete/example-blog/node_modules/prisma2/migration-engine-darwin cli --datasource file:./dev.db --create_database
    at o (/Users/matthias/repos/work/delete/example-blog/node_modules/prisma2/build/index.js:2:638464)
    at C (/Users/matthias/repos/work/delete/example-blog/node_modules/prisma2/build/index.js:2:903458)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async Object.createDatabase (/Users/matthias/repos/work/delete/example-blog/node_modules/prisma2/build/index.js:2:106196)
    at async Object.ensureDatabaseExists (/Users/matthias/repos/work/delete/example-blog/node_modules/prisma2/build/index.js:2:2453211)
    at async LiftUp.parse (/Users/matthias/repos/work/delete/example-blog/node_modules/prisma2/build/index.js:2:993714)
    at async main (/Users/matthias/repos/work/delete/example-blog/node_modules/prisma2/build/index.js:2:1490795)
Migrate database up... [failed]
→ Command failed with exit code 1: prisma2 migrate up --experimental --create-db
Command failed with exit code 1: prisma2 migrate up --experimental --create-db
Generating the Prisma client... [started]

Version:

matthias at MacBook-Pro-6 in ~/repos/work/delete/example-blog on master [!]
$ /Users/matthias/repos/work/delete/example-blog/node_modules/prisma2/migration-engine-darwin --version
d3b0ceed5d87544b9d2decb70e08664f9047bb73

Manual execution of binary directly:

matthias at MacBook-Pro-6 in ~/repos/work/delete/example-blog on master [!]
$ /Users/matthias/repos/work/delete/example-blog/node_modules/prisma2/migration-engine-darwin cli --datasource file:./dev.db --create_database
Bus error: 10

---

And the error output from prisma/migrate#364:

 ✘  redwoodblog   master ●  yarn rw db up 
yarn run v1.19.2
$ /Users/swyx/Work/testbeds/redwoodblog/node_modules/.bin/rw db up
Migrate database up... [started]
Error: Get config Error: Command was killed with SIGSEGV (Segmentation fault): /Users/swyx/Work/testbeds/redwoodblog/node_modules/prisma2/query-engine-darwin cli --get_config /private/var/folders/r7/x4h9vkx94tz1dw96tn1lfs9h0000gn/T/efdb9944-a3b7-456f-9603-5ba9f548dbfc
Migrate database up... [failed]
→ Command failed with exit code 1: prisma2 migrate up --experimental --create-db
Command failed with exit code 1: prisma2 migrate up --experimental --create-db
Generating the Prisma client... [started]

Interesting to note here that our internal reproduction was not with query-engine-darwin but migration-engine-darwin.

[dizzyn]

I use 2.0.0-preview021 because I can not upgrade without #1777

Please, how can I be affected by the bug and will the future fix force the upgrade to preview23 please?

Thank You

[janpio]

I use 2.0.0-preview021 because I can not upgrade without #1777

Please, how can I be affected by the bug and will the future fix force the upgrade to preview23 please?

Sorry, I did not see this comment when you first posted it. Excuse the wait @dizzyn .

Can you post more information how exactly you get this error? What are you doing, what is the exact error message?

[janpio]

Ok, we have another case where the binary was broken:

• [Introspection] Operation timed out #1493 (comment)
  https://community.redwoodjs.com/t/tutorial-fetching-posts-throws-errors-fix-broken-prisma2-binary-requires-clearing-cache-prisma/162/12

Luckily @olance spent quite some time figuring stuff out - and could if it in the end.

[janpio]

An update on our investigation:

Somehow the binary is corrupted between download, unpacking, and being moved to the correct location. This is super hard and flaky to reproduce, and we have no real idea what is causing it: It might be the CDN, the download process, the unpacking, the moving, any third party or even OS software - or a combination of all of that as it seems under some circumstances multiple processes to the same thing at the same time.

To avoid that as good as possible we for now implemented a lockfile that makes sure only one process downloads and unpacks (which also makes sense as doing the same thing twice is kind of useless).

If that does not help enough, we might also implement some checksum comparison for the downloaded (or unpacked) files soon.

We are open to other suggestions and ideas.

[olance]

Hi @janpio thanks for pointing me to this issue!

I think I'm now 99% sure this was a corrupted binary that caused the issue:

• Working project on one machine and non-working one on another were using the exact same version of Prisma packages

• However, the query-engine-darwin binaries were different: there was at least one big 3MB chunk of zeroes in one of them, that was actually filled with data in the other.

• Deleting the cache (rm -rf ~/.cache/prisma) and regenerating the Prisma Client fetched a more recent version and fixed everything on the machine that kept on failing

• the faulty binary was fetched on March the 19th

• the working binary was fetched on March the 22nd

• I currently have, on the machine where things were failing initially, an engine binary in my cache whose lastModified-query-engine file says Tue, 03 Mar 2020 14:19:39 GMT, and it works. Not sure why it got downloaded along with the other binary from the 23rd of March though, but it really points to a corrupted download that got fixed by clearing the cache.

I'd really encourage implementation of a checksum indeed, that would help at least catch the issue early on if it's really a corruption problem.

I don't think crash reports will be very helpful if we're really looking at corrupted downloads, but just in case, here goes:

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       EXC_I386_GPFLT
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Segmentation fault: 11
Termination Reason:    Namespace SIGNAL, Code 0xb
Terminating Process:   exc handler [98592]

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   query-engine-darwin           	0x000000010b9d7000 alloc::slice::_$LT$impl$u20$$u5b$T$u5d$$GT$::concat::h35a9f1e00e88c910 + 80
1   query-engine-darwin           	0x000000010b3dba13 _$LT$std..future..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h7008d3eac8096ec8 + 115
2   query-engine-darwin           	0x000000010b29c22c prisma::context::PrismaContext::new::_$u7b$$u7b$closure$u7d$$u7d$::hf502954b5965bede + 940
3   query-engine-darwin           	0x000000010b27e389 std::future::poll_with_tls_context::hf4d7793409e8640a + 281
4   query-engine-darwin           	0x000000010b29ff7e prisma::main::_$u7b$$u7b$closure$u7d$$u7d$::h834229e5af9fe8cc (.llvm.2251309141205072438) + 814
5   query-engine-darwin           	0x000000010b29086d _$LT$std..future..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h03ecd98f516356aa + 45
6   query-engine-darwin           	0x000000010b2e9e3e tokio::runtime::enter::Enter::block_on::h04f3649305bad2e3 + 110
7   query-engine-darwin           	0x000000010b256bc0 tokio::runtime::thread_pool::ThreadPool::block_on::h608917198b4ecc46 + 64
8   query-engine-darwin           	0x000000010b267b24 tokio::runtime::context::enter::h341c7c7ce3d85be9 + 180
9   query-engine-darwin           	0x000000010b31aab2 tokio::runtime::handle::Handle::enter::h783938b04a1eb2bf + 258
10  query-engine-darwin           	0x000000010b27b086 prisma::main::ha25e5272c0b99b72 + 150
11  query-engine-darwin           	0x000000010b25be16 std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h81750dd2b0df955a + 6
12  query-engine-darwin           	0x000000010bbc8c58 std::panicking::try::do_call::hb5f9c52a170af65b + 24
13  query-engine-darwin           	0x000000010bbccd6f __rust_maybe_catch_panic + 31
14  query-engine-darwin           	0x000000010bbc96ae std::rt::lang_start_internal::h032b1be013493933 + 542
15  query-engine-darwin           	0x000000010b27b249 main + 41
16  query-engine-darwin           	0x000000010b20c9d4 start + 52

The 20 crash reports I have for query-engine-darwin have an identical stack for the crashed thread.

I have uploaded all crash reports for your review, including one for introspection-engine-darwin and one for migration-engine-darwin (which I don't think will be very helpful, but who knows).

The archive also contains a copy of the working runtime and one of the faulty runtime.
Hopefully you can manually checksum the last one and figure out whether it matches any of the generated runtimes you've produced lately... or even reproduce crashes locally 🙂

That's it!
Let me know if I can help in any way!

[janpio]

Thanks @olance - this confirms our observations: There is a chunk of data missing in the binaries for an unknown reason. As I wrote above, we have our theories but we do not know for sure. We will get more data when preview025 is released and adopted by e.g. Redwood and you can try this again.

[janpio]

We haven't seen a segfault caused by a corrupt binary for quite some time, so I think the best move is to close this issue for now. We can still reopen if someone experiences the same problem again <3

[WesCossick]

@janpio Just found this post via a Google search seconds before you closed it; crazy timing. Anyway, I am hitting a segmentation fault error when running npx prisma generate using the latest beta version. Here's the full output:

% npx prisma generate
Environment variables loaded from ./prisma/.env

✔ Generated Prisma Client to ./node_modules/@prisma/client in 33ms

You can now start using Prisma Client in your code:

```
import { PrismaClient } from '@prisma/client'
// or const { PrismaClient } = require('@prisma/client')

const prisma = new PrismaClient()
```

Explore the full API: http://pris.ly/d/client

zsh: segmentation fault  npx prisma generate

And here's the full schema.prisma file:

generator client {
  provider = "prisma-client-js"
}

datasource db {
  provider = "mysql"
  url      = env("DATABASE_URL")
}

model User {
  consecutiveFailedLogins Int       @default(0) @map("consecutive_failed_logins")
  created                 DateTime  @default(now())
  email                   String    @unique
  firstName               String    @map("first_name")
  lastLogin               DateTime? @map("last_login")
  lastName                String    @map("last_name")
  password                String
  privileges              Json?
  slackUserID             String?   @map("slack_user_id")
  id                      Int       @id @map("user_id")

  @@map("users")
}

There's not much detail provided about why there's a segmentation fault happening.

FWIW, the npx prisma generate command seems to have actually worked, even though it dies with code 139.

Also, this problem was not occurring on Friday. I was using the same Prisma version, same dependencies, same sequence of commands, same schema file, etc.

[cephalization]

Happening as well when trying out redwoodjs.

yarn rw dev                   
yarn run v1.22.4
$ /home/tony/repos/page/node_modules/.bin/rw dev
  ✖ Generating the Prisma client...
    → info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Command failed with exit code 139: yarn prisma generate
Segmentation fault (core dumped)
error Command failed with exit code 139.
$ /home/tony/repos/page/node_modules/.bin/prisma generate

✔ Generated Prisma Client to ./../node_modules/@prisma/client in 27ms

You can now start using Prisma Client in your code:

import { PrismaClient } from '@prisma/client'
// or const { PrismaClient } = require('@prisma/client')

const prisma = new PrismaClient()

Explore the full API: http://pris.ly/d/client

info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

• created a new redwood app

• modified prisma schema

• ran yarn rw db save

• Selected 'no' when it asked me if I wanted to create sqlite db

• Realized I wanted one, hit ctrl-c

• Re-ran yarn rw db save

• Hit the above error

Since then I blew out my entire git repo, yarn cache, prisma cache. I now get an error even trying yarn rw dev from a new redwood project when I hit the prisma generate step so something got messed up globally from a local repo operation somehow :(

Using Pop_os 20.04 (ubuntu basically)

[pantharshit00]

@cephalization

Are you using Node js 14? If yes, please try the latest alpha npx @prisma/cli@alpha generate. See #2361 , in case of node 14 the segfault is coming from Node js and not from Prisma when we try to concurrently write files to the file system. We need to open a bug in Node for that but for now we have implemented a workaround which is available in alpha.

[cephalization]

I sure am, I knew it was a bad idea to use newest node 🙃 I’ll try an older node, thank you

EDIT

node 12.16.3 works perfectly, thanks again!

[WesCossick]

Using Node.js v12 instead of v14 fixed the issue for me as well.

[janpio]

Sorry, should have highlighted issue #2361 in my closing comment as well to point out that there is/was a problem with Node 14.

Thanks for the update, happy that this one is not back 🚀

[joneslloyd]

This solved it for me too! Thank you. I used the snippet here to force the node version to 12.16.3 for my Prisma-containing project only.

[wh1337]

I ran into this issue when attempting seed 1,230 objects into a Supabase table. I was using node v17.x at the time and every time I ran yarn prisma db seed it would get terminated tossing the following error:

Error: Command was killed with SIGSEGV (Segmentation fault): node --require esbuild-register prisma/seed.ts

I then changed back to the most recent node LTS (v16.16.0 at the time of this writing) and it worked without issue.

[parent-tobias]

Same situation, Supabase table tutorial, node v18.x was causing the SIGSEGV issue. Changed back to v16.3, ran fine