Directus crashed with SIGSEGV running in docker-container

[waza-ari]

Preflight Checklist

• I have completed all Troubleshooting Steps.
• I'm on the latest version of Directus.
• There's no other issue that already describes my problem.

Describe the Bug

I am using Directus 9.6.0 in a docker-container using the following docker-compose statements:

  directus:
    container_name: directus
    image: directus/directus:{{ directus_docker_version }}
    environment:
      - KEY={{ directus_auth_pubkey }}
      - SECRET={{ directus_auth_secretkey }}
      - DB_CLIENT=mysql
      - DB_HOST=mysql
      - DB_PORT=3306
      - DB_DATABASE=directus
      - DB_USER=directus
      - DB_PASSWORD={{ directus_mysql_password }}
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.directus.rule=Host(`{{ directus_external_url }}`)"
      - "traefik.http.routers.directus.entrypoints=websecure"
      - "traefik.http.routers.directus.tls.certresolver=letsencrypt"
      - "traefik.http.services.directus.loadbalancer.server.port=8055"

Pretty basic stuff. {{ directus_docker_version }} evaluates to 9.6.0. The container worked fine for a couple of days but then suddenly exited with an exit code 1. The last few log entries were:

[... many requests go here ... ]
directus            | 13:23:52 ✨ request completed GET 200 /admin 1ms
directus            | 13:24:13 ✨ request completed GET 200 /items/website_content/ 10ms
directus            | 13:24:15 ✨ request completed GET 200 /items/website_content/ 15ms
directus            | 13:24:22 ✨ request completed GET 302 / 1ms
directus            | 13:24:22 ✨ request completed GET 200 /admin 1ms
directus            | 13:24:43 ✨ request completed GET 200 /items/website_content/ 15ms
directus            | npm ERR! path /directus
directus            | npm ERR! command failed
directus            | npm ERR! signal SIGSEGV
directus            | npm ERR! command sh -c directus "start"
directus            | 
directus            | npm ERR! A complete log of this run can be found in:
directus            | npm ERR!     /home/node/.npm/_logs/2022-03-06T13_27_24_449Z-debug-0.log

The complete log that is referenced there contains:

0 verbose cli [
0 verbose cli   '/usr/local/bin/node',
0 verbose cli   '/usr/local/lib/node_modules/npm/bin/npm-cli.js',
0 verbose cli   'exec',
0 verbose cli   '--',
0 verbose cli   'directus',
0 verbose cli   'start'
0 verbose cli ]
1 info using npm@8.3.1
2 info using node@v16.14.0
3 timing npm:load:whichnode Completed in 1ms
4 timing config:load:defaults Completed in 3ms
5 timing config:load:file:/usr/local/lib/node_modules/npm/npmrc Completed in 2ms
6 timing config:load:builtin Completed in 2ms
7 timing config:load:cli Completed in 3ms
8 timing config:load:env Completed in 0ms
9 timing config:load:file:/directus/.npmrc Completed in 0ms
10 timing config:load:project Completed in 2ms
11 timing config:load:file:/home/node/.npmrc Completed in 0ms
12 timing config:load:user Completed in 1ms
13 timing config:load:file:/usr/local/etc/npmrc Completed in 0ms
14 timing config:load:global Completed in 1ms
15 timing config:load:validate Completed in 0ms
16 timing config:load:credentials Completed in 4ms
17 timing config:load:setEnvs Completed in 2ms
18 timing config:load Completed in 21ms
19 timing npm:load:configload Completed in 21ms
20 timing npm:load:setTitle Completed in 1ms
21 timing config:load:flatten Completed in 8ms
22 timing npm:load:display Completed in 10ms
23 verbose logfile /home/node/.npm/_logs/2022-03-06T13_27_24_449Z-debug-0.log
24 timing npm:load:logFile Completed in 10ms
25 timing npm:load:timers Completed in 0ms
26 timing npm:load:configScope Completed in 0ms
27 timing npm:load Completed in 46ms
28 timing command:exec Completed in 431841373ms
29 verbose stack Error: command failed
29 verbose stack     at ChildProcess.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/@npmcli/promise-spawn/index.js:64:27)
29 verbose stack     at ChildProcess.emit (node:events:520:28)
29 verbose stack     at maybeClose (node:internal/child_process:1092:16)
29 verbose stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)
30 verbose cwd /directus
31 verbose Linux 4.18.0-365.el8.x86_64
32 verbose argv "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/bin/npm-cli.js" "exec" "--" "directus" "start"
33 verbose node v16.14.0
34 verbose npm  v8.3.1
35 error path /directus
36 error command failed
37 error signal SIGSEGV
38 error command sh -c directus "start"
39 verbose exit 1
40 timing npm Completed in 431841947ms
41 verbose code 1
42 error A complete log of this run can be found in:
42 error     /home/node/.npm/_logs/2022-03-06T13_27_24_449Z-debug-0.log

Restarting the docker container fixed the issue apparently.

To Reproduce

I was unable to reproduce the issue since then

Errors Shown

See bug description above

What version of Directus are you using?

9.6.0

What version of Node.js are you using?

v16.14.0

What database are you using?

MySQL

What browser are you using?

Doesn't matter

What operating system are you using?

Linux CentOS 8 Stream

How are you deploying Directus?

Docker

[rijkvanzanten]

This is going to be near impossible to debug.. Especially seeing you weren't able to reproduce it after the fact 👀 I don't recall ever seeing SIGSEGV in any of our running containers of Directus, so I'm wondering if there's any specific mix of variables here that causes this.

I might have to close this for now, as I wouldn't really know where to start looking for an answer..

[azrikahar]

I'm also wondering what are the actionable items we can take on our side in this situation here, as it seems to be a somewhat rare issue. Granted I don't really have much of a clue here myself, I'm wondering if it is something to do with the host instead (memory/drive issue).

Perhaps a resource log such as cpu/memory usage up until the crash if you can reproduce it? It would also be very helpful if you can confirm whether this happens on another host as well.

I'll close this for now, but if there is any new breakthroughs on this peculiar occurrence, feel free to update us 👍

[waza-ari]

Sorry guys for not responding earlier. I can fully understand your point of view, since then it never happened again and I'm not sure what originally caused it. At the time when the directus container crashed, all other containers on the same host were running fine. Unfortunately I don't run directus on another host, so I cannot confirm the issue on other hosts either. Just wanted to document it in case I'm not the only one and others have the same problem.

I'll come back if there is a similar occurrence again and I'd kindly like to request everyone else stumbling across this issue and experiencing the same to comment here and re-open the issue.

Thanks for your time!

[waza-ari]

Hi there,

tonight the very same thing happened again. Directus crashed with SIGSEGV. The log file is below, although I doubt there is much useful content in there. It did run fine in the last 20 days since issue creation, no problem there.

I guess my main question: can I enable any kind of more verbose logging in case it happens again, such that we have more information to troubleshoot?

For now I'll update to the most recent version and see if the problem still persists.

/directus $ cat /home/node/.npm/_logs/2022-03-13T09_18_01_273Z-debug-0.log
0 verbose cli [
0 verbose cli   '/usr/local/bin/node',
0 verbose cli   '/usr/local/lib/node_modules/npm/bin/npm-cli.js',
0 verbose cli   'exec',
0 verbose cli   '--',
0 verbose cli   'directus',
0 verbose cli   'start'
0 verbose cli ]
1 info using npm@8.3.1
2 info using node@v16.14.0
3 timing npm:load:whichnode Completed in 1ms
4 timing config:load:defaults Completed in 3ms
5 timing config:load:file:/usr/local/lib/node_modules/npm/npmrc Completed in 3ms
6 timing config:load:builtin Completed in 4ms
7 timing config:load:cli Completed in 3ms
8 timing config:load:env Completed in 1ms
9 timing config:load:file:/directus/.npmrc Completed in 2ms
10 timing config:load:project Completed in 10ms
11 timing config:load:file:/home/node/.npmrc Completed in 0ms
12 timing config:load:user Completed in 1ms
13 timing config:load:file:/usr/local/etc/npmrc Completed in 1ms
14 timing config:load:global Completed in 1ms
15 timing config:load:validate Completed in 0ms
16 timing config:load:credentials Completed in 5ms
17 timing config:load:setEnvs Completed in 2ms
18 timing config:load Completed in 32ms
19 timing npm:load:configload Completed in 33ms
20 timing npm:load:setTitle Completed in 1ms
21 timing config:load:flatten Completed in 9ms
22 timing npm:load:display Completed in 13ms
23 verbose logfile /home/node/.npm/_logs/2022-03-13T09_18_01_273Z-debug-0.log
24 timing npm:load:logFile Completed in 31ms
25 timing npm:load:timers Completed in 0ms
26 timing npm:load:configScope Completed in 0ms
27 timing npm:load Completed in 80ms
28 timing command:exec Completed in 1795273009ms
29 verbose stack Error: command failed
29 verbose stack     at ChildProcess.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/@npmcli/promise-spawn/index.js:64:27)
29 verbose stack     at ChildProcess.emit (node:events:520:28)
29 verbose stack     at maybeClose (node:internal/child_process:1092:16)
29 verbose stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)
30 verbose cwd /directus
31 verbose Linux 4.18.0-365.el8.x86_64
32 verbose argv "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/bin/npm-cli.js" "exec" "--" "directus" "start"
33 verbose node v16.14.0
34 verbose npm  v8.3.1
35 error path /directus
36 error command failed
37 error signal SIGSEGV
38 error command sh -c directus "start"
39 verbose exit 1
40 timing npm Completed in 1795274829ms
41 verbose code 1
42 error A complete log of this run can be found in:
42 error     /home/node/.npm/_logs/2022-03-13T09_18_01_273Z-debug-0.log

[rijkvanzanten]

I'm wondering if this is Directus specific issue, or something bigger in running NodeJS projects through Docker on Linux Centos 8. From a quick search around, it seems like we're not the only ones seeing this: https://stackoverflow.com/questions/43167128/centos-docker-container-crashes-with-6-segmentation-fault-wheres-the-core-dum

[waza-ari]

I know this issue is closed, just wanted to give some feedback. For a variety of reasons we switched from CentOS8 to Ubuntu 22.04. Since then, we haven't seen a single crash of the Directus container. Apparently it looks like the issue is somehow related to the underlying OS.