Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core cluster dies when starting misskey v12.119.0 #9135

Closed
stevolucidity opened this issue Nov 1, 2022 · 24 comments
Closed

core cluster dies when starting misskey v12.119.0 #9135

stevolucidity opened this issue Nov 1, 2022 · 24 comments
Labels
⚠️bug? This might be a bug

Comments

@stevolucidity
Copy link

💡 Summary

ERR * [core cluster] [1] died :(

🥰 Expected Behavior

misskey should continue to start normally.

🤬 Actual Behavior

misskey@social:~/misskey$ NODE_ENV=production npm start

> misskey@12.119.0 start
> cd packages/backend && node --experimental-json-modules ./built/index.js

  _____ _         _
 |     |_|___ ___| |_ ___ _ _
 | | | | |_ -|_ -| '_| -_| | |
 |_|_|_|_|___|___|_,_|___|_  |
 v12.119.0               |___|

 Misskey is an open-source decentralized microblogging platform.
 If you like Misskey, please donate to support development. https://www.patreon.com/syuilo

--- social (PID: 134071) ---
INFO *	[core boot]	Welcome to Misskey!
INFO *	[core boot]	Misskey v12.119.0
INFO *	[core boot env]	NODE_ENV: production
INFO *	[core boot nodejs]	Version v18.12.0 detected.
DONE *	[core boot config]	Loaded
INFO *	[core boot db]	Connecting...
DONE *	[core boot db]	Connected: v13.8 (Debian 13.8-0+deb11u1)
DONE *	[core boot]	Misskey initialized
INFO *	[core boot]	Starting 1 worker...
(node:134083) ExperimentalWarning: Importing JSON modules is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
ERR  *	[core cluster]	[1] died :(
(node:134097) ExperimentalWarning: Importing JSON modules is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
ERR  *	[core cluster]	[2] died :(

📝 Steps to Reproduce

  1. Follow guide at https://misskey-hub.net/en/docs/install/manual.html
  2. Execute step 7 That is it..
  • NODE_ENV=production npm start

📌 Environment

Misskey version: 12.119.0
Your OS: Linux social 5.10.0-19-amd64 #1 SMP Debian 5.10.149-2 (2022-10-21) x86_64 GNU/Linux
Your browser: Safari

misskey@social:~/misskey$ npm --version
8.19.2
misskey@social:~/misskey$ node --version
v18.12.0
misskey@social:~/misskey$ yarn --version
1.22.19
@stevolucidity stevolucidity added the ⚠️bug? This might be a bug label Nov 1, 2022
@stevolucidity
Copy link
Author

Might be related...

Build is using an old yarn version:

  • root yarn version = 3.2.4
  • misskey yarn version = 1.22.19

@ThatOneCalculator
Copy link
Contributor

Likely urelated, as me and a few other people have been using yarn 3.2.2 without problems for months now.

@tamaina
Copy link
Member

tamaina commented Nov 1, 2022

Now, try installing the package with npx yarn install.

However, it is unlikely that a package mistake will cause died (sudden death).

@stevolucidity
Copy link
Author

I tried npx yarn install. It gave me:

misskey@social:~/misskey$ npx yarn install
yarn install v1.22.19
[1/4] Resolving packages...
success Already up-to-date.
$ node ./scripts/install-packages.js
/home/misskey/misskey/scripts/install-packages.js:1
const execa = require('execa');
              ^

Error [ERR_REQUIRE_ESM]: require() of ES Module /home/misskey/misskey/node_modules/execa/index.js from /home/misskey/misskey/scripts/install-packages.js not supported.
Instead change the require of index.js in /home/misskey/misskey/scripts/install-packages.js to a dynamic import() which is available in all CommonJS modules.
    at Object.<anonymous> (/home/misskey/misskey/scripts/install-packages.js:1:15) {
  code: 'ERR_REQUIRE_ESM'
}

Node.js v18.12.0
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

Can you help me resolve this new error please?

@ThatOneCalculator
Copy link
Contributor

npx yarn install is never advised as it doesn't read packages properly a lot of the time, especially from different yarn versions.

@tamaina
Copy link
Member

tamaina commented Nov 1, 2022

require() of ES Module /home/misskey/misskey/node_modules/execa/index.js from /home/misskey/misskey/scripts/install-packages.js not supported.

I have no idea why this error would occur.

@stevolucidity
Copy link
Author

Can someone please provide their current versions of npm, node and yarn?

@ThatOneCalculator
Copy link
Contributor

Node: 18.4.0
Yarn: 3.2.2
NPM: 8.12.1

@tamaina
Copy link
Member

tamaina commented Nov 1, 2022

ubuntu@primary:〜$ node --version
v18.11.0
ubuntu@primary:〜$ npm --version
8.19.2
ubuntu@primary:〜$ yarn --version
1.22.19

@stevolucidity
Copy link
Author

I thought I'd try upgrading the project version of yarn to latest stable (3.2.4).
I removed the misskey user and /home/misskey folder before starting from scratch.
This really hasn't helped.

root@social:/home# rm misskey/ -rf
root@social:/home# userdel misskey
root@social:/home# adduser --disabled-password --disabled-login misskey
Adding user `misskey' ...
Adding new group `misskey' (1004) ...
Adding new user `misskey' (1004) with group `misskey' ...
Creating home directory `/home/misskey' ...
Copying files from `/etc/skel' ...
Changing the user information for misskey
Enter the new value, or press ENTER for the default
	Full Name []:
	Room Number []:
	Work Phone []:
	Home Phone []:
	Other []:
Is the information correct? [Y/n] y
root@social:/home# su - misskey
misskey@social:~$ yarn --version
1.22.19
misskey@social:~$ yarn set version stable
➤ YN0000: Retrieving https://repo.yarnpkg.com/3.2.4/packages/yarnpkg-cli/bin/yarn.js
➤ YN0000: Saving the new release in .yarn/releases/yarn-3.2.4.cjs
➤ YN0000: Done in 0s 752ms
misskey@social:~$ yarn --version
3.2.4
misskey@social:~$ git clone --recursive https://github.com/misskey-dev/misskey.git
Cloning into 'misskey'...
remote: Enumerating objects: 178602, done.
remote: Total 178602 (delta 0), reused 0 (delta 0), pack-reused 178602
Receiving objects: 100% (178602/178602), 109.22 MiB | 20.19 MiB/s, done.
Resolving deltas: 100% (134767/134767), done.
Submodule 'misskey-assets' (https://github.com/misskey-dev/assets.git) registered for path 'misskey-assets'
Cloning into '/home/misskey/misskey/misskey-assets'...
remote: Enumerating objects: 183, done.
remote: Counting objects: 100% (182/182), done.
remote: Compressing objects: 100% (133/133), done.
remote: Total 183 (delta 49), reused 175 (delta 43), pack-reused 1
Receiving objects: 100% (183/183), 69.76 MiB | 5.42 MiB/s, done.
Resolving deltas: 100% (49/49), done.
Submodule path 'misskey-assets': checked out '0179793ec891856d6f37a3be16ba4c22f67a81b5'
misskey@social:~$ cd misskey
misskey@social:~/misskey$ git checkout master
Branch 'master' set up to track remote branch 'master' from 'origin'.
Switched to a new branch 'master'
misskey@social:~/misskey$ yarn install

#
➤ YN0070: Migrating from Yarn 1; automatically enabling the compatibility node-modules linker 👍

➤ YN0000: ┌ Resolution step
➤ YN0061: │ chokidar@npm:2.1.8 is deprecated: Chokidar 2 does not receive security updates since 2019. Upgrade to chokidar 3 with 15x fewer dependencies
➤ YN0061: │ svgo@npm:0.7.2 is deprecated: This SVGO version is no longer supported. Upgrade to v2.x.x.
➤ YN0032: │ fsevents@npm:2.1.3: Implicit dependencies on node-gyp are discouraged
➤ YN0061: │ fsevents@npm:2.1.3 is deprecated: "Please update to latest v2.3 or v2.2"
➤ YN0061: │ fsevents@npm:1.2.13 is deprecated: fsevents 1 will break on node v14+ and could be using insecure binaries. Upgrade to fsevents 2.
➤ YN0061: │ flatten@npm:1.0.3 is deprecated: flatten is deprecated in favor of utility frameworks such as lodash.
➤ YN0061: │ browserslist@npm:1.7.7 is deprecated: Browserslist 2 could fail on reading Browserslist >3.0 config used in other tools.
➤ YN0032: │ nan@npm:2.15.0: Implicit dependencies on node-gyp are discouraged
➤ YN0061: │ source-map-resolve@npm:0.5.3 is deprecated: See https://github.com/lydell/source-map-resolve#deprecated
➤ YN0061: │ source-map-url@npm:0.4.0 is deprecated: See https://github.com/lydell/source-map-url#deprecated
➤ YN0061: │ resolve-url@npm:0.2.1 is deprecated: https://github.com/lydell/resolve-url#deprecated
➤ YN0061: │ urix@npm:0.1.0 is deprecated: Please see https://github.com/lydell/urix#deprecated
➤ YN0002: │ misskey@workspace:. doesn't provide eslint (pff94e), requested by @typescript-eslint/parser
➤ YN0000: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code
➤ YN0000: └ Completed in 10s 677ms
➤ YN0000: ┌ Fetch step
➤ YN0013: │ yargs-parser@npm:20.2.7 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ yargs-parser@npm:5.0.1 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ yargs@npm:7.1.0 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ yauzl@npm:2.10.0 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ typescript@npm:4.8.3 can't be found in the cache and will be fetched from the remote registry
➤ YN0000: └ Completed in 55s 44ms
➤ YN0000: ┌ Link step
➤ YN0007: │ cypress@npm:10.7.0 must be built because it never has been before or the last one failed
➤ YN0007: │ misskey@workspace:. must be built because it never has been before or the last one failed
➤ YN0009: │ misskey@workspace:. couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-5eecbd43/build.log)
➤ YN0000: └ Completed in 26s 894ms
➤ YN0000: Failed with errors in 1m 33s

@stevolucidity
Copy link
Author

I've reverted to my original configs and versions and I'm back to the original error.

Clutching at straws now, but as I'm also running diaspora, friendica, mastodon, pixelfed and peertube instances, could this be a clash caused in my config somewhere?

@ThatOneCalculator
Copy link
Contributor

ThatOneCalculator commented Nov 1, 2022

Doubt it, as long as your postgres and redis instances are healthy. Could you try out my fork? I optimized it for yarn berry so you might get some better results.

@stevolucidity
Copy link
Author

Doubt it, as long as your postgres and redis instances are healthy. Could you try out my fork? I optimized it for yarn berry so you might get some better results.

On it now...

@stevolucidity
Copy link
Author

misskey@social:~$ rm misskey/ -rf
misskey@social:~$ git clone https://codeberg.org/thatonecalculator/calckey.git misskey
Cloning into 'misskey'...
remote: Enumerating objects: 244778, done.
remote: Counting objects: 100% (244778/244778), done.
remote: Compressing objects: 100% (62289/62289), done.
remote: Total 244778 (delta 184150), reused 241327 (delta 181101), pack-reused 0
Receiving objects: 100% (244778/244778), 146.11 MiB | 30.15 MiB/s, done.
Resolving deltas: 100% (184150/184150), done.
misskey@social:~$ yarn set version stable
➤ YN0000: Retrieving https://repo.yarnpkg.com/3.2.4/packages/yarnpkg-cli/bin/yarn.js
➤ YN0000: Saving the new release in .yarn/releases/yarn-3.2.4.cjs
➤ YN0000: Done in 0s 777ms
misskey@social:~$ cd misskey/
misskey@social:~/misskey$ yarn install
➤ YN0000: ┌ Project validation
➤ YN0057: │ backend: Resolutions field will be ignored
➤ YN0000: └ Completed
➤ YN0000: ┌ Resolution step
➤ YN0002: │ @tensorflow/tfjs@npm:3.21.0 doesn't provide seedrandom (p563bf), requested by @tensorflow/tfjs-data
➤ YN0002: │ backend@workspace:packages/backend doesn't provide @tensorflow/tfjs (pd0e11), requested by nsfwjs
➤ YN0060: │ backend@workspace:packages/backend provides ioredis (p6a45b) with version 4.28.5, which doesn't satisfy what typeorm requests
➤ YN0002: │ backend@workspace:packages/backend doesn't provide webpack (p029b1), requested by json5-loader
➤ YN0002: │ backend@workspace:packages/backend doesn't provide webpack (p4a450), requested by ts-loader
➤ YN0002: │ tesseract.js@npm:3.0.3 doesn't provide eslint (pd77b8), requested by babel-eslint
➤ YN0000: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code
➤ YN0000: └ Completed in 0s 572ms
➤ YN0000: ┌ Post-resolution validation
➤ YN0068: │ chalk ➤ dependencies ➤ #ansi-styles: No matching package in the dependency tree; you may not need this rule anymore.
➤ YN0068: │ chalk ➤ dependencies ➤ #supports-color: No matching package in the dependency tree; you may not need this rule anymore.
➤ YN0000: └ Completed
➤ YN0000: ┌ Fetch step
➤ YN0013: │ yn@npm:3.1.1 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ yocto-queue@npm:0.1.0 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ zip-stream@npm:4.1.0 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ zlibjs@npm:0.3.1 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ browser-image-resizer@https://github.com/misskey-dev/browser-image-resizer.git#commit=0380d12c8e736788ea7f4e6e985175521ea7b23c can't be found in the cache and will be fetched from the remote repository
➤ YN0018: │ browser-image-resizer@https://github.com/misskey-dev/browser-image-resizer.git#commit=0380d12c8e736788ea7f4e6e985175521ea7b23c: The remote archive doesn't match the expected checksum
➤ YN0013: │ plyr@https://github.com/sampotts/plyr.git#commit=d434c9af16e641400aaee93188594208d88f2658 can't be found in the cache and will be fetched from the remote repository
➤ YN0018: │ plyr@https://github.com/sampotts/plyr.git#commit=d434c9af16e641400aaee93188594208d88f2658: The remote archive doesn't match the expected checksum
➤ YN0000: └ Completed in 2m 58s
➤ YN0000: Failed with errors in 2m 59s

@ThatOneCalculator
Copy link
Contributor

The yarn lock is broke rn, run YARN_CHECKSUM_BEHAVIOR=update yarn install (as per the readme)

@stevolucidity
Copy link
Author

Thanks for your efforts. The build process worked, but alas I'm still seeing the core cluster died error.


> calckey@12.119.0-calc.6.5 start
> yarn workspace backend run start

   ___      _      _
  / __\__ _| | ___| | _____ _   _
 / /  / _` | |/ __| |/ / _  | | |
/ /__| (_| | | (__|   <  __/ |_| |
\____/\__,_|_|\___|_|\_\___|\__, |
                            (___/
 Calckey is an open-source decentralized microblogging platform.
 If you like Calckey, please consider starring or contributing to the repo. https://codeberg.org/thatonecalculator/calckey

--- social (PID: 30697) ---
INFO *	[core boot]	Welcome to Calckey!
INFO *	[core boot]	Calckey v12.119.0-calc.6.5
INFO *	[core boot env]	NODE_ENV: production
INFO *	[core boot nodejs]	Version v18.12.0 detected.
DONE *	[core boot config]	Loaded
INFO *	[core boot db]	Connecting...
DONE *	[core boot db]	Connected: v13.8 (Debian 13.8-0+deb11u1)
DONE *	[core boot]	Calckey initialized
INFO *	[core boot]	Starting 1 worker...
(node:30709) ExperimentalWarning: Importing JSON modules is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
ERR  *	[core cluster]	[1] died :(
(node:30722) ExperimentalWarning: Importing JSON modules is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
ERR  *	[core cluster]	[2] died :(
(node:30734) ExperimentalWarning: Importing JSON modules is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
ERR  *	[core cluster]	[3] died :(
(node:30751) ExperimentalWarning: Importing JSON modules is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
ERR  *	[core cluster]	[4] died :(
(node:30774) ExperimentalWarning: Importing JSON modules is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
ERR  *	[core cluster]	[5] died :(
^C

Something else must be amiss if no-one else has this problem. 🤨

@ThatOneCalculator
Copy link
Contributor

Can you check how much CPU/RAM you have available?

@stevolucidity
Copy link
Author

top - 18:34:26 up  2:22,  2 users,  load average: 0.33, 0.44, 0.84
Tasks: 149 total,   1 running, 148 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.0 us,  0.5 sy,  0.0 ni, 97.2 id,  0.3 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   7856.3 total,   1613.7 free,   2624.4 used,   3618.1 buff/cache
MiB Swap:   6144.0 total,   6089.5 free,     54.5 used.   4817.8 avail Mem
root@social:/home/steve# df -h
Filesystem                     Size  Used Avail Use% Mounted on
udev                           3.9G     0  3.9G   0% /dev
tmpfs                          786M  1.1M  785M   1% /run
/dev/mapper/social--vg-root     23G   10G   12G  47% /
tmpfs                          3.9G   16K  3.9G   1% /dev/shm
tmpfs                          5.0M     0  5.0M   0% /run/lock
/dev/sda1                      470M  116M  330M  27% /boot
/dev/mapper/social--vg-tmp     1.8G  286M  1.5G  17% /tmp
/dev/mapper/social--vg-home    184G   11G  164G   7% /home
/dev/mapper/social--vg-bigvar  452G   14G  415G   4% /var
tmpfs                          786M     0  786M   0% /run/user/1000

@stevolucidity
Copy link
Author

root@social:/home/steve# mpstat
Linux 5.10.0-19-amd64 (social) 	01/11/22 	_x86_64_	(2 CPU)

18:43:11     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
18:43:11     all   34.44    0.00    4.95    2.33    0.00    0.39    0.00    0.00    0.00   57.88

@ThatOneCalculator
Copy link
Contributor

Ok, so resources aren't a problem. Can you try with Docker?

@stevolucidity
Copy link
Author

I can try. I'm building it in docker now.
I'd much prefer it to run natively like all my other Fediverse instances, though.

@stevolucidity
Copy link
Author

You're not going to believe this...

web_1    |
web_1    | > misskey@12.119.0 migrateandstart
web_1    | > npm run migrate && npm run start
web_1    |
web_1    |
web_1    | > misskey@12.119.0 migrate
web_1    | > cd packages/backend && npx typeorm migration:run -d ormconfig.js
web_1    |
web_1    | query: SELECT * FROM current_schema()
web_1    | query: SELECT version();
web_1    | query: SELECT * FROM "information_schema"."tables" WHERE "table_schema" = 'public' AND "table_name" = 'migrations'
web_1    | query: SELECT * FROM "migrations" "migrations" ORDER BY "id" DESC
web_1    | No migrations are pending
web_1    |
web_1    | > misskey@12.119.0 start
web_1    | > cd packages/backend && node --experimental-json-modules ./built/index.js
web_1    |
web_1    |   _____ _         _
web_1    |  |     |_|___ ___| |_ ___ _ _
web_1    |  | | | | |_ -|_ -| '_| -_| | |
web_1    |  |_|_|_|_|___|___|_,_|___|_  |
web_1    |  v12.119.0               |___|
web_1    |
web_1    |  Misskey is an open-source decentralized microblogging platform.
web_1    |  If you like Misskey, please donate to support development. https://www.patreon.com/syuilo
web_1    |
web_1    | --- 804bcc91efcd (PID: 67) ---
web_1    | INFO *	[core boot]	Welcome to Misskey!
web_1    | INFO *	[core boot]	Misskey v12.119.0
web_1    | INFO *	[core boot env]	NODE_ENV: production
web_1    | INFO *	[core boot nodejs]	Version v16.15.1 detected.
web_1    | DONE *	[core boot config]	Loaded
web_1    | INFO *	[core boot db]	Connecting...
web_1    | DONE *	[core boot db]	Connected: v12.2
web_1    | DONE *	[core boot]	Misskey initialized
web_1    | INFO *	[core boot]	Starting 1 worker...
web_1    | (node:78) ExperimentalWarning: Importing JSON modules is an experimental feature. This feature could change at any time
web_1    | (Use `node --trace-warnings ...` to show where the warning was created)
web_1    | ERR  *	[core cluster]	[1] died :(
web_1    | (node:90) ExperimentalWarning: Importing JSON modules is an experimental feature. This feature could change at any time
web_1    | (Use `node --trace-warnings ...` to show where the warning was created)
web_1    | ERR  *	[core cluster]	[2] died :(
^CGracefully stopping... (press Ctrl+C again to force)

@ThatOneCalculator
Copy link
Contributor

Frankly in my entire year of hosting Misskey & Calckey across multiple machines, I've never seen this before and I'm at a loss. I really want to make sure it's not local to your machine -- could you try a new server?

@stevolucidity
Copy link
Author

I could. In fact I started to do that last night on may host that runs nothing but Docker instances, but hit some odd yarn dependency issues.

I realised that I don't really need a Misskey instance. I'm covered with Mastodon, Diaspora, Friendica, Pixelfed and Peertube all working fine (on the same host).

Thanks for trying to help but agree it most be something local to this box that's not right.
I'll maybe try again in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
⚠️bug? This might be a bug
Projects
None yet
Development

No branches or pull requests

3 participants