Error: Cannot find module './core/yargs' when running migrations

[tc-dev-git]

Issue Description

Getting Error: Cannot find module './core/yargs' from AWS CodeDeploy when running sequelize db:migrate

What are you doing?

Running migrations as the last step on a CI/CD pipeline.

What do you expect to happen?

Migrations to run as they do in the local environment.

What is actually happening?

The deployment task stops due to the error importing a module on sequelize.

Additional context

LifecycleEvent - AfterInstall
Script - scripts/run_migrations
[stdout]
[stdout]> api@2.0.0 migrate /var/www/backend
[stdout]> sequelize db:migrate
[stdout]
[stderr]internal/modules/cjs/loader.js:968
[stderr] throw err;
[stderr] ^
[stderr]
[stderr]Error: Cannot find module './core/yargs'
[stderr]Require stack:
[stderr]- /var/www/backend/node_modules/.bin/sequelize
[stderr] at Function.Module._resolveFilename (internal/modules/cjs/loader.js:965:15)
[stderr] at Function.Module._load (internal/modules/cjs/loader.js:841:27)
[stderr] at Module.require (internal/modules/cjs/loader.js:1025:19)
[stderr] at require (internal/modules/cjs/helpers.js:72:18)
[stderr] at Object.<anonymous> (/var/www/backend/node_modules/.bin/sequelize:4:37)
[stderr] at Module._compile (internal/modules/cjs/loader.js:1137:30)
[stderr] at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)
[stderr] at Module.load (internal/modules/cjs/loader.js:985:32)
[stderr] at Function.Module._load (internal/modules/cjs/loader.js:878:14)
[stderr] at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12) {
[stderr] code: 'MODULE_NOT_FOUND',
[stderr] requireStack: [ '/var/www/backend/node_modules/.bin/sequelize' ]
[stderr]}
[stderr]npm ERR! code ELIFECYCLE
[stderr]npm ERR! errno 1
[stderr]npm ERR! api@2.0.0 migrate: `sequelize db:migrate`
[stderr]npm ERR! Exit status 1
[stderr]npm ERR!
[stderr]npm ERR! Failed at the api@2.0.0 migrate script.
[stderr]npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
[stderr]
[stderr]npm ERR! A complete log of this run can be found in:
[stderr]npm ERR! /home/ubuntu/.npm/_logs/2020-12-15T16_40_58_026Z-debug.log

The run_migrations script is a simple:

#!/bin/bash
cd /var/www/backend
npm run migrate

where migrate is set as sequelize db:migrate on package.json scripts.

All the other steps and scripts on the pipeline are working just fine, the same for the system itself.
Since there are no issues running migrations locally, I wonder what am I missing on the production env.

Environment

• Sequelize version: 6.3.5
• Node.js version: 12.18.3
• Operating System: Ubuntu 18.04

Issue Template Checklist

How does this problem relate to dialects?

• I think this problem happens regardless of the dialect.
• I think this problem happens only for the following dialect(s):
• I don't know, I was using PUT-YOUR-DIALECT-HERE, with connector library version XXX and database version XXX

Would you be willing to resolve this issue by submitting a Pull Request?

• Yes, I have the time and I know how to start.
• Yes, I have the time but I don't know how to start, I would need guidance.
• No, I don't have the time, although I believe I could do it if I had the time...
• No, I don't have the time and I wouldn't even know how to start.

[tc-dev-git]

Never mind, it seems to be a problem with how AWS CodeBuild install packages. I've just moved the npm install to a later stage of the CI/CD.

[DavidStinghen]

what stage did you move npm install to?

[tc-dev-git]

I had a npm install on CodeBuild (as usual), so the deployment stage would only run migrations. I had to move it to CodeDeploy, before my run_migrations script (on AfterInstall).
Luckly, since this project in specific doesn't require a proper build and doesn't have unit tests, the build stage simply takes code from the repo and pass it on to CodeDeploy. For a more complex build stage, I imagine my solution wouldn't work.
If I had to guess what's the problem, I'd say it's some with what node version CodeBuild was running on.

[DavidStinghen]

Got it, thanks for your answer, it helped a lot!

[lazartravica]

The problem exists because AWS CodeBuild archives the artifacts in such a way which converts symlinks into regular files with contents.

node_modules/.bin/sequelize is in fact a symlink to node_modules/sequelize-cli/lib/sequelize and because that symlink gets replaced with the file itself, it cannot find it's dependency because it's looking for it in the node_modules/.bin/ and that folder has no ./core/yargs.js present.

TLDR: Don't run npm install in the Deploy step, this is both a security and a performance issue.
Run node_modules/sequelize-cli/lib/sequelize db:migrate in the Build step to fix the issue.

[YavorK]

Had the same error with Azure DevOps. The solution of @lazartravica worked for us too.

[AlexTech314]

The problem exists because AWS CodeBuild archives the artifacts in such a way which converts symlinks into regular files with contents.

node_modules/.bin/sequelize is in fact a symlink to node_modules/sequelize-cli/lib/sequelize and because that symlink gets replaced with the file itself, it cannot find it's dependency because it's looking for it in the node_modules/.bin/ and that folder has no ./core/yargs.js present.

TLDR: Don't run npm install in the Deploy step, this is both a security and a performance issue. Run node_modules/sequelize-cli/lib/sequelize db:migrate in the Build step to fix the issue.

Awesome stuff, solved the issue for me. Cheers!

[Adriano-Martins]

@lazartravica answer is correct.
In case you are doing a CodeBuild run and create a layer containing only the node_modules and this fails, remember to use the option -y store symbolic links as the link instead of the referenced file while zipping.
E.g.:

      - mkdir nodejs && mv node_modules package* yarn*  nodejs/
      - zip -ryq ../dependencies.zip ./nodejs