How are plugin's own dependencies installed?

[bahmutov]

I am trying to create cypress plugin locally in a forked build-example-basic repo (so I can run it on Netlify under https://app.netlify.com/sites/condescending-knuth-7cff53)

my root package has plugin as a dev dependency

{
  "name": "example-basic",
  "private": true,
  "version": "0.0.0",
  "description": "Example Netlify Build Site",
  "scripts": {
    "docs": "md-magic --path '**/*.md' --ignore 'node_modules'",
    "build": "mkdir -p build && cp src/index.html build",
    "postinstall": "echo nothing to do post NPM install"
  },
  "author": "David Wells",
  "license": "MIT",
  "devDependencies": {
    "markdown-magic": "^1.0.0",
    "netlify-plugin-cypress": "file:plugins/cypress"
  },
  "dependencies": {
    "@netlify/sitemap-plugin": "0.0.2"
  }
}

The plugin in plugins/cypress has its own package.json

{
  "name": "netlify-plugin-cypress",
  "private": true,
  "version": "0.0.1",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "cypress": "3.4.1"
  }
}

On the first deploy, plugin's dependencies ARE installed, for example in https://app.netlify.com/sites/condescending-knuth-7cff53/deploys/5da086c8bc913d00086e6fa9

9:42:42 AM: Installing dependencies
9:42:42 AM: Started restoring cached node version
9:42:45 AM: Finished restoring cached node version
9:42:46 AM: v10.16.3 is already installed.
9:42:47 AM: Now using node v10.16.3 (npm v6.9.0)
9:42:47 AM: Attempting ruby version 2.6.2, read from environment
9:42:48 AM: Using ruby version 2.6.2
9:42:49 AM: Using PHP version 5.6
9:42:49 AM: Started restoring cached node modules
9:42:49 AM: Finished restoring cached node modules
9:42:49 AM: Installing NPM modules using NPM version 6.9.0
9:42:54 AM: > cypress@3.4.1 postinstall /opt/build/repo/node_modules/netlify-plugin-cypress/node_modules/cypress
9:42:54 AM: > node index.js --exec install
9:42:55 AM: Installing Cypress (version: 3.4.1)
9:42:55 AM: 
9:42:55 AM: [13:42:55]  Downloading Cypress     [started]
9:42:57 AM: [13:42:57]  Downloading Cypress     [completed]
9:42:57 AM: [13:42:57]  Unzipping Cypress       [started]
9:43:46 AM: [13:43:46]  Unzipping Cypress       [completed]
9:43:46 AM: [13:43:46]  Finishing Installation  [started]
9:43:46 AM: [13:43:46]  Finishing Installation  [completed]
9:43:46 AM: 
9:43:47 AM: You can now open Cypress by running: node_modules/.bin/cypress open
9:43:47 AM: https://on.cypress.io/installing-cypress
9:43:47 AM: 
9:43:47 AM: > example-basic@0.0.0 postinstall /opt/build/repo
9:43:47 AM: > echo nothing to do post NPM install
9:43:47 AM: nothing to do post NPM install
9:43:47 AM: added 188 packages from 125 contributors and audited 541 packages in 57.744s
9:43:47 AM: found 0 vulnerabilities
9:43:47 AM: NPM modules installed

There package.json and package-lock.json files in both root and plugin's folders.

When running the plugin requires cypress npm module like this

const cypress = require('cypress')

module.exports = function cypressPlugin (config) {
  console.log('cypressPlugin config', config)
  postDeploy ....
}

The plugin works just fine, testing deployed site

9:44:11 AM: ❯ Loading plugins
9:44:11 AM:   Loading plugin "@netlify/functions" from /opt/buildhome/.nvm/versions/node/v10.16.3/lib/node_modules/@netlify/build/src/plugins/functions/index.js
9:44:11 AM:   Loading plugin "cypressPlugin" from netlify-plugin-cypress
9:44:11 AM: cypressPlugin config { baseUrl: 'http://localhost:8080' }
9:44:11 AM: ❯ Running Netlify Build Lifecycle
9:44:11 AM:   Found 2 steps. Lets do this!
9:44:11 AM: ┌───────────────────────────────────────────────────────────────┐
9:44:11 AM: │ 1. Running buildFunctions lifecycle from @netlify/functions   │
9:44:11 AM: └───────────────────────────────────────────────────────────────┘
9:44:11 AM: No functions directory set. Skipping functions build step
9:44:11 AM:  ✔  @netlify/functions.buildFunctions completed in 2ms
9:44:11 AM: ┌──────────────────────────────────────────────────────┐
9:44:11 AM: │ 2. Running postDeploy lifecycle from cypressPlugin   │
9:44:11 AM: └──────────────────────────────────────────────────────┘
9:44:11 AM: cypressPlugin start against https://condescending-knuth-7cff53.netlify.com/
...

On the second deploy though, NPM dependencies are not found, here is the build log

9:53:51 AM: Now using node v10.16.3 (npm v6.9.0)
9:53:51 AM: Attempting ruby version 2.6.2, read from environment
9:53:52 AM: Using ruby version 2.6.2
9:53:53 AM: Using PHP version 5.6
9:53:53 AM: Started restoring cached node modules
9:53:53 AM: Finished restoring cached node modules
9:53:53 AM: Started restoring cached go cache
9:53:53 AM: Finished restoring cached go cache
9:53:53 AM: unset GOOS;
9:53:53 AM: unset GOARCH;
9:53:53 AM: export GOROOT='/opt/buildhome/.gimme/versions/go1.12.linux.amd64';
9:53:53 AM: export PATH="/opt/buildhome/.gimme/versions/go1.12.linux.amd64/bin:${PATH}";
9:53:53 AM: go version >&2;
9:53:53 AM: export GIMME_ENV='/opt/buildhome/.gimme/env/go1.12.linux.amd64.env';
9:53:53 AM: go version go1.12 linux/amd64
9:53:53 AM: Installing missing commands
9:53:53 AM: Verify run directory
9:53:55 AM: Found Netlify Build that didn't match the version latest. Installing now.
9:54:05 AM: npm
9:54:05 AM:  WARN deprecated flatten@1.0.2: I wrote this module a very long time ago; you should use something else.
9:54:17 AM: /opt/buildhome/.nvm/versions/node/v10.16.3/bin/netlify-build -> /opt/buildhome/.nvm/versions/node/v10.16.3/lib/node_modules/@netlify/build/src/core/bin.js
9:54:17 AM: + @netlify/build@0.0.21
9:54:17 AM: updated 3 packages in 21.739s
9:54:17 AM: Executing netlify-build command (lifecycle trial enabled): netlify-build
9:54:18 AM: ❯ Starting Netlify Build
9:54:18 AM:   https://github.com/netlify/build
9:54:18 AM: ❯ Using config file:
9:54:18 AM:   /opt/build/repo/netlify.yml
9:54:18 AM: ❯ Loading plugins
9:54:18 AM:   Loading plugin "@netlify/functions" from /opt/buildhome/.nvm/versions/node/v10.16.3/lib/node_modules/@netlify/build/src/plugins/functions/index.js
9:54:18 AM:   Loading plugin "cypressPlugin" from netlify-plugin-cypress
9:54:18 AM: ┌─────────────────────────────┐
9:54:18 AM: │    Netlify Build Error!     │
9:54:18 AM: └─────────────────────────────┘
9:54:18 AM:  Error importing 'cypressPlugin' plugin:
9:54:18 AM: Cannot find module 'cypress'
9:54:18 AM: ┌─────────────────────────────┐
9:54:18 AM: │      Error Stack Trace      │
9:54:18 AM: └─────────────────────────────┘
9:54:18 AM:  Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
9:54:18 AM: Function.Module._load (internal/modules/cjs/loader.js:562:25)
9:54:18 AM: Module.require (internal/modules/cjs/loader.js:692:17)
9:54:18 AM: require (internal/modules/cjs/helpers.js:25:18)
9:54:18 AM: Object.<anonymous> (plugins/cypress/index.js:4:17)
9:54:18 AM: Module._compile (internal/modules/cjs/loader.js:778:30)
9:54:18 AM: Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
9:54:18 AM: Module.load (internal/modules/cjs/loader.js:653:32)
9:54:18 AM: tryModuleLoad (internal/modules/cjs/loader.js:593:12)
9:54:18 AM: Function.Module._load (internal/modules/cjs/loader.js:585:3)
9:54:18 AM:  Use the --verbose option for deep traces
9:54:19 AM: Caching artifacts
9:54:19 AM: Started saving node modules
9:54:19 AM: Finished saving node modules

Can I ask which node modules folder is restored?
Second question - is it possible to specify additional folders to cache and restore. For example when doing yarn I would like to cache and restore ~/.cache/yarn folder, and for Cypress itself I would love to cache ~/.cache/cypress folder (that's where the binary is unzipped into)

[DavidWells]

Can I ask which node modules folder is restored?

Right now only the root level node_modules is restored from the remote CI cache

I opened an issue to automatically ensure local plugin deps are installed if they are missing #273

In the future, all node_modules installed will be cached to speed everything up.

---

A temp workaround for this is cd plugins/cypress && npm install

 "scripts": {
    "docs": "md-magic --path '**/*.md' --ignore 'node_modules'",
    "build": "mkdir -p build && cp src/index.html build",
    "postbuild: "cd plugins/cypress && npm install"
  },

Until #273 is updated

[bahmutov]

but isn't this running AFTER netlify build (and Cypress plugin)? Maybe better to run it in prebuild?

[DavidWells]

Aha good point! Prebuild it is!

We should have this fixed within the next day or so

[bahmutov]

interesting, I don't think prebuild hook works anymore

[DavidWells]

try preBuild camelcased

On Fri, Oct 18, 2019 at 9:26 AM Gleb Bahmutov ***@***.***> wrote: interesting, I don't think prebuild hook works anymore — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#271?email_source=notifications&email_token=AAEB6MBE2EAHKAZ5VSWVD7DQPHPS7A5CNFSM4I72EPZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBVA5XI#issuecomment-543821533>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEB6MAVNBVOBNJTHQVSCTLQPHPS7ANCNFSM4I72EPZA> .

-- - David Wells Setup a call with me <https://calendly.com/david-wells-netlify>

[ehmicky]

This should be fixed now that #273 is done.

[bahmutov]

I still see failing build because plugin's Node dependencies are not restored

https://app.netlify.com/sites/condescending-knuth-7cff53/deploys/5daf10e9f48b4f000811b78a

10:23:47 AM: Installing dependencies
10:23:48 AM: Started restoring cached node version
10:23:52 AM: Finished restoring cached node version
10:23:53 AM: v10.16.3 is already installed.
10:23:54 AM: Now using node v10.16.3 (npm v6.9.0)
10:23:54 AM: Attempting ruby version 2.6.2, read from environment
10:23:55 AM: Using ruby version 2.6.2
10:23:56 AM: Using PHP version 5.6
10:23:56 AM: Started restoring cached node modules
10:23:56 AM: Finished restoring cached node modules
10:23:56 AM: Started restoring cached go cache
10:23:56 AM: Finished restoring cached go cache
10:23:56 AM: unset GOOS;
10:23:56 AM: unset GOARCH;
10:23:56 AM: export GOROOT='/opt/buildhome/.gimme/versions/go1.12.linux.amd64';
10:23:56 AM: export PATH="/opt/buildhome/.gimme/versions/go1.12.linux.amd64/bin:${PATH}";
10:23:56 AM: go version >&2;
10:23:56 AM: export GIMME_ENV='/opt/buildhome/.gimme/env/go1.12.linux.amd64.env';
10:23:56 AM: go version go1.12 linux/amd64
10:23:56 AM: Installing missing commands
10:23:56 AM: Verify run directory
10:23:58 AM: Found Netlify Build that didn't match the version latest. Installing now.
10:24:13 AM: npm
10:24:13 AM: WARN deprecated flatten@1.0.2: I wrote this module a very long time ago; you should use something else.
10:24:35 AM: /opt/buildhome/.nvm/versions/node/v10.16.3/bin/netlify-build -> /opt/buildhome/.nvm/versions/node/v10.16.3/lib/node_modules/@netlify/build/src/core/bin.js
10:24:35 AM: > core-js@2.6.10 postinstall /opt/buildhome/.nvm/versions/node/v10.16.3/lib/node_modules/@netlify/build/node_modules/core-js
10:24:35 AM: > node postinstall || echo "ignore"
10:24:35 AM: + @netlify/build@0.0.23
10:24:35 AM: added 35 packages from 18 contributors, removed 1 package and updated 15 packages in 35.552s
10:24:35 AM: Executing netlify-build command (lifecycle trial enabled): netlify-build
10:24:36 AM: ❯ Starting Netlify Build
10:24:36 AM:   https://github.com/netlify/build
10:24:36 AM: Options
10:24:36 AM:   { base: '/opt/build', lifecycleTrial: 'enabled=true' }
10:24:36 AM: ❯ Using config file:
10:24:36 AM:   /opt/build/repo/netlify.yaml
10:24:36 AM: ❯ Installing plugins dependencies
10:24:36 AM: ❯ Loading plugins
10:24:36 AM:   Loading plugin "@netlify/functions" from build core
10:24:36 AM:   Loading plugin "cypressPlugin" from netlify-plugin-cypress
10:24:37 AM: { Error: Error importing plugin:
10:24:37 AM: Cannot find module 'cypress'
10:24:37 AM:     at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
10:24:37 AM:     at Function.Module._load (internal/modules/cjs/loader.js:562:25)
10:24:37 AM:     at Module.require (internal/modules/cjs/loader.js:692:17)
10:24:37 AM:     at require (internal/modules/cjs/helpers.js:25:18)
10:24:37 AM:     at Object.<anonymous> (/opt/build/repo/plugins/cypress/index.js:4:17)
10:24:37 AM:     at Module._compile (internal/modules/cjs/loader.js:778:30)
10:24:37 AM:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
10:24:37 AM:     at Module.load (internal/modules/cjs/loader.js:653:32)
10:24:37 AM:     at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
10:24:37 AM:     a
10:24:37 AM: t Function.Module._load (internal/modules/cjs/loader.js:585:3) code: 'MODULE_NOT_FOUND' }
10:24:37 AM: ┌─────────────────────────────┐
10:24:37 AM: │    Netlify Build Error!     │
10:24:37 AM: └─────────────────────────────┘
10:24:37 AM:  Error loading 'cypressPlugin' plugin:
10:24:37 AM: Command failed with exit code 1
10:24:37 AM: { Error: Error importing plugin:
10:24:37 AM: Cannot find module 'cypress'
10:24:37 AM:     at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
10:24:37 AM:     at Function.Module._load (internal/modules/cjs/loader.js:562:25)
10:24:37 AM:     at Module.require (internal/modules/cjs/loader.js:692:17)
10:24:37 AM:     at require (internal/modules/cjs/helpers.js:25:18)
10:24:37 AM:     at Object.<anonymous> (/opt/build/repo/plugins/cypress/index.js:4:17)
10:24:37 AM:     at Module._compile (internal/modules/cjs/loader.js:778:30)
10:24:37 AM:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
10:24:37 AM:     at Module.load (internal/modules/cjs/loader.js:653:32)
10:24:37 AM:     at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
10:24:38 AM: Starting post processing
10:24:37 AM:     at Function.Module._load (internal/modules/cjs/loader.js:585:3) code: 'MODULE_NOT_FOUND' }
10:24:37 AM: ┌─────────────────────────────┐
10:24:37 AM: │      Error Stack Trace      │
10:24:37 AM: └─────────────────────────────┘
10:24:37 AM:  Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
10:24:38 AM: Post processing done
10:24:37 AM: Function.Module._load (internal/modules/cjs/loader.js:562:25)
10:24:37 AM: Module.require (internal/modules/cjs/loader.js:692:17)
10:24:37 AM: require (internal/modules/cjs/helpers.js:25:18)
10:24:37 AM: Object.<anonymous> (plugins/cypress/index.js:4:17)
10:24:37 AM: Module._compile (internal/modules/cjs/loader.js:778:30)
10:24:37 AM: Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
10:24:37 AM: Module.load (internal/modules/cjs/loader.js:653:32)
10:24:37 AM: tryModuleLoad (internal/modules/cjs/loader.js:593:12)
10:24:37 AM: makeError (/opt/buildhome/.nvm/versions/node/v10.16.3/lib/node_modules/@netlify/build/node_modules/execa/lib/error.js:56:11)
10:24:37 AM: handlePromise (/opt/buildhome/.nvm/versions/node/v10.16.3/lib/node_modules/@netlify/build/node_modules/execa/index.js:114:26)
10:24:37 AM: process._tickCallback (internal/process/next_tick.js:68:7)
10:24:37 AM:  Use the --verbose option for deep traces
10:24:37 AM: Caching artifacts
10:24:37 AM: Started saving node modules
10:24:38 AM: Site is live
10:24:37 AM: Finished saving node modules

and the failed plugin run still exits with 0

[ehmicky]

@bahmutov we haven't made a release yet.
Please note in the coming release, the plugins configuration has been changed to an array.

@DavidWells could you please make a release?