New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to deploy a service using local packages #370

Closed
mzmiric5 opened this Issue Apr 16, 2018 · 18 comments

Comments

Projects
None yet
6 participants
@mzmiric5
Copy link

mzmiric5 commented Apr 16, 2018

This is a Bug Report

Description

When attempting to package or deploy a service which uses local packages either referenced via file: or yarn-workspace definitions, the deploy/package step that triggers the webpack build throws an error with

Command failed: yarn install --frozen-lockfile --non-interactive
error An unexpected error occurred: "Your lockfile needs to be updated, but yarn was run with `--frozen-lockfile`.".

Running yarn install --frozen-lockfile --non-interactive manually works just fine, so that points to an issue with serverless-webpack.

For bug reports:

  • What went wrong?
    sls deploy and sls package fail with "Your lockfile needs to be updated, but yarn was run with `--frozen-lockfile`." error when using a local package
  • What did you expect should have happened?
    The service should have been packaged and deployed successfully
  • What was the config you used?
  • What stacktrace or error message from your provider did you see?

@HyperBrain as requested in #369 I've opened a separate issue. I also went ahead and prepared a minimal repro repo with the config and dependencies set up that cause these issues to surface. https://github.com/mzmiric5/sls-wp-build-error-repro
Some notes:

  • services/sls-only is a template without webpack, which deploys correctly and has no problems running on AWS
  • shared/third-party-private is a placeholder for a package we are unable to share, but the fact that the package we are using has this one specific dependency is causing npm repro to fail with an error on npm install inside the sls package/deploy step
  • our team primarily uses yarn, so this is our primary concern, and services/sls-wp-yarn shows the issue the best, as it fails with both the third-party-private package and the moment-wrapper example package (which the npm packager doesn't have an issue with)

For feature proposals:

  • What is the use case that should be solved. The more detail you describe this in the easier it is to understand for us.
  • If there is additional config how would it look

Similar or dependent issue(s):

Additional Data

  • Serverless-Webpack Version you're using: 5.1.1
  • Webpack version you're using: 4.5.0
  • Serverless Framework Version you're using: 1.26.1
  • Operating System: macOS Sierra 10.12.6
  • ***Stack Trace (if available)***:

@HyperBrain HyperBrain added the bug label Apr 16, 2018

@nickelstar

This comment has been minimized.

Copy link

nickelstar commented Apr 24, 2018

I ran into this exact same problem today. My yarn.lock file is up-to-date but the build still reports that it isn't. The command that fails in the build process runs without issue when executed manually.

@HyperBrain

This comment has been minimized.

Copy link
Member

HyperBrain commented Apr 25, 2018

Hi @nickelstar ,

Can you check the package.json files that are generated by the plugin during deployment? You should find them after the error happens in the .webpack/dependencies and .webpack/service or .webpack/---function-name-- folders. It might be that a dependency you have is wrongly put into devDependencies and as a consequence is not installed and packaged during the deployment.

The frozen-lockfile error should reliably detect such errors, even if it does not show the exact reason without deeper inspection.

@nickelstar

This comment has been minimized.

Copy link

nickelstar commented Apr 25, 2018

Thanks for the helpful info. When I looked at .webpack/dependencies/package.json, I noticed a package listed with an empty version number:

"dependencies": {
    "aws-sdk": "",

This struck me as odd since I don't use aws-sdk and it's not listed in my package.json. So I added it to my package.json via 'yarn add aws-sdk'. After that, the error went away and deployments started working with yarn specified as the webpack packager.

@HyperBrain

This comment has been minimized.

Copy link
Member

HyperBrain commented Apr 25, 2018

@nickelstar Good to hear that. I think you can even move the aws-sdk to devDependencies as it will be found but eventually omitted when packaging, because AWS already provides it in theiy Lambda environment. That will reduce your package sizes again.

@franciscocpg

This comment has been minimized.

Copy link
Member

franciscocpg commented Apr 25, 2018

@HyperBrain
I have this dependencies:

"dependencies": {
    "moment": "^2.20.1",
    "moment-timezone": "^0.5.14",
    "mongodb": "^2.2.33",
    "mssql": "^4.1.0",
    "object.values": "^1.0.4",
    "papaparse": "^4.3.6",
    "slug": "^0.9.1",
    "source-map-support": "^0.5.0",
    "sprintf-js": "^1.1.1",
    "underscore": "^1.8.3"
  },
  "devDependencies": {
    "aws-sdk": "^2.172.0",
    "babel-core": "^6.26.0",
    "babel-eslint": "^8.0.3",
    "babel-jest": "^22.1.0",
    "babel-loader": "^7.1.2",
    "babel-plugin-transform-runtime": "^6.23.0",
    "babel-preset-env": "^1.6.1",
    "babel-preset-flow": "^6.23.0",
    "eslint": "^4.16.0",
    "eslint-config-airbnb-base": "^12.1.0",
    "eslint-plugin-flowtype": "^2.40.1",
    "eslint-plugin-import": "^2.8.0",
    "eslint-plugin-jest": "^21.7.0",
    "flow-bin": "^0.63.1",
    "flow-typed": "^2.4.0",
    "glob": "^7.1.2",
    "jest": "^22.2.0",
    "serverless": "^1.25.0",
    "serverless-log-forwarding": "^1.1.6",
    "serverless-offline": "^3.16.0",
    "serverless-secrets": "https://github.com/franciscocpg/serverless-secrets.git#e623994662366b6fcfe7cc40f1d85fd19b5029cd",
    "serverless-webpack": "^5.1.1",
    "webpack": "^3.10.0",
    "webpack-node-externals": "^1.6.0",
    "webpack-plugin-copy": "^1.0.1"
  }

and then I see this .webpack/dependencies/package.json generated:

{
  (...)
  "dependencies": {
    "mongodb": "^2.2.33",
    "mssql": "^4.1.0",
    "object.values": "^1.0.4",
    "source-map-support": "^0.5.0",
    "moment": "^2.20.1",
    "lodash": "",
    "slug": "^0.9.1",
    "moment-timezone": "^0.5.14",
    "papaparse": "^4.3.6",
    "underscore": "^1.8.3",
    "sprintf-js": "^1.1.1"
  }
}

This empty lodash ("lodash":"") is causing the Your lockfile needs to be updated, but yarn was run with --frozen-lockfile error for me.

If I remove it yarn install --frozen-lockfile --non-interactive runs successfully. Do you have some idea what is causing this empty lodash?

Also when I run sls package I got a lot of warnings like this: WARNING: Could not determine version of module lodash.

@HyperBrain

This comment has been minimized.

Copy link
Member

HyperBrain commented Apr 25, 2018

@franciscocpg That is strange. The lodash in the package.json points out that webpack deteceted it as a primary dependency of your code, but it seems to be missing in your original package.json.

This can happen if you use it directly in your code (by using import or require lodash) but forgot to add it as dependency. If any of your dependencies brings lodash in, you won't recognize the error locally, because it is implicitly in the node_modules folder. NPM might swallow such issues, but yarn - with it's better optimization - will bring it to the surface.
So I'm sure that the root cause is, that you use lodash somewhere but did not add it as explicit dependency.

@franciscocpg

This comment has been minimized.

Copy link
Member

franciscocpg commented Apr 25, 2018

@HyperBrain
I've search for all occurrences of "lodash" in my entire project and I'm not using it anywhere.
I guess is some problem related to transitive dependencies and how yarn is resolving it.
Anyway I was able to workaround it by adding "lodash": "^4.17.10" to my dependencies.

@HyperBrain

This comment has been minimized.

Copy link
Member

HyperBrain commented Apr 25, 2018

@franciscocpg Yes, you're right. This looks like a bug in the Yarn dependency tree transform (packager/yarn.js) and it seems to fail to retrieve the version numbers of 2nd level dependencies.
Cannot be much. I'll have a look later this week.
However, the case that a version cannot be retrieved can still happen even if the bug is fixed. Would it be wise and safe to assume @latest in these cases (additionally to the printed warning)?

@damonmaria

This comment has been minimized.

Copy link

damonmaria commented Apr 27, 2018

Just struck this too (the original issue with local packages, not the lodash problem).

My local package is called shared. In the project directory it's referred to in files as:

  • package.json: "shared": "file:../shared/dist/",
  • yarn.lock: "shared@file:../shared/dist":

But in .webpack/dependencies it's referred to as:

  • package.json: "shared": "file:../../../shared/dist/",
  • yarn.lock: "shared@file:../shared/dist":

So the relative link in package.json has been fixed up with the move, but not the lock file. So yarn correctly fails but that might be tricky to correct for...

@HyperBrain

This comment has been minimized.

Copy link
Member

HyperBrain commented Apr 27, 2018

@damonmaria Yes, that's exactly the problem. For npm we implemented a lock file rebasing for file references, for yarn I did not, because I missed it.

In theory the same approach as with npm should work (that's an easy one). I'll prepare a PR soon and handle this with high prio so that you can try and test that later.

@HyperBrain

This comment has been minimized.

Copy link
Member

HyperBrain commented Apr 27, 2018

I'm working on a solution right now. However, there are 2 cases how local files can be integrated with yarn, so both need a proper handling:

case 1: yarn add my-module ../../somedir/mymodule
case 2: yarn add my-module file:../../somedir/mymodule
which lead to different representations in package.json as well as in yarn.lock

I'll do a first try to rebase any version reference that is file-like. If we encounter issues with that, we could - at least for now - restrict file references to use only the file: notation to make the use case work.

@HyperBrain

This comment has been minimized.

Copy link
Member

HyperBrain commented Apr 27, 2018

@franciscocpg I think we should separate the 2 issues contained in this one into 2 issues.

I will fix the "local file references not handled" here and close it with the PR, but I think thet the dependency issue with yarn and transient dependencies is important too and should be solved and discussed properly.

@franciscocpg

This comment has been minimized.

Copy link
Member

franciscocpg commented Apr 27, 2018

@HyperBrain
That makes sense.

I'm going to be out on the next 2 weeks so I'm unable to help during this period.

About the yarn and transient dependencies currently we can use this #370 (comment) as a workaround so I think is not too critical.

@HyperBrain

This comment has been minimized.

Copy link
Member

HyperBrain commented Apr 27, 2018

@franciscocpg Agree & no problem 😄 . Just tested the attached PR #376 and for me it worked now locally with yarn + file references. I'll add some unit tests, but anyone here (@damonmaria , @mzmiric5 ) can already test if it solves the problem.

@damonmaria

This comment has been minimized.

Copy link

damonmaria commented Apr 27, 2018

I actually worked around this. My use case was sharing a project package across the different parts of the project. Once I struck this (and similar issues with the same shared package in non-Serverless parts of the project ) I instead installed the package (again as a file: URL) in the root of the project. It no longer needs to be moved as each part of the project is built, and when node and Webpack search for it they look up the tree, out of the sub-project into the root of the project. Webpack seems to handle this fine, pulling in the code at build time.

@mzmiric5

This comment has been minimized.

Copy link
Author

mzmiric5 commented Apr 28, 2018

@HyperBrain just tested the PR branch with our private package and this fixed the issue we were having. Thanks a lot for a quick fix.

@HyperBrain

This comment has been minimized.

Copy link
Member

HyperBrain commented Apr 28, 2018

Released with 5.1.3

@chris-feist

This comment has been minimized.

Copy link

chris-feist commented Nov 2, 2018

I'm experiencing this issue using serverless-webpack 5.2.0. I have private scoped packages, and one of the dependencies that are defined in my private package is showing up without a version in .webpack/dependencies/package.json. However, it appears to be correct in the yarn.lock in that directory. I am whitelisting my scoped packages in the webpack config.

externals: [nodeExternals({
    whitelist: [/^@my-scope/],
  })],

@renovate renovate bot referenced this issue Nov 9, 2018

Open

chore(deps): serverless-webpack #17

0 of 1 task complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment