Serverless fails to deploy if packager is yarn

[ArpitaSrivas]

This is a Bug Report

Unable to deploy serverless, works fine with npm packager but not with yarn, getting the following error with yarn packager:

 Syntax Error -------------------------------------------
 
  Unexpected token { in JSON at position 41
 
     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com
 
  Your Environment Information -----------------------------
     OS:                     linux
     Node Version:           9.11.1
     Serverless Version:     1.27.2

This issue is not happening on macOS 10.13.4

The build actually fails when the process tries to run yarn list --depth=1 --json --production, which is returning multiple objects, as a result JSON.parse is failing.
Sample Output:

{"type":"activityStart","data":{"id":0}}

{"type":"activityTick","data":{"id":0,"name":"@babel/polyfill@^7.0.0-beta.46"}}

{"type":"activityTick","data":{"id":0,"name":"validator@^10.1.0"}}

{"type":"activityTick","data":{"id":0,"name":"@babel/cli@7.0.0-beta.44"}}

{"type":"activityTick","data":{"id":0,"name":"@babel/core@7.0.0-beta.44"}}

{"type":"activityTick","data":{"id":0,"name":"@babel/preset-env@7.0.0-beta.44"}}

{"type":"activityTick","data":{"id":0,"name":"@babel/preset-stage-0@7.0.0-beta.44"}}

{"type":"activityTick","data":{"id":0,"name":"babel-eslint@^8.2.2"}}

{"type":"tree","data":{"type":"list","trees":[{"name":"@babel/polyfill@7.0.0-beta.46","children":[],"hint":null,"color":"bold","depth":0},{"name":"validator@10.2.0","children":[],"hint":null,"color":"bold","depth":0},{"name":"@babel/cli@7.0.0-beta.44","children":[],"hint":null,"color":"bold","depth":0},{"name":"@babel/core@7.0.0-beta.44","children":[{"name":"debug@3.1.0","children":[],"hint":null,"color":"bold","depth":0}],"hint":null,"color":"bold","depth":0},{"name":"@babel/preset-env@7.0.0-beta.44","children":[],"hint":null,"color":"bold","depth":0},{"name":"@babel/preset-stage-0@7.0.0-beta.44","children":[],"hint":null,"color":"bold","depth":0},{"name":"babel-eslint@8.2.3","children":[],"hint":null,"color":"bold","depth":0},{"name":"babel-loader@8.0.0-beta.2","children":[],"hint":null,"color":"bold","depth":0},{"name":"eslint@4.19.1","children":[{"name":"debug@3.1.0","children":[],"hint":null,"color":"bold","depth":0},{"name":"doctrine@2.1.0","children":[],"hint":null,"color":"bold","depth":0}],"hint":null,"color":"bold","depth":0}]}}

Additional Data

• Serverless-Webpack Version: 5.1.5
• Webpack version: 4.8.1
• Serverless Framework Version:1.27.2
• Operating System:ubuntu 16.04 LTS

[HyperBrain]

Hi @ArpitaSrivas that's interesting. Do you know where the "activityTick" entries are coming from? Is Yarn configured differently on your linux system than on macOS? Can you do some tests and figure out, if we need some additional switch to get the linux output the same as on macOS?

For me this looks like the output of yarn list contains progress information. The plugin parses the "tree" information which is the actual dependency tree.

Additionally, can you post the yarn versions for the systems? Maybe they are different too.

[ArpitaSrivas]

Hi @HyperBrain,
yarn versions on both the systems are same (1.6.0)
Also, interestingly, the command outputs same 'activityTick' entries on both macOs and linux system, when run separately on terminal. But when the serverless process is running, only type: "tree" entry is output on the macOS, but linux system obtains the same output as mentioned above.

I have tested this in mulitple mac systems where it works fine.
Tested this in another linux system, also worked fine. Seems like an issue with my system alone. Do you have any idea what it could be?

Thanks.

[HyperBrain]

Hi @ArpitaSrivas , honestly I have not really a clue what's happening. Just guessing, it could be something with the environment, the serverless command is executed on the failing linux machine, i.e. that the user account it is executed with, has a different PATH, so that a different (?) yarn executable is run, or that there is some env var set in that case that triggers the strange behavior.
However, you could add a log of the process.env into the dependency function in packagers/yarn.js where the command is executed and compare the contents of the environment.

[HyperBrain]

@ArpitaSrivas Did you find out anything?

[ArpitaSrivas]

Hi @HyperBrain, I didn't get a chance to look into it further, will try to get back on this soon.

[HyperBrain]

@ArpitaSrivas Any update here?

[kziemski]

@ArpitaSrivas @HyperBrain Hi i just got hit with this problem as well had to switch back to npm.
yarn prints invalid json
this references the problem. i'm trying to see if its some kind of environment variable that causes these extra json to be added.

[kziemski]

@ArpitaSrivas @HyperBrain I temporarily solved it with two things.
somehow there's an older version of yarn somewhere in my path so i moved yarn up to the beginning and that worked. but the problem can still occur. i can't find it but its was definitely there. if the yarn list ---json .... works in command line but doesn't work inside the plugin thats the cause.

Separately, i think you need to add these flags '--no-progress' '--silent' to lib/packagers/yarn.js inside getProdDependencies
This is where in yarn the activity json frags get spit out

reading the background it feels like using yarn list --json is problematic but those two flags i think control the extra output. the other issue that it reports those blobs if 'is-ci' returns true. i'm not sure those flags are good enough alone.

[hassankhan]

Hi all, I think this may be a bug in the plugin after all. It seems an older version of Yarn used to output a single line of JSON, but subsequent versions now use the json-lines format as per this comment. That said, I haven't encountered the problem myself 😕

Yarn: v1.15.2
Serverless: 1.40.0
Serverless-Webpack: 5.2.0

[designfrontier]

The change we would likely need to https://github.com/serverless-heaven/serverless-webpack/blob/master/lib/packagers/yarn.js#L61 to implement json-lines doesn't look terrible? https://www.npmjs.com/package/jsonlines would make it pretty straightforward if it is something that we want to do @hassankhan @HyperBrain