Webpack does not run before tests with serverless-jest-plugin

[0o-de-lally]

This is a Documentation issue

Description

(I'm assuming this is a documentation issue, but it may be a feature if this case has not been contemplated.)

I'm trying to ensure that webpack runs before the testing plugin. Currently I'm using the https://github.com/SC5/serverless-jest-plugin

It appears tests are running the uncompiled code. This means I'm not catching all the syntax errors lambda runtime does not support.

Reproduction

It's a bit tricky to reproduce this error, since you'd have to compare the jest test run results to that of a deployed function.
You can break the webpack integration by breaking the config file :

module.exports = {
  entry: null,
...

The test runner will not catch this error, which implies that webpack is not running before the tests.
The only way to catch this is by running serverless webpack serve

Additional Data

package.json

    "babel-core": "^6.13.2",
    "babel-loader": "^6.2.4",
    "babel-plugin-transform-runtime": "^6.12.0",
    "babel-polyfill": "6.13.0",
    "babel-preset-es2015": "^6.13.2",
    "babel-preset-stage-0": "^6.5.0",
    "serverless-jest-plugin": "^0.1.5",
    "serverless-webpack": "^1.0.0-rc.3",
    "webpack": "^1.13.1"

serverles.yml

plugins:
  - serverless-jest-plugin
  - serverless-webpack

[HyperBrain]

Can you post the plugins section from you serverless.yml? The order there defines the execution order of the plugins. It would be great to know, if the test plugin is set before or after the webpack plugin.

[0o-de-lally]

Thanks @HyperBrain this is the order of plugins.

plugins:
  - serverless-jest-plugin
  - serverless-webpack

[0o-de-lally]

It's a bit tricky to reproduce this error, since you'd have to compare the jest test run results to that of a deployed function.
You can break the webpack integration by breaking the config file :

module.exports = {
  entry: null,
...

The test runner will not catch this error, which implies that webpack is not running before the tests.
The only way to catch this is by running serverless webpack serve

[HyperBrain]

Did you try to reverse the order of the plugins already, so that webpack comes before jest? The reason I ask, is that Serverless executes the plugin integrations exactly in the order how they are specified in the serverless.yml.
If you add webpack AFTER jest, it could be that jest operates on uncompiled sources.

If a reverse order does not help, I will try to check the sources of the jest plugin and see how exactly it integrates with serverless.

[0o-de-lally]

@HyperBrain Yes I've tried both orders, the results are the same. Webpack seems to not be running before the test suite. Thanks in advance.

[HyperBrain]

Then this is definitively a bug (although I'm not sure where - in the jest plugin or here) that has to be fixed after a deeper analysis. I'll compare the hooks that are used by both plugins and check if the hooks can be adjusted in one of the plugins to make it work with a well-defined execution order.
If anyone else could dive into that too, would be very appreciated 😃

[HyperBrain]

Just thought about easier reproduction steps. It should be reproducible, if you use e.g. Node 4 and include some ES6 code, that is not understood natively by the Node version.
If it runs on uncompiled code then, it should bail out.

Maybe using the "import and export" syntax or destructuring should do the trick (see https://kangax.github.io/compat-table/es6/)

[hassankhan]

Just had a look at serverless-jest-plugin's hooks from here, and it doesn't look like it binds to any event that serverless-webpack uses (I might be wrong).

Do you run your tests through the plugin or directly with Jest, @lpgeiger?

[HyperBrain]

@hassankhan Yes, you're right. The jest plugin commands do not trigger package:compileFunctions (see https://gist.github.com/HyperBrain/50d38027a8f57778d5b0f135d80ea406) at all, so invoking the plugin through one of its commands won't trigger any build. In fact the plugin won't work with any other plugin that adds a special handling to function compilations.

A possible solution would be to change the jest plugin's lifecycle as follows here. Or in the first event run through with all commands:

'create:function:create': () =>
  BbPromise.bind(this)
   .then(() => this.serverless.pluginManager.spawn('package'))    // INSERT THIS
   .then(() => createFunction(this.serverless, this.options))
   .then(() => createTest(this.serverless, this.options)),

Then the test function created here could be built from the function compiled with Webpack
which is located in .serverless at that point of time.

I'm not sure if the whole package lifecycle has to be started or the compileFunctions would be sufficient. For an initial test I would go with the package.

So this is not anything that can be solved here, but has to be solved in the jest plugin.
Maybe I can spare some time to fix it...

[hassankhan]

At @EndemolShineGroup, we only have integration tests for our handler functions. We keep the handlers as thin as possible, and shift logic to a src/ folder, which is unit-tested.

To run the tests, we invoke Jest directly. For integration tests, we deploy before (since we need an environment to actually test in). We find that running any sort of integration tests locally tends to mostly be a waste of time because of all the various differences between a Lambda environment and a local developer environment (different Node versions, no Cognito auth support, how to test SNS lambdas etc.). Of course, this does mean a bit of extra setup and some attention-to-detail when setting up the CI/CD pipeline but it can definitely be done 👍 .

What I'm trying to say is ... maybe you can give it a go without the plugin, @lpgeiger 😄

[0o-de-lally]

@HyperBrain thanks for identifying this. Are there ANY serverless test suites that would work out of the box?

@hassankhan thanks for the input.

We find that running any sort of integration tests locally tends to mostly be a waste of time

Yes that's what I'm finding. It seems strange that none of the local emulation solutions package the correct target runtime.
I don't think I'd be comfortable doing integration tests only as you describe. That would be fine for simple resources, but more complex apps would need unit tests.

[HyperBrain]

@lpgeiger Maybe mocha-webpack would work for the unit tests - you can specify a webpack config. I did not find nor do I know anything that directly integrates with Serverless.

[HyperBrain]

I'll remove the bug tag, as it is obviously no bug for this project. I'll close it as soon as the discussion in here has been finalized - feel free to continue discussing even after that 😄

[0o-de-lally]

@HyperBrain Thanks for looking into it. I'm not familiar with the serverless plugin system. Can you give me a hint about how one might implement a hook into 'serverless-webpack' the test runner I'm using: 'serverless-jest-plugin' ? Thanks in advance,

[0o-de-lally]

@HyperBrain I See that the issues is fixed in serverless-webpack, it seems to work in my tests. Can you confirm that your implementation is working too? Thanks in advance!

[HyperBrain]

@lpgeiger Just saw that there were fixes done in the SC5 plugins (mocha plugin). I'm not sure if I can double check it in the next days. Good, if it works for you. Maybe someone else can try it too.
We can leave this task open for some time until we're really sure that it works now.

[HyperBrain]

I'll close this one for now. There was no additional feedback for a long time.
Feel free to continue discussion here, if there's anything left to mention.