Deployment messes up the dev environment because everything happens in the same folder

[mnapoli]

This is a Bug Report

Description

I am writing a plugin for deploying PHP applications. My plugin basically optimizes caches for production when running serverless deploy.

Expected: the optimized cache files are uploaded to lambda, and my local project is not affected.

Actual: the optimized cache files are generated in my project, are correctly uploaded to lambda but then they are left in my project.

This messes up the dev environment because those production files should not exist in local dev environment.

Idea: duplicate the project in a temporary directory, then execute the "packaging" step in that directory before zipping the directory to create the archive?

[HyperBrain]

Hi @mnapoli ,

Could you explain the problem at a more detailed level? The Serverless framework uses a temporary directory .serverless that is used to build and package everything. Everything in there should be treated only for the specific package/deploy that is currently running.
Additionally you can add a --package=<directory> to deploy and package to have the output persisted, so that the deploy can fetch it again and continue with that.

As I understand, you're writing a plugin - if your plugin creates and owns the cache files, it is also its responsibility to keep them somewhere separated by stages.

[mnapoli]

Thanks for the detailed answer I really appreciate it. Here is an example of the plugin I'm writing:

class PhpServerlessPlugin {
    constructor(serverless, options) {
        this.hooks = {
            'before:package:createDeploymentArtifacts': this.build.bind(this),
        };
    }
    build() {
        execSync('composer install --no-dev', {stdio: 'inherit' });
    }
}

Composer is a bit like NPM for PHP.

What this does is optimize PHP dependencies (basically node_modules) for production by removing dev dependencies.

The command seems to be run in the directory of the project because after serverless deploy my dev dependencies are gone.

This will be a problem if the PHP plugin "messes up" the dev environment on every deploy.

The Serverless framework uses a temporary directory .serverless that is used to build and package everything. Everything in there should be treated only for the specific package/deploy that is currently running.

Maybe I'm missing something here but it seems that .serverless directory is only meant to contain the ZIP file?

[HyperBrain]

Tip: When writing a plugin you should return a promise in any case as your hook, so that the action invoked there is awaited properly - and change to execution to use exec instead of execSync to keep system resources unblocked.

const BbPromise = require('bluebird');
const { exec } = require('child_process');

class PhpServerlessPlugin {
    constructor(serverless, options) {
        this.serverless = serverless;
        this.options = options;
        this.hooks = {
            'before:package:createDeploymentArtifacts': () => BbPromise.bind(this)
              .then(this.build),
        };
    }
    build() {
      return BbPromise.fromCallback(cb => {
        exec(
          'composer install --no-dev',
          { cwd: XXXXXXX },
          cb
        );
      });
    }
}

The question now is, where (in which directory) do you want to execute the composer command? You just have to set the cwd option in the exec command above to the CWD for composer. You can set that to anything, derive it from some variable in serverless.yml (e.g. this.serverless.service.custom.XXXXX) or from the options (this.options.XXXXX).

.serverless directory is only meant to contain the ZIP

Somehow. Everything contained in the .serverless directory will be persisted if you use deploy and package separately with the --package option, but only the zip files will be uploaded.

[mnapoli]

where (in which directory) do you want to execute the composer command?

Right! It should be executed at the project root so it generates the caches correctly, but in a temporary directory so that the dev environnement is not affected.

Basically composer install --no-dev is like npm install --only=production, it is meant to install the dependencies for production. Does that make sense? I don't want that command to change my local project.

Imagine it were a C application: I would like to upload the compiled executable to the lambda, not the source code. And I don't want the executable to be generated in my project because my project is a dev environment.

So I want to generate an archive and create and edit files (in my plugin) during the deployment process without affecting the actual project.

As such, I think serverless deploy (or package) should create a temporary directory, copy all the files of the project inside it, and then zip that temp directory. That way plugins can mess things up as they want, the original project is not impacted.

---

Trying to explain it another way:

git clone https://github.com/my/project
cd project/
npm install
# […] work on the project
serverless deploy # that will run `npm install --only=production`
# try to work again on the project => ERROR, dev dependencies are not installed anymore!

[HyperBrain]

@mnapoli Understood what you what to achieve 😄 .

In this case I would go s slightly different way - do not force the framework to do things differently, but use a plugin, that already does the packaging in a way that you want and let your plugin hook into that plugin.

I'm talking about serverless-webpack here! That plugin already provides exactly the packaging algorithm that you want. It compiles the code and installs the dependencies separately, rooted in a completely separate directory hive. Since version 4.x it implements it's own extension points, so you can hook a plugin directly into it.

That should be quite easy: The project will create an independent dependency hive at ./.webpack/dependencies that contains a complete package JSON and a node_modules folder. If your plugin now hooks after:webpack:package:packExternalModules it is guaranteed that the dependencies there are complete, and it just should execute the compose command in that directory or even in ./.webpack to modify the temporary dependencies.
You even can plain copy files into the output if that is needed as webpack supports such scenarios out of the box.
After your plugin has been run, the webpack plugin will finally create the zip files.

I think this is the easiest and fastest way to achieve exactly what you want to achieve, without having implementations in the Serverless framework for a specific rare problem.

[mnapoli]

OK thank you, so to summarize before closing the issue:

• this need is outside of the scope of serverless and it needs to be implemented as a plugin or something else outside this project
• option 1: do it myself with a plugin by copying the logic of serverless-webpack
• option 2: use serverless-webpack (even if I don't use webpack) and use its hooks to do what I want
• option 3: encapsulate serverless with my own CLI tool that would create the temp directories (e.g. phplambda deploy that would basically run serverless deploy in a temp directory)

Thanks!

[mnapoli]

I have had a look at all those options and the first seems best. I had a good lock at serverless-webpack but it's just too complicated and out of scope for PHP users.

Is there a simpler solution than what serverless-webpack is doing (for my need)?

Do you think it is doable to write a plugin that would:

• create a temp directory
• copy all the project's files into that temp directory
• change the current working directory of the Node process to that temp directory (so that the serverless command works in this directory instead of the project's directory

? (I'm trying to gather your opinion before trying it, I spend so much time on that that I figure maybe you can save me a few days of work :p)

If that is doable, which before event do you think I should do that?

That would be simpler than option 3 (which means completely hiding serverless behind a new and custom CLI tool, which is not ideal at all).

---

Also to be honest the current behavior is (IMO) very limiting. Even for JS applications it makes no sense to generate production artifacts inside the development environment. All the logic you have in serveless-webpack is a good example: it could be much simpler than that. All deployment tools work in a temporary directory in order to create the production artifact.