CLI won't run in AWS Lambda

[kokokenada]

Bug description

npx prisma -v when invoked in an AWS Lambda fails with 'Cannot find module '../package.json'

How to reproduce

Try a Lambda such as:

export const handler: APIGatewayProxyHandler = async (event, context) => {
  let retVal;
  try {
    if (event?.queryStringParameters?.secret !== process.env['DB_DEPLOY_SECRET']) {
      console.warn(`Invalid secret given ${event?.queryStringParameters?.secret}`);
      return {
        statusCode: 401,
        body: JSON.stringify({ message: `missing or invalid secret` }),
      };
    }

    console.log(`process.cwd()=${process.cwd()}`);

    process.env['PATH'] = process.env['PATH'] + ':' + process.env['LAMBDA_TASK_ROOT'];
    // https://aws.amazon.com/blogs/compute/running-executables-in-aws-lambda/

    console.log('process.env:');
    console.log(process.env);
    console.log('running ./db-deploy.sh');

    const { execSync } = require('child_process');
    console.log('ls -al');
    console.log(execSync('ls -al').toString());

    console.log('node -v');
    console.log(execSync('node -v').toString());

    console.log('npx -v');
    console.log(execSync('npx -v').toString());

    console.log('id');
    console.log(execSync('id').toString());

    console.log('ls -al node_modules');
    console.log(execSync('ls -al node_modules').toString());

    console.log('ls -al node_modules/.bin');
    console.log(execSync('ls -al node_modules/.bin').toString());

    console.log('ls -al node_modules/.prisma');
    console.log(execSync('ls -al node_modules/.prisma').toString());

    console.log('ls -al node_modules/.prisma/client');
    console.log(execSync('ls -al node_modules/.prisma/client').toString());

    // console.log('cp "node_modules/@prisma/client/package.json" node_modules/');
    // console.log(execSync('cp "node_modules/@prisma/client/package.json" node_modules/').toString());

    console.log('npx prisma -v');
    console.log(execSync('npx prisma -v').toString());

Expected behavior

Prisma information

Environment & setup

• Prisma version: 2.1.3

[pantharshit00]

I can reproduce this with the following function deployed via the a zip to AWS lambda:

exports.handler = async (event, context) => {
  console.log(`process.cwd()=${process.cwd()}`);

  process.env["PATH"] =
    process.env["PATH"] + ":" + process.env["LAMBDA_TASK_ROOT"];
  // https://aws.amazon.com/blogs/compute/running-executables-in-aws-lambda/

  console.log("process.env:");

  const { execSync } = require("child_process");
  console.log("ls -al");
  console.log(execSync("ls -al").toString());

  console.log("node -v");
  console.log(execSync("node -v").toString());

  console.log("npx -v");
  console.log(execSync("npx -v").toString());

  console.log("id");
  console.log(execSync("id").toString());

  // console.log('cp "node_modules/@prisma/client/package.json" node_modules/');
  // console.log(execSync('cp "node_modules/@prisma/client/package.json" node_modules/').toString());

  console.log("npx prisma -v");
  console.log(execSync("npx prisma -v").toString());
  return { ok: "test" };
};

package.json:

{
  "name": "prisma-test",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "dependencies": {
    "@prisma/cli": "^2.3.0-dev.30",
    "@prisma/client": "^2.3.0-dev.30"
  },
  "devDependencies": {},
  "scripts": {
    "start": "ts-node main.ts",
    "generate": "prisma generate"
  }
}

Stacktrace:

 "Error: Command failed: npx prisma -v",
        "Cannot find module '../package.json'",
        "Require stack:",
        "- /var/task/node_modules/.bin/prisma",
        "",
        "    at checkExecSyncError (child_process.js:630:11)",
        "    at execSync (child_process.js:666:15)",
        "    at Runtime.exports.handler (/var/task/index.js:27:15)",
        "    at Runtime.handleOnce (/var/runtime/Runtime.js:66:25)"

Still @kokokenada I would like to know what is your use case of executing the cli inside of lambda.

[mattmanske]

same happens in EC2 (Amazon Linux 2 AMI) - the Cannot find module '../package.json' error seems to happen for every Prisma CLI command I try to run

[mattmanske]

Hopefully this is helpful - I was able to get this working by directly referencing the build file:

ie: change this:
npx prisma generate
to:
./node_modules/@prisma/cli/build/index.js generate

[kokokenada]

Still @kokokenada I would like to know what is your use case of executing the cli inside of lambda.

I'm using GitLab CICD to use the Serverless framework to deploy the API to AWS Lambda. As part of the deployment process, I want to run the the upgrade (prisma migrate). The DB is Aurora and has no external IP address, so I thought I'd create a Lambda so I could reach the database.

[timsuchanek]

Thanks for all the information everyone here!

We looked into the issue and while it's tricky to get anything related to the filesystem running in Lambda, because it has a read-only filesystem, we got it running!
There were a few adjustments necessary, which landed in @prisma/cli@2.4.0-dev.29.

In order for the CLI to work in lambda, please already install it locally on your machine and upload it to the lambda function. Getting npx running did not work in our tests, as it needs to write to the filesystem.

To get the right binaries downloaded for Lambda, you need to provide the PRISMA_CLI_BINARY_TARGETS env var during npm install:

PRISMA_CLI_BINARY_TARGETS=darwin,rhel-openssl-1.0.x npm install

This env var is unstable and might be removed. It's just here for now to validate the use-case.
Note, that you also need to provide your local platform, in my case darwin.

Running execSync did not work to fork the command in my tests, instead it worked with execa:

const execa = require('execa')

module.exports.hello = async (event) => {
  const { stdout, stderr } = await execa.command('node node_modules/@prisma/cli/build/index.js generate')
  console.log('stdout', stdout)
  console.log('stderr', stderr)

  return {
    statusCode: 200,
    body: JSON.stringify(
      {
        message: 'Gó Sørverless v1.0! Your function executed successfully!',
        input: event,
        env: process.env,
      },
      null,
      2,
    ),
  }
}

Please let us know if this works for you. If not, we can reopen the issue.

[Jolg42]

I think an interesting approach would be using Lambda layers and create one for the Prisma CLI and upload it with the "rhel-openssl-1.0.x" binary.
So the Prisma CLI / node_modules folder is only uploaded once and can be used in other functions easily.

[RokoTechnology]

having the same issue trying to run execSync('npx prisma db push') or execSync('./node_modules/.bin/prisma db push') in openfaas / kubernetes node.js function. the database is not accessible from the outside. i want to sync the database schema in production when starting my function.

[janpio]

Open a new issue or discussion about this please. Provide all the information the issue template asks for.

[RokoTechnology]

Fixed my issue, leaving solution for future google searches:

The script assumes your package.json is on the same level as your entry point index.js

If your main is nested like "src/index.js" it won't work.

[Sina-Veo]

Hopefully this is helpful - I was able to get this working by directly referencing the build file:

ie: change this: npx prisma generate to: ./node_modules/@prisma/cli/build/index.js generate

As the prisma/cli is deprecated the following worked for me:

node node_modules/prisma/build/index.js migrate

[jsonpj3wt]

Talk about a massive headache with this.

After fighting this for over a week, PRISMA_CLI_BINARY_TARGETS is very much still needed. Make sure to also set your binaryTargets in your schema accordingly.

PRISMA_CLI_BINARY_TARGETS: rhel-openssl-1.0.x,debian-openssl-1.1.x
binaryTargets = ["native", "rhel-openssl-1.0.x", "debian-openssl-1.1.x"]

This is to get migrations running from Lambda, and honestly, this should be documented (because in a secure environment, you should not be running db calls outside the VPC - some companies do have good networking to allow a secure VPN, but that is a dependency and doing it this way reduces risks). I am wondering if I even need to specify debian at all anymore, but here is the lambda code for anyone else running into this problem:

import { PrismaClient } from '<your lib>/prisma/generated';
import { execFile, execSync } from 'child_process';
import * as dotenv from 'dotenv';
import path from 'path';

dotenv.config();

/**
 * Definition of the event this lambda expects to process.
 */
interface migrationEvent {
    /**
     * Optional migration name to run.  If not specified, the lambda will run all migrations.
     */
    migrationName?: string;
}

/**
 * Handler for the AWS Lambda.
 * @param event The event payload to process.
 * @param context The context.
 * @returns a promise containing the results of processed events.
 */
export const handler = async (event?: migrationEvent, context?: null) => {
    console.log(`Handling new event: ${JSON.stringify(event, undefined, 2)}`);

    const prisma = new PrismaClient({ datasources: { db: { url: process.env.DATABASE_URL } } });

    try {
        await prisma.$connect();
        console.log('Connected to the database.');
        await prisma.$disconnect();
    } catch (error) {
        console.error('Error connecting to the database:', error);
        process.exit(1);
    }

    console.log('Running migrations...');

    let command = 'deploy';

    const prismaBuild = process.env.PRISMA_BUILD_LOCATION;

    const exitCode = await new Promise((resolve, _) => {
        const args: string[] = ['migrate', command, '--schema', process.env.PRISMA_SCHEMA_LOCATION || ''];
        if (event?.migrationName?.length) {
            args.push('--name', event.migrationName);
        }
        execFile(
            path.resolve(prismaBuild || './node_modules/prisma/build/index.js'),
            args,
            (error, stdout, stderr) => {
                console.log(stdout);
                if (error != null) {
                    console.log(`prisma migrate ${command} exited with error ${error.message}`);
                    resolve(error.code ?? 1);
                } else {
                    console.log(error, stderr);
                    resolve(0);
                }
            },
        );
    });

    if (exitCode != 0) throw Error(`command ${command} failed with exit code ${exitCode}`);
    });
    return {
        statusCode: 200,
    };
};

if (require.main === module) {
    handler({}).catch(console.error);
}