Github Actions cancelled for a specific build

[steima]

Why are you starting this discussion?

Question

What GitHub Actions topic or product is this about?

Actions Runner

Discussion Details

We have an application which built (and deployed) just fine from Github Actions. The application is a Serverless API built in TypeScript. It was built from around 20 Serverless microservices and due to a migration we are running right now, we could reduce these microservices down to only 4 service files that we still need to deploy.

What we basically did is we refactored our API to run off of a single serverless.yml file, with a "Lambdalith" model. Where basically a proxy Lambda routes traffic to our business logic. After refactoring we ended up with the following service files (serverless.yml).

• ressources: Just CloudFormation resources that form the infrastructure of the application (deploys fine)
• proxy-api: The Lambdalith proxy (deploys fine)
• attachments-api: Reacts to some S3 events to generate previews of office and PDF documents (deploys fine)
• cron-api: Holds a collection of cron jobs (fails deployment)

We have strong indication that we violate some resource limit of the Actions Runner as the runs jump into state "Cancelled" but are not cancelled by us. Also in the logs we have seen the Signal 137, which indicates the same.

And the final indicator that it has something to do with the Actions Runner is that the same build and deployment runs just fine from local developer machines, with the exact same versions of Serverless, NodeJS and TypeScript.

The error we get from Serverless indicates that the build was stopped / cancelled during esbuild - in the transpiling phase:

      ],
      "error": {
        "name": "ServerlessError2",
        "message": "The service was stopped",
        "stacktrace": "ServerlessError2: The service was stopped\n    at Esbuild._build (file:///home/runner/.serverless/releases/4.19.1/package/dist/sf-core.js:1251:16316)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async before:package:createDeploymentArtifacts (file:///home/runner/.serverless/releases/4.19.1/package/dist/sf-core.js:1251:4787)\n    at async PluginManager.runHooks (file:///home/runner/.serverless/releases/4.19.1/package/dist/sf-core.js:1303:9865)\n    at async PluginManager.invoke (file:///home/runner/.serverless/releases/4.19.1/package/dist/sf-core.js:1303:10575)\n    at async PluginManager.spawn (file:///home/runner/.serverless/releases/4.19.1/package/dist/sf-core.js:1303:10993)\n    at async before:deploy:deploy (file:///home/runner/.serverless/releases/4.19.1/package/dist/sf-core.js:1031:38784)\n    at async PluginManager.runHooks (file:///home/runner/.serverless/releases/4.19.1/package/dist/sf-core.js:1303:9865)\n    at async PluginManager.invoke (file:///home/runner/.serverless/releases/4.19.1/package/dist/sf-core.js:1303:10575)\n    at async PluginManager.run (file:///home/runner/.serverless/releases/4.19.1/package/dist/sf-core.js:1303:11364)\n    at async Serverless.run (file:///home/runner/.serverless/releases/4.19.1/package/dist/sf-core.js:1310:10523)\n    at async runFramework (file:///home/runner/.serverless/releases/4.19.1/package/dist/sf-core.js:1337:1777)\n    at async TraditionalRunner.run (file:///home/runner/.serverless/releases/4.19.1/package/dist/sf-core.js:1333:28331)\n    at async route (file:///home/runner/.serverless/releases/4.19.1/package/dist/sf-core.js:1508:2878)\n    at async Object.run (file:///home/runner/.serverless/releases/4.19.1/package/dist/sf-core.js:1509:3661)\n    at async run2 (file:///home/runner/.serverless/releases/4.19.1/package/dist/sf-core.js:1509:4815)",
        "code": "ESBULD_BUILD_ERROR"
      }
    }
  }
]
s:core:router: Saving meta record to global .serverless directory.
s:cli-error-handler: ServerlessError2: The service was stopped
    at Esbuild._build (file:///home/runner/.serverless/releases/4.19.1/package/dist/sf-core.js:1251:16316)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)

Is there a way to get detailed logging for a runner to see why it cancelled the run?

[steima]

Thanks, this input is indeed very helpful. Your input shows clearly how to retrieve post-mortem information about the build.

I did something similar but did not realize that I could view dmesg.

[steima]

The issue could be resolved, it is was in fact really esbuild getting killed due to resource limitations. The error stems from the way esbuild can be configured in serverless.yml. The project matured from Serverless V2, V3 over to V4 over the years. In V3 esbuild was introduced as an alternative bundler via a plugin and with V4 it became standard.

For V3 the configuration for esbuild lived in custom.esbuild where we correctly set up concurrency: 3 to not exhaust resources on CI pipelines. In V4 these settings moved to build.esbuild, as V4 has slightly different defaults (in our case does generate sourcemaps by default) we did not realize, that the settings in custom.esbuild were not applied. So in fact the faulty build always ran with concurrency default of 10.

We have now moved the settings to the correct keys and adapted them where new names had to be applied.

So in fact this is not a Github Actions issue at all but rather a bug and oversight on our end.

What lead us to believe it had something to do with Github Actions is, that this issue showed up slowly over time. Alreay weeks back we had failed/cancelled builds every now and then but did not think much of it but rather thought perhaps some platform rescaling that hits us. But after a larger migration the issue showed up consistently.