Sudden issue with NodeJs Application (AWS) : terminate called after throwing an instance of 'std::bad_alloc'

[JonathonJulian]

Hello, we have a recent issue where a NodeJS application that was previously working fine is now throwing the error:
terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc signal 6 (no core generated: limit 0

I have ruled out the following.

1. Ops version. This seems to be happening across versions including the code on master branch, I am currently testing with
   Ops version: 0.1.37 Nanos version: 0.1.45

2. The application. I am getting identical behaviour with this test application which was also previously working fine.

3. Memory resources, the test app is lightweight and i have tried increasing memory allocation on the actual app, the error remains.

Running the image locally with ops pkg load -c config.ops eyberg/node:v18.12.1 works as expected.
I am generating the AMI with ops image create -t aws -c config.ops --package eyberg/node:v18.12.1

It fails with NightlyBuild as true and false

Any assistance to help isolate this issue would be greatly appreciated

[eyberg]

ok, so looks like there are 2 issues here:

1. the NightlyBuild flag looks like it's being ignored via pkg load currently (but -n toggles it correctly)
   (you can verify this via -v --show-debug and see which kernel it is building with)

2. the issue you are running into I see on 0.1.45 but not nightly - can you try with ops pkg load -c config.ops eyberg/node:v18.12.1 -n (w/the cli flag not the option inside config)?

[JonathonJulian]

I will try but just to be clear it's the AMI that's failing, load works as expected.

[JonathonJulian]

ok ops image create -t aws -c config.ops --package eyberg/node:v18.12.1 -n worked for the test app, im trying the real app now. any ideas why this would suddenly occur without any changes being made on our end?

[JonathonJulian]

i also noticed that stdout seems to stop working in this case, is serial console off by default with the latest?

[eyberg]

https://github.com/nanovms/ops/pull/1484/files fixes the first issue I noted - it was unconditionally overriding the nightly var

the nightly flag you were using in your config wasn't actually being toggled correctly - that pr fixes that

as for serial output, no it's not off by default, although we did discuss doing that when syslog is enabled
master...disableSerialVGA

on aws it depends on the instance type - on nitro systems like your c6a's you actually have to be connected to the console to see the output - that is if you send a bunch of output and then connect you won't see it; for this particular target/instance type it's best to ship to a syslog or something like that

[JonathonJulian]

@eyberg is it possibility to set the specific nanos version in the ops config? Nailing down both versions i think would prevent this from happening again.

[francescolavra]

Setting the nanos version in the ops configuration file is currently not supported, but you can set it in the ops command line, wih the --nanos-version option. Example:
ops image create -t aws -c config.ops --package eyberg/node:v18.12.1 --nanos-version 0.1.44

[JonathonJulian]

that works, thanks!