SSM agent is using loads of memory - "cannot allocate memory"

[mtremer]

I am running amazon-ssm-agent on IPFire which was compiled using GCC 9.3.0. I have tried this with various version from 2.3.701.0 - 2.3.930.0 and the problem is always the same.

The process starts and ssm-cli is able to execute normally. The daemon forks into background as usual.

When a command is being sent to the daemon using AWS CLI, the process fails to fork another process and ultimately logs this:

2020-04-09 17:43:42 INFO [MessagingDeliveryService] [EngineProcessor] [BasicExecuter] Sending plugin ssmTest completion message
2020-04-09 17:43:42 INFO [MessagingDeliveryService] [EngineProcessor] sending reply for plugin update: ssmTest
2020-04-09 17:43:42 INFO [MessagingDeliveryService] [EngineProcessor] sending document: 7f16c05d-5dcb-401c-831e-dfb9284f8b65 complete response
2020-04-09 17:43:42 INFO [MessagingDeliveryService] received plugin: ssmTest result from Processor
2020-04-09 17:43:42 INFO [MessagingDeliveryService] Sending reply {
  "additionalInfo": {
    "agent": {
      "lang": "en-US",
      "name": "amazon-ssm-agent",
      "os": "",
      "osver": "1",
      "ver": "2.3.0.0"
    },
    "dateTime": "2020-04-09T17:43:42.856Z",
    "runId": "",
    "runtimeStatusCounts": {
      "Failed": 1
    }
  },
  "documentStatus": "InProgress",
  "documentTraceOutput": "",
  "runtimeStatus": {
    "ssmTest": {
      "status": "Failed",
      "code": 1,
      "name": "aws:runShellScript",
      "output": "\n----------ERROR-------\nfailed to run commands: fork/exec /bin/sh: cannot allocate memory",
      "startDateTime": "2020-04-09T17:43:42.851Z",
      "endDateTime": "2020-04-09T17:43:42.854Z",
      "outputS3BucketName": "",
      "outputS3KeyPrefix": "",
      "stepName": "",
      "standardOutput": "",
      "standardError": "failed to run commands: fork/exec /bin/sh: cannot allocate memory"
    }
  }
}

strace shows that clone() did not succeed as it could not allocate any memory. The agent process is using a lot of memory when it is being started and maps in about 5GB of "virtual memory". The total RSS memory usage is around 99MB. That is a lot more than on a system with Amazon Linux.

Could anyone give me any pointers on how to debug this?

[shihuazhang]

Thanks for reaching out.

We do not support compiling SSM Agent with gccgo at the moment.

If you could provide Agent debug log, it may help me to understand the problem slightly better,
questions for your environment

1. the Go version you are using
2. the maximum processes your sever allows cat /proc/sys/kernel/pid_max
3. the instance type

[mtremer]

Thanks for your reply.

Using the official Go compiler seems to fix it. What is the reason gccgo is incompatible?

[shihuazhang]

The root cause it still not clear to us. We do not have any plan to support this. However, we will monitor the number of requests for supporting gccgo in the future.