M1 Mac (Apple Silicon): Debugging SAM .NET Function not working

[mattgillard]

When debugging the simple SAM .NET 3.1 or 2.1 hello world function you get a pop up error "No process with the specified name is currently running". And a timeout error: 2021-03-13 17:19:51 [ERROR]: Unable to attach Debugger. Check AWS Toolkit logs. If it took longer than expected to start, you can still attach.

Steps to reproduce

1. run sam init

$ sam init
Which template source would you like to use?
	1 - AWS Quick Start Templates
	2 - Custom Template Location
Choice: 1
What package type would you like to use?
	1 - Zip (artifact is a zip uploaded to S3)
	2 - Image (artifact is an image uploaded to an ECR image repository)
Package type: 1

Which runtime would you like to use?
	1 - nodejs14.x
	2 - python3.8
	3 - ruby2.7
	4 - go1.x
	5 - java11
	6 - dotnetcore3.1
	7 - nodejs12.x
	8 - nodejs10.x
	9 - python3.7
	10 - python3.6
	11 - python2.7
	12 - ruby2.5
	13 - java8.al2
	14 - java8
	15 - dotnetcore2.1
Runtime: 6

Project name [sam-app]:

Cloning app templates from https://github.com/aws/aws-sam-cli-app-templates

AWS quick start application templates:
	1 - Hello World Example
	2 - Step Functions Sample App (Stock Trader)
Template selection: 1

    -----------------------
    Generating application:
    -----------------------
    Name: sam-app
    Runtime: dotnetcore3.1
    Dependency Manager: cli-package
    Application Template: hello-world
    Output Directory: .

    Next steps can be found in the README file at ./sam-app/README.md

2. cd sam-app
3. code .
4. Create debug configuration for src/HelloWorld/Function.cs
5. Click run and debug
6. Choose sam-app:HelloWorldFunction configuration and press green play button
7. Wait for config and docker container to start
8. Error pops up.
9. however container is running:

 sam-app docker ps
CONTAINER ID   IMAGE                                                           COMMAND                  CREATED              STATUS              PORTS                                                NAMES
07cf7754be6d   amazon/aws-sam-cli-emulation-image-dotnetcore3.1:rapid-1.20.0   "/var/rapid/aws-lamb…"   About a minute ago   Up About a minute   127.0.0.1:5860->5860/tcp, 127.0.0.1:6046->8080/tcp   silly_noether

10. Can connect to container fine and ls:

sam-app docker exec -i $(docker ps -q -f publish=5860) sh
ls
Amazon.Lambda.APIGatewayEvents.dll
Amazon.Lambda.Core.dll
Amazon.Lambda.Serialization.Json.dll
HelloWorld.deps.json
HelloWorld.dll
HelloWorld.pdb
HelloWorld.runtimeconfig.json
Newtonsoft.Json.dll

Expected behavior

I expect a debug session to start successfully which i confirmed works fine on an Intel Mac.

Screenshots

Desktop (please complete the following information):

• OS:
• Visual Studio Code Version: Version: 1.54.2
  Commit: fd6f3bce6709b121a895d042d343d71f317d74e7
  Date: 2021-03-11T00:56:09.783Z
  Electron: 11.3.0
  Chrome: 87.0.4280.141
  Node.js: 12.18.3
  V8: 8.7.220.31-electron.0
  OS: Darwin x64 20.3.0
• AWS Toolkit Version: 1.20.0
  Also running latest docker M1 pre-release Docker Desktop preview 3.1.0

Additional context
Tried both x64 and arm64 modes for vs code. Same result.
I am wondering if it is related to: dotnet/vscode-csharp#4277

logs.txt
Extra logging debug:

Starting: "sh" -c "docker exec -i $(docker ps -q -f publish=5860) /tmp/lambci_debug_files/vsdbg --interpreter=vscode"
-> (C) {"command":"initialize","arguments":{"clientID":"vscode","clientName":"Visual Studio Code","adapterID":"coreclr","pathFormat":"path","linesStartAt1":true,"columnsStartAt1":true,"supportsVariableType":true,"supportsVariablePaging":true,"supportsRunInTerminalRequest":true,"locale":"en-gb","supportsProgressReporting":true,"supportsInvalidatedEvent":true},"type":"request","seq":1}
-> (C) {"command":"attach","arguments":{"type":"coreclr","request":"attach","name":"sam-app:HelloWorldFunction","logging":{"engineLogging":false,"trace":true},"invokeTarget":{"target":"template","templatePath":"/Users/matt/dev/github/sam-app/template.yaml","logicalId":"HelloWorldFunction"},"lambda":{"payload":{},"environmentVariables":{},"timeoutSec":10},"__configurationTarget":5,"codeRoot":"/Users/matt/dev/github/sam-app","workspaceFolder":{"uri":{"$mid":1,"fsPath":"/Users/matt/dev/github/sam-app","external":"file:///Users/matt/dev/github/sam-app","path":"/Users/matt/dev/github/sam-app","scheme":"file"},"name":"sam-app","index":0},"runtime":"dotnetcore3.1","runtimeFamily":3,"handlerName":"HelloWorld::HelloWorld.Function::FunctionHandler","documentUri":{"$mid":1,"fsPath":"/Users/matt/dev/github/sam-app/.vscode/launch.json","external":"file:///Users/matt/dev/github/sam-app/.vscode/launch.json","path":"/Users/matt/dev/github/sam-app/.vscode/launch.json","scheme":"file"},"templatePath":"/tmp/aws-toolkit-vscode/vsctk47aQPw/output/template.yaml","eventPayloadFile":"/tmp/aws-toolkit-vscode/vsctk47aQPw/event.json","envFile":"/tmp/aws-toolkit-vscode/vsctk47aQPw/env-vars.json","debugPort":5860,"useIkpdb":false,"baseBuildDir":"/tmp/aws-toolkit-vscode/vsctk47aQPw","debuggerPath":"/Users/matt/dev/github/sam-app/src/HelloWorld/.vsdbg","processName":"dotnet","pipeTransport":{"pipeProgram":"sh","pipeArgs":["-c","docker exec -i $(docker ps -q -f publish=5860) ${debuggerCommand}"],"debuggerPath":"/tmp/lambci_debug_files/vsdbg","pipeCwd":"/Users/matt/dev/github/sam-app/src/HelloWorld"},"__sessionId":"8196f443-def6-4632-8ed6-909792d9890b"},"type":"request","seq":2}
<- (E) {"seq":3,"type":"event","event":"output","body":{"category":"telemetry","output":"VS/Diagnostics/Debugger/vsdbg/AttachFailed","data":{"VS.Diagnostics.Debugger.vsdbg.AdapterId":"coreclr","VS.Diagnostics.Debugger.vsdbg.ErrorCode":2001,"VS.Diagnostics.Debugger.vsdbg.Distribution.Name":"bc4f1536","VS.Diagnostics.Debugger.vsdbg.Distribution.Version":"2","VS.Diagnostics.Debugger.vsdbg.OSFamily":"Linux","VS.Diagnostics.Debugger.vsdbg.Version":"16.9.20122.2 commit:c3006f59a1d3a160c9fc06f5e2e14402522190e3"}}}
<- (R) {"seq":4,"type":"response","request_seq":2,"success":false,"command":"attach","message":"No process with the specified name is currently running."}

[justinmk3]

Thanks for the report!

1. Based on your logs, sam build succeeds so probably not related to sam build gets killed on Mac with Apple Silicon aws-sam-cli#2656
2. Looks like /tmp/lambci_debug_files/vsdbg (mounted in-container from /Users/.../sam-app/src/HelloWorld/.vsdbg on the host) is successfully running.
3. The attach command then looks for "processName":"dotnet" in the container... which is not found.

What does ps report from the container? To observe it without having a debugger attached,

1. Edit the handler code to return the output of ps -e :

    public APIGatewayProxyResponse FunctionHandler(APIGatewayProxyRequest apigProxyEvent, ILambdaContext context)
    {
        var proc = new System.Diagnostics.Process {
            StartInfo = new System.Diagnostics.ProcessStartInfo {
                FileName = @"/bin/ps",
                Arguments = "-e",
                UseShellExecute = false,
                RedirectStandardOutput = true,
            }
        };
        proc.Start();
        var output = proc.StandardOutput.ReadToEnd();
        proc.WaitForExit();
   
        var body = new Dictionary<string, string> {
            { "message", output },
        };
   
        return new APIGatewayProxyResponse {
            Body = JsonConvert.SerializeObject(body),
            StatusCode = 200,
            Headers = new Dictionary<string, string> { { "Content-Type", "application/json" } }
        };
    }
   

2. Invoke the lambda without debugging using either:
   • From shell: sam local invoke ...
   • From VSCode, the Debug: Start Without Debugging command (after selecting the correct launch config in the VSCode Run panel)

[mattgillard]

The plot thickens. There is no ps in the container....(!)

$ docker ps
CONTAINER ID   IMAGE                                                           COMMAND                  CREATED          STATUS          PORTS                                                NAMES
ad0358050730   amazon/aws-sam-cli-emulation-image-dotnetcore3.1:rapid-1.20.0   "/var/rapid/aws-lamb…"   16 seconds ago   Up 14 seconds   127.0.0.1:5858->5858/tcp, 127.0.0.1:8336->8080/tcp   flamboyant_mclaren
$ docker exec -i $(docker ps -q -f publish=5858) ps
OCI runtime exec failed: exec failed: container_linux.go:370: starting container process caused: exec: "ps": executable file not found in $PATH: unknown
$ docker exec -i $(docker ps -q -f publish=5858) sh
ps
/usr/bin/sh: line 1: ps: command not found
/bin/ps
/usr/bin/sh: line 2: /bin/ps: No such file or directory

I did a docker top instead:

docker top ad0358050730
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                6777                6751                0                   11:12               ?                   00:00:00            /usr/bin/qemu-x86_64 /var/rapid/aws-lambda-rie --log-level error /var/runtime/bootstrap
root                6821                6777                0                   11:12               ?                   00:00:00            /usr/bin/qemu-x86_64 /var/runtime/bootstrap
root                6824                6821                3                   11:12               ?                   00:00:09            /usr/bin/qemu-x86_64 /var/lang/bin/dotnet exec --depsfile --runtimeconfig /var/runtime/Bootstrap.dll --invokeid dummy --flag 0 --throttled 1 --handler HelloWorld::HelloWorld.Function::FunctionHandler --errorcode 2
root                6894                6751                0                   11:12               ?                   00:00:00            /usr/bin/qemu-x86_64 /usr/bin/sh

[justinmk3]

Indeed, both ps and top are missing in the dotnet3.1 image! Can you try cat /proc/*/comm instead:

Edit: I just noticed the CMD column in your "docker top" result. Your result shows /usr/bin/qemu-x86_64 /var/lang/bin/dotnet whereas we normally expect dotnet.

Presumably this is because Docker is running an ARM Linux kernel, and userspace in the container is all running under /usr/bin/qemu-x86_64 ?

Status

We may need to do something more complex similar to the AWS Toolkit for JetBrains: https://github.com/aws/aws-toolkit-jetbrains/blob/0649bbb18c440b23a95e4753d442a32455409ed8/jetbrains-rider/src/software/aws/toolkits/jetbrains/services/lambda/dotnet/DotnetDebugUtils.kt#L224

[mattgillard]

I am not sure what is going on.
Looks like an x86 kernel.

uname -a
Linux 7bd3e77b5b7d 4.19.121-linuxkit #1 SMP PREEMPT Thu Jan 21 15:45:22 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

cat /proc/*/comm
bootstrap
aws-lambda-rie
dotnet
sh

[mattgillard]

ok ... i understand now. The mac docker vm is arm64. Which runs the x86 emulation - which maps to your hypothesis.

docker run -it --rm --privileged --pid=host justincormack/nsenter1
Unable to find image 'justincormack/nsenter1:latest' locally
latest: Pulling from justincormack/nsenter1
726619a9fa8c: Pull complete
Digest: sha256:e876f694a4cb6ff9e6861197ea3680fe2e3c5ab773a1e37ca1f13171f7f5798e
Status: Downloaded newer image for justincormack/nsenter1:latest
/ #
/ # find / -name qemu-x86_64
/var/lib/mount-docker-cache/entries/services.tar/d083ad549ba5be0ab0cb2ecfab00891427b9e55c8383277a12bb0fca18916a35/containers/services/binfmt/rootfs/usr/bin/qemu-x86_64
/var/lib/mount-docker-cache/entries/services.tar/d083ad549ba5be0ab0cb2ecfab00891427b9e55c8383277a12bb0fca18916a35/containers/services/binfmt/lower/usr/bin/qemu-x86_64
/proc/sys/fs/binfmt_misc/qemu-x86_64
/containers/services/binfmt/rootfs/usr/bin/qemu-x86_64
/containers/services/binfmt/lower/usr/bin/qemu-x86_64
/ # uname -a
Linux docker-desktop 4.19.121-linuxkit #1 SMP PREEMPT Thu Jan 21 15:45:22 UTC 2021 aarch64 Linux

[mrBaas]

I´m experiencing the same issue.
Any updates on this?

[cosbor11]

+1

[justinmk3]

consider adding your upvote 👍 to the main description, this helps us decide priorities.

But note that this issue likely needs to be resolved by https://github.com/aws/aws-sam-cli .

[jackmcguire1]

bump

[Jaffacakes82]

Same issue here! 😢

[mick-io]

So sad. bump bump.

[dgokcin]

had the same issue for a dotnet app. no updates?

[CasperM15]

This is blocked on aws/aws-sam-cli#3283 .

Workaround via aws/aws-sam-cli#3389 (comment) is to specify arm64 architecture in template.yaml :

Architectures:
   - arm64

If we specify arm64 architecture in template.yaml then debugger does not work because vsdbg debugger does not currently support the arm64 architecture and the function will run locally without debug.

[justinmk3]

Status

• Blocked until Go Lambda functions can run on ARM. https://docs.aws.amazon.com/lambda/latest/dg/lambda-golang.html
  • See Lambda ARM (Apple M1): AWS SAM Golang lambda debugging does not work aws-sam-cli#3283 (comment)

    the issue is related to the fact that Go Lambda functions can only run on x86_64 architecture (https://docs.aws.amazon.com/lambda/latest/dg/lambda-golang.html), and that Docker Desktop does not support running containers using this architecture (see these issues Mac M1 Ptrace - warning: ptrace: Function not implemented docker/for-mac#5191 (comment), M1 Mac (Apple Silicon): AWS SAM Golang lambda debugging does not work #2020).

• Workaround: use Colima instead of Docker Desktop....
• Workaround (run only, not debug): specify arm64 architecture in template.yaml :

  Architectures:
     - arm64
  

[JD1000]

Can someone please explain why this is blocked until Go Lambda functions can run on ARM? The title specifies that this is related to .NET functions.

I'm seeing this same issue except I'm using .NET 6. It runs if I change the architecture to arm64 but then I can't debug.

[Bogatinovski]

Having the same issue here in 2024. Running a .NET6 function with Architecture set to arm64 on an M1 Macbook Pro works but debugging doesn't work. The process hangs at "Waiting for a debugger to Attach" and then times out. How can I overcome this?