New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Debugging dotnet core apps with the SAM CLI #568

Closed
burck1 opened this Issue Jul 21, 2018 · 45 comments

Comments

Projects
None yet
@burck1
Copy link

burck1 commented Jul 21, 2018

Currently the sam cli does not yet support debugging the dotnet core 2.0/2.1 runtimes with vscode. After #565 / #281 has been merged (thanks @jfuss and @austinlparker), I think it should be pretty straightforward to add in debugging support. I've been using vscode for dotnet core + lambda development a lot recently and this feature would be be very useful.

The basic requirements to attach a remote debugger to dotnet core from the C# extension for vscode is outlined pretty well here: https://github.com/OmniSharp/omnisharp-vscode/wiki/Attaching-to-remote-processes. The main requirement is that VSDBG is setup within the container. It handles sending and receiving the debug adapter protocol messages to the remote debugger.

Visual Studio can now communicate using the debug adapter protocol which is used by Visual Studio, Visual Studio Code and Visual Studio for Mac. This protocol is now used for debugging cross-platform .NET Core applications.

After that is setup, it's just a matter of configuring vscode's launch.json to use pipeTransport.

My plan would be to wait until #565 is merged before starting work on this. Thoughts?

(also related to #500)

@sanathkr

This comment has been minimized.

Copy link
Contributor

sanathkr commented Jul 21, 2018

Yaay! This is awesome! Sounds like a plan.

I have a few questions:

  • will this method of using vsdbg work across all Visual studio products?

  • Does debugging require a change in the way function code is authored or compiled?

  • does this work for both dotnetcore 2.0 and 2.1?

  • Do you know if this requires a way in which the dotnetcore runtime is started? ie. Would it require a change to the Docker container?

Looking forward to working with you on this!

@austinlparker

This comment has been minimized.

Copy link
Contributor

austinlparker commented Jul 21, 2018

Good to see someone else taking this up, here's some notes I've got from looking into this already.

  1. Ought to be possible to map VSDBG from the host into the container in a similar way to how Delve is being mapped in for Golang debugging.
  2. VS Code and VS both use VSDBG as the remote C# debugger.
  3. The user's function code will need to be compiled in Debug mode (with embedded PDB's) for the best debugging experience, but this is where things get tricky...

The bootstrapper in the docker container loads the user's compiled function into memory and calls the handler method directly. The problem in a nutshell is that the actual process being executed is not the user's function, it's the bootstrapper. This might be work-aroundable by having users add a Debugger.Launch() statement to their handler, but I'm not sure.

@sanathkr

This comment has been minimized.

Copy link
Contributor

sanathkr commented Jul 21, 2018

@austinlparker Awesome! thanks for clarifying.

In nodejs debugging, when you start the debugger, it actually breaks at the bootstrap script. And when you hit Continue, it runs and hits the next breakpoint in your actual function. Would this be acceptable in dotnetcore too?

I want to make the debugging experience as transparent as possible.

@burck1

This comment has been minimized.

Copy link

burck1 commented Jul 21, 2018

Will this method of using vsdbg work across all Visual studio products?

It will work for Visual Studio 2017 & Visual Studio Code. Visual Studio 2015 only supports dotnet core 1.1 (project.json) and below, so I don't think we need to worry about it.

Does debugging require a change in the way function code is authored or compiled?

In the dotnet world (beyond just dotnet core), there are different build configurations. The defaults are Debug and Release. In the dotnet cli, this is set with the [-c|--configuration] parameter: dotnet build -c Debug.
For our purposes, the user should always be in Debug mode. I'm not sure what vscode does, but I know Visual Studio will popup an error message if you attempt to run the debugger in Release mode. That being said, it is possible to run the debugger in release mode by disabling justMyCode in the launch.json.

does this work for both dotnetcore 2.0 and 2.1?

Yes!

Would it require a change to the Docker container?

Based on @austinlparker's answer, I don't think we should need to change the container. We should be able to handle this similar to the go implementation.

Do you know if this requires a way in which the dotnetcore runtime is started?

Also based on @austinlparker's answer, I don't think we'll have an answer until we begin work on this. Since there is a bootstrapper that runs our code within the container, we would be attaching the debugger to the bootstrapper's process and the bootstrapper would be treating our code almost like a class library (unless it's doing some reflection magic). In the past, I have been able to debug my own class libraries that are executed by a different project using Visual Studio 2017, but I'm not sure if this is the situation.

@burck1

This comment has been minimized.

Copy link

burck1 commented Jul 21, 2018

@sanathkr I'm not sure if the same thing will happen with compiled languages. In order for VSDBG to debug, it needs the .pdbs (similar to Source Maps in front-end JS development) associated with the process it's trying to debug. When compiled in Debug mode, the .pdbs are embedded in the binary. In Release mode they are separate files. If we attach the debugger to the bootstrapper which is most likely compiled in Release mode, it may not break within the bootstrapper script. This is all speculation though until we actually try it.

@austinlparker

This comment has been minimized.

Copy link
Contributor

austinlparker commented Jul 21, 2018

@burck1

This comment has been minimized.

Copy link

burck1 commented Jul 21, 2018

I'll give it a go sometime in the next couple days and let you all know what I find out.

@burck1 burck1 closed this Jul 21, 2018

@burck1 burck1 reopened this Jul 21, 2018

@sanathkr

This comment has been minimized.

Copy link
Contributor

sanathkr commented Jul 22, 2018

Sounds good! @burck1 Hit us up on #samdev Slack if you need more help :)

@jfuss

This comment has been minimized.

Copy link
Contributor

jfuss commented Jul 23, 2018

This might be work-aroundable by having users add a Debugger.Launch() statement to their handler, but I'm not sure.

Even if that is a workaround, I don't think we should put this on customers. The way I am thinking about this is, I am going to be writing my code in my IDE. At some point in time, I will run into some issue and need to jump into my debugger. Yes, I will need to run some command (Ideally, I shouldn't even have though) to start the container with a debug port, etc, but that should be it. I shouldn't have to remember to add another statement in my handler code or have extra code that isn't needed/executed while the function is deployed to the cloud.

Honestly, I am not familiar with the dotnet world. This might not be a big deal but I think creating a streamlined experienced here will set this off on a good foot.

@burck1

This comment has been minimized.

Copy link

burck1 commented Jul 24, 2018

Hi All, quick update. I got dotnet debugging working using the lambci/lambda:dotnetcore2.1 container image. For now I had to extend it to add in VSDBG. Check out my repo here for more info and to test it out yourself:
https://github.com/burck1/aws-sam-cli/tree/dotnet-debug/dotnet-debugging. Screenshot in the repo for proof ;)
I'll work on integrating this with the sam cli this week.

@sanathkr

This comment has been minimized.

Copy link
Contributor

sanathkr commented Jul 24, 2018

This is awesome!! I glanced thru the run.ps1 file to understand how you did it. Looks straightforward.

Looking forward to the PR! Let us know if we can help you in anyway

@janaka

This comment has been minimized.

Copy link

janaka commented Jul 28, 2018

Looks like the container and function is invoked outside of sam local. Is that correct?

@hercax

This comment has been minimized.

Copy link

hercax commented Oct 31, 2018

I am currently having issues with a dotnet core Lambda function that I am trying to debug, any ideas if this is going to make it to a release soon? I was looking at the work @burck1 did and it looks awesome.

@abhurtun

This comment has been minimized.

Copy link

abhurtun commented Nov 2, 2018

the work looks awesome in which release of sam cli will this be included?

@sanathkr

This comment has been minimized.

Copy link
Contributor

sanathkr commented Nov 2, 2018

We have some initial designs but there are some blockers and we don’t have anyone driving this forward at the moment.

We are looking for someone to help us drive this, to get this done sooner.

@rschiefer

This comment has been minimized.

Copy link

rschiefer commented Nov 2, 2018

@sanathkr What kind of help do you need?

@ndobryanskyy

This comment has been minimized.

Copy link
Contributor

ndobryanskyy commented Nov 4, 2018

@sanathkr , I've started looking into the issue as well. I would provide the feedback if I have some more formed approach.

@TheSriram

This comment has been minimized.

Copy link
Contributor

TheSriram commented Nov 4, 2018

@sanathkr

This comment has been minimized.

Copy link
Contributor

sanathkr commented Nov 4, 2018

@rschiefer @ndobryanskyy Thank you so much for offering to help. @TheSriram already did some deep diving to come up with a checklist of work remaining. Between us, hopefully we should be able to standardize on an approach and start implementing.

@ndobryanskyy

This comment has been minimized.

Copy link
Contributor

ndobryanskyy commented Nov 4, 2018

@sanathkr @TheSriram , I'm glad to see, the work going on at this. I've done some deep research too. I suggest we sync at Monday and share the findings.

@TheSriram

This comment has been minimized.

Copy link
Contributor

TheSriram commented Nov 5, 2018

Some thoughts on this.

We need a way for sam cli to be orchestrating the debugging process for dotnet.
sam cli should be the one maintaining the lifecycle of the container. Therefore, the entrypoint needs to be something that sam cli controls, so that we can close the container at the end of the debugging session.

The debugger that is used for debugging is called vsdbg. The debugger can be obtained by downloading it from the following link.

curl -sSL https://aka.ms/getvsdbgsh | bash /dev/stdin -v latest -l ~/vsdbg 

The transport mechanism for the debugging session needs to be figured out, the options could be docker and ssh.

Tasks I had envisioned for debugging to work are below.

  • Establish a ssh transport mechanism by installing a ssh server on the container, and copy over authorized keys into the container. (SSH OPTION)
  • VSDBG can be mounted into the container after downloading it onto the host machine.
  • Add support for coreclr and its entrypoint in sam cli. https://github.com/dotnet/coreclr/blob/master/Documentation/workflow/UsingCoreRun.md
  • Establish a debuggable application by supplying the debug flag when building the dotnet application.
  • For finding the process or the container name needed to be attached
    • sam cli might need to reveal that information, becasue the docker container is not up till an invoke is started.
    • docker ps will reveal that information.(?)

Do let us know your thoughts and any steps that you had thought of.

@ndobryanskyy

This comment has been minimized.

Copy link
Contributor

ndobryanskyy commented Nov 5, 2018

@TheSriram Hi!
I had just the same thoughts on this. And I've actually implemented the POC which I'll share here with all the explanation, after I get home.

I'd very glad if you take a look on it. I'll include the Readme.md with steps and thought process

@TheSriram

This comment has been minimized.

Copy link
Contributor

TheSriram commented Nov 5, 2018

@ndobryanskyy woot! 🎆 💯 definitely!

@ndobryanskyy

This comment has been minimized.

Copy link
Contributor

ndobryanskyy commented Nov 5, 2018

@TheSriram , @sanathkr I've uploaded my POC to this repo.

I'd appreciate if you check it out and say what do you think. I will document all my decisions and plans for integrating it into the CLI tomorrow, because it is midnight at my place now.

At high level:

  • I've modified Docker image of dotnetcore runtime included vsdbg there. As alternative we can build it with separate Docker run and mount at runtime with CLI, what do you think is better?
  • I've modified Program.cs to include debugging support, I'll explain the way tomorrow.
  • I've attached to the running container with VS Code C# extension using pipeTransport and docker, see launch.json

For me this approach seems pretty decent. The only thing we can still improve is auto continue of the program after attaching, I have some hack, but I need some more time to implement it properly (this is
all because dotnet could launch the program and wait for the debugger to attach out of the box "debug mode', see issue track here)

I don't have so much of the insights of the SAM CLI internals, but I hope it does pretty much the same as my POC code.

I'm open for propositions and discussion- waiting for your feedback guys.

EDIT: fixed commands for sh

@sanathkr

This comment has been minimized.

Copy link
Contributor

sanathkr commented Nov 6, 2018

@ndobryanskyy This is awesome! Actually yours is much different than what we originally imagined. It aligns closely with what how SAM CLI already supports debugging for other languages..

@mikemorain

This comment has been minimized.

Copy link

mikemorain commented Nov 6, 2018

@sanathkr as we discussed, I'm also happy to help on this as the design comes together.

@ndobryanskyy

This comment has been minimized.

Copy link
Contributor

ndobryanskyy commented Nov 6, 2018

@sanathkr , I'm glad you liked the solution. I can now try and integrate it in the CLI itself. I'll fork the repo and implement initial design, similar to how the GO was added I think.

But first, could you please answer those questions:

  • Will the vsdbg be already included in lambci container? As for me I'd go with yes for the end user of CLI this will simplify the experience. As opposed to write some script and build the debugger from inside the similar container with mounted volume, and remount it to the runtime container. As we answer that question, I will go and submit the PR to lambci repo with modified Program.cs and optionally Dockerfile.
  • Is this press any key to continue... okay to start with? Or I can investigate the possibility to run some task on the target machine (Docker container) after the debugger has been attached. For example write some character to stdin to continue execution (echo "." > proc/1/fd/0). Maybe you have some other options for this?

As for help, I would appreciate if someone experienced with the CLI will explain me the overall design to accelerate the onboarding, maybe you have some detailed docs, (especially for Windows 🐱‍👤). Apart from lack of knowledge of the product implementing the feature seems straightforward to me.

@mikemorain maybe you can help with that?

Also I'll work on explanations of the solution today.

@ndobryanskyy

This comment has been minimized.

Copy link
Contributor

ndobryanskyy commented Nov 6, 2018

I've updated the detailed explanation section. I would appreciate if you look through it, ask questions (if any) and give feedback - I'm open for discussion and suggestions / improvements, as I now plan to start working with SAM CLI integration of this approach.

@TheSriram

This comment has been minimized.

Copy link
Contributor

TheSriram commented Nov 7, 2018

@ndobryanskyy Sorry progress on this has been slow, the team has been focused on tasks at hand.

This is awesome work!, going through explanation sections. I personally like the third option where we the supply the debugger path, because its the most flexible approach and we dont need to tamper with the image at all. Are you on the slack channel? #samdev . You should totally get on it! :)

I think what we can do is to encapsulate all the work that is to be done as an issue, that clearly shows the tasks. (maybe as checkboxes or other issues). Then we can go through this implementation process by just finishing those tasks. Does that make sense?

@sanathkr has a WIP PR: https://github.com/awslabs/aws-sam-cli/pull/738/files which talks about how we are to have a design doc for the features we do. I think this feature could be a pilot for that!

@ndobryanskyy

This comment has been minimized.

Copy link
Contributor

ndobryanskyy commented Nov 7, 2018

@TheSriram , no problem you had the release it is understandable and thanks for reading that "TL;DR" of mine 😅.

I've submitted the request to be added to the slack channel, but no invitation yet 😔. Maybe you can streamline the process.

I will go with third approach then, and am going to submit the PR with modified runner with debugging flag support to Docker lambda repo (check out the issue above).

As for the docs you've mentioned, I like the idea, I can open up an issue and write down all of the required steps with checkboxes. I've already grasped how the CLI works, and now trying to make some draft with .net core debugging on SAM develop branch.

Does this sound like a plan?

@ndobryanskyy

This comment has been minimized.

Copy link
Contributor

ndobryanskyy commented Nov 8, 2018

I've filed PR with just the debugging support at docker-lamda (no vsdbg in the container as discussed) For the first iteration Console.Read() would be sufficient I hope.

@mikemorain

This comment has been minimized.

Copy link

mikemorain commented Nov 8, 2018

@ndobryanskyy Nice work on the design section for this! I'm wondering if, for the long term, we can do a combination of 1 or 2 and 3. I think that would give the best UX overall, since it would be easy to get going quickly, but also provide the flexibility for the user to provide a separate debugger. I like the idea of making it incredibly easy out of the box, as it goes along with the theme of the CLI generally.

With regard to the debugger attachment generally, I think we could also elect to use the System.Diagnostics.Debugger.IsAttached property to wait for the debugger and proceed once it's attached. It provides a slightly more stream-lined process than a Console.Read().

Thoughts?

@ndobryanskyy

This comment has been minimized.

Copy link
Contributor

ndobryanskyy commented Nov 8, 2018

@mikemorain, thanks for the feedback I really do appreciate that, I am totally agreed with you, that the UX must be consistent and streamlined, so the second option with some plugin support maybe, could be the best, but again in the long run. Got to get some grooming on that, and take into the consideration all of the runtimes also, to he as consistent as possible.

As for the waiting, have you checked my WIP: PR it now uses both the IsAttached (to check) and Console.Read() (to wait). If we go with only IsAttached property we must have some while loop which will check the property every, let's say, 100ms (because .net does not have event like debugger attached as far as I know) is that what you're talking about?

As some other thought we can actually communicate with VS Code C# extension team, and make some event like debuggerAttached which can be assigned a task in launch.json, which will in our case just go to the Docker and put some character into the stdin of PID 1 to make it automatically continue the execution. But that sounds like a hack 😅.

Maybe you have some other suggestions or so e debugger port listening socket kind of magic?

@mikemorain

This comment has been minimized.

Copy link

mikemorain commented Nov 8, 2018

@ndobryanskyy Ah, I had missed that in the PR; I was looking at the design document; nice work already putting that check in there. I'm wondering if, since to trigger this behavior, a specific flag or environment variable will need to be set, that putting in a waiting loop would be slightly smoother. Obviously not the biggest of deals, but might be a slightly better UX.

And while I like the creativity of the your C# extension idea, I agree that it's probably not the most robust way to go forward :-D.

Looking at the way that the Go integration is handled, delve seems to handle the waiting of execution until the remote debugger is attached to the server and says continue. Given that experience, I think it might be best to emulate the friction-free UX and have a loop that waits for attachment given the flag.

I'm definitely open to counterpoints though!

@ndobryanskyy

This comment has been minimized.

Copy link
Contributor

ndobryanskyy commented Nov 9, 2018

@mikemorain, well, I agree with you, that if it will all happen automatically, without user ever pressing any key, that would be considerably easier.

In that case I suggest that we go like this:

  • Run infinite loop, that will check IsAttached property
  • If user press any button while the loop runs we will continue.

It could be like Waiting for the debugger to attach, press any key to continue immediately...
That way the user will be able to stop the loop at any time and still will have smooth UX

What do you think?

@sanathkr

This comment has been minimized.

Copy link
Contributor

sanathkr commented Nov 9, 2018

I prefer the infinite loop approach, without key press. When this is integrated with IDEs or other tools, then pressing key might not be an option. Also if you are hitting a local API, then you are seeing the browser window and wondering why it didn’t run. So I would just do a tight while loop, may be with a timeout of like 5minutes.

@ndobryanskyy

This comment has been minimized.

Copy link
Contributor

ndobryanskyy commented Nov 9, 2018

@sanathkr yep, nice point about timeout! The user would be able to stop waiting either by pressing any key to "continue without debugging" or just a simple timeout.

@ndobryanskyy

This comment has been minimized.

Copy link
Contributor

ndobryanskyy commented Nov 9, 2018

@sanathkr, small off-top (because I don't know the other place to ask this) how can I join the Slack channel, because I've sent the request like 5 days ago, and still no invite do I have to do anything else, to get it?

@sanathkr

This comment has been minimized.

Copy link
Contributor

sanathkr commented Nov 9, 2018

@ndobryanskyy Can you send your preferred Slack email address to my email listed in my profile? https://github.com/sanathkr I will add you

@sanathkr

This comment has been minimized.

Copy link
Contributor

sanathkr commented Nov 9, 2018

@ndobryanskyy I was suggesting removing the Press Any Key option altogether. It doesn't make sense because customers won't have direct access to the stdin of the Docker container inside which this code runs. Instead a plain infinite loop to wait for debugger to attach (with timeout) is the best idea IMO

@ndobryanskyy

This comment has been minimized.

Copy link
Contributor

ndobryanskyy commented Nov 9, 2018

@sanathkr, I will send you my email, thanks!
Regarding stdin, yes sometimes when -i is not passed, that is really the case.

I will update the PR according to our discussion. Thank for the feedback @sanathkr, @mikemorain

@ndobryanskyy

This comment has been minimized.

Copy link
Contributor

ndobryanskyy commented Dec 7, 2018

I think we could close this

@TheSriram

This comment has been minimized.

Copy link
Contributor

TheSriram commented Dec 11, 2018

@ndobryanskyy sounds good. Do you want to open an issue for debugging support with Rider in a separate issue. (Think you covered this in your RFC) and then we can close this guy?

@ndobryanskyy

This comment has been minimized.

Copy link
Contributor

ndobryanskyy commented Dec 12, 2018

@TheSriram, no problem I will open issue for Rider support and issue on csharp VSCode today 👍

@TheSriram

This comment has been minimized.

Copy link
Contributor

TheSriram commented Dec 12, 2018

Closing this issue, another issue has been opened for explicit Rider support in .NET debugging. #855

@TheSriram TheSriram closed this Dec 12, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment