New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve NuGet package debugging and symbols experience #6104

Open
diverdan92 opened this Issue Oct 25, 2017 · 76 comments

Comments

Projects
None yet
@diverdan92

diverdan92 commented Oct 25, 2017

Currently the NuGet package debugging and symbols experience is not streamlined.

Some areas that need improvement are:

  1. Optimize the NuGet package and symbol authoring story to encourage ####production of symbols
  2. Support for the new portable PDB format for .NET Core and .NET Standard applications
  3. Better integration in Visual Studio tooling.

This issue will be used for discussion of the spec here.

@maartenba

This comment has been minimized.

Contributor

maartenba commented Oct 27, 2017

Would love to see this! Have had an offline discussion with some other folks, and came up with a dream workflow:

The main issue today is that when running “nuget.exe pack -Symbols” on a .NET Standard project, SourceLink is not enabled by default. Which makes a LOT of sense as not everyone is keen on sharing their sources. However: when people do want to have sources support, it’s usually too late and discovered during a debugging/profiling/dump analysis situation where symbols and sources would have been handy…

Going out on a limb, perhaps the ideal workflow would be something like this (trying to shape a flow and feature):

Package creation/publishing phase

  • User, when packaging using nuget.exe pack -Symbols for Portable PDB
    • If the Portable PDB does not contain embedded sources + sourcelink
      • Check whether source_link.json is available. If not:
        • Generate one based on the current folder and make sure the Portable PDB contains it (may need a recompile… so maybe generating one should be a default of csc.exe instead of a NuGet feature)
          • The mapping would be “”: “%sourceservertemplate%/” (where documentname is actually filled out, and %sourceservertemplate% is a placeholder that is embedded into the file.
  • User can then publish NuPkg and symbols NuPkg to wherever they want. If that “wherever” supports symbols, then yay!

Symbols consumption phase (user wants to fetch symbols and sources)

  • If the Portable PDB is embedded in the assembly and has embedded sources etc, use the VS SourceLink feature to show sources
  • If the Portable PDB must be downloaded, then download it from the configured symbol server using the traditional semantics of PDB ID and PDB name
    • After download, if sources are embedded, use the embedded sources
    • If sources are not embedded but the source_link.json is, and has a URL for document mappings, use the URL
    • If sources are not embedded but the source_link.json is, and has a URL that contains %sourceservertemplate%, interesting things happen!
        1. The debugger/VS/… would reach out to the symbol servers configured in VS and fetch, for example, http://symbolserver.local/services.json. This services.json would contain the actual value for %sourceservertemplate%. The value would be cached for the entire debugging session.
        1. The debugger/VS/… uses the %sourceservertemplate% placeholder in the URL that was in source_link.json, then fetches the sources from there.

That seems, for Portable PDB, to give the entire spectrum of embedded + sourcelink, to sourcelink with url mappings, to no sourcelink at all but still have debuggability.

@diverdan92

This comment has been minimized.

diverdan92 commented Nov 14, 2017

@maartenba -

a. The debugger/VS/… would reach out to the symbol servers configured in VS and fetch, for example, http://symbolserver.local/services.json. This services.json would contain the actual value for %sourceservertemplate%. The value would be cached for the entire debugging session.
b. The debugger/VS/… uses the %sourceservertemplate% placeholder in the URL that was in source_link.json, then fetches the sources from there.

The way you envision this working, where would the source be uploaded to support these scenarios? It seems like the package author has not uploaded the source in this case?

@maartenba

This comment has been minimized.

Contributor

maartenba commented Nov 14, 2017

That comes from the creation phase:

  • User, when packaging using nuget.exe pack -Symbols for Portable PDB
    • If the Portable PDB does not contain embedded sources + sourcelink
      • Check whether source_link.json is available. If not:
        • Generate one based on the current folder and make sure the Portable PDB contains it (may need a recompile… so maybe generating one should be a default of csc.exe instead of a NuGet feature)
        • The mapping would be “”: “%sourceservertemplate%/” (where documentname is actually filled out, and %sourceservertemplate% is a placeholder that is embedded into the file.

This is essentially the current NuGet behavior, where the generated package contains sources and a sourcelink file that holds a placeholder URL. That way, a server can serve up symbols based on the embedded sources.

Imagine the following scenario that you can do today:

  • User runs nuget pack -Symbols and generates a .nupkg and .symbols.nupkg.
  • User uploads both to server, and consumes only .nupkg in their project
  • All of a sudden, debugging is required! They configure a symbol server that serves up the Portable PDB that was in .symbols.nupkg, and expect things to work. Except: they do not!
    • The Portable PDB contains some info, but not sources. It also does not contain a link to the sources. In other words: the user will not be able to do debugging.
    • If instead, NuGet had added the template URL to sourcelink (and VS could work with it), VS could query sources using that template URL, fetching the source files that were in the .symbols.nupkg

Another option is embedding the sources in the Portable PDB, which would be awesome, except for one thing: there is no user control possible. E.g. if I am a tools vendor and I want only my team to have debugging possibilities and not my clients, how do I prevent one group from using the symbols embedded in the Portabe PDB? For this scenario, having to reach out to the server will make it possible to limit access to PDB and source files.

@tmat

This comment has been minimized.

tmat commented Nov 22, 2017

@maartenba We are working on making it super easy to enable source link in builds for both public and private repos, as well as embed sources to PDBs that not tracked by a source control server (e.g. sources generated during the build). Our goal is to reduce the number of packages that were not built with source link or embedded sources. That said, we understand that there will be cases where it will be necessary to add sources after the fact. The scenario is valid and we have some ideas how to address it in future. I don't think NuGet is the right tool to implement this support though (by creating source link files or recompiling PDBs).

@anangaur

This comment has been minimized.

Member

anangaur commented Nov 30, 2017

When publishing packages, both the .symbols.nupkg and the .nupkg will be published to NuGet.org. We will make the nuget and dotnet push CLI automatically push both .nupkgs and .symbols.nupkgs (if they exist). We will add a parameter to the dotnet and nuget CLIs to opt out of automatic pushing of .symbols packages.

I would say we should opt in to publish the symbols and by default only the .nupkg is pushed not the .symbols.nupkg.

We are already taking the change to not auto-push package. (See #6113). Lets not randomize :)

@diverdan92

This comment has been minimized.

diverdan92 commented Jan 29, 2018

Draft of the spec is now available here: https://github.com/NuGet/Home/wiki/NuGet-Package-Debugging-&-Symbols-Improvements

Please let us know any feedback you have in this thread!

@dotMorten

This comment has been minimized.

dotMorten commented Jan 29, 2018

Questions from reading the draft spec:

  1. Will be able to host our own symbol server / will the nuget server nupkg be updated to support this?
  2. Will the below requirements also apply to private symbol servers?

image

Requiring 4.7.1 is way too soon for us, and also we'd like to continue using our existing native PDBs as well.

  1. Will source linking support multiple repos (at the very least one per .dll), and will you support on-premise git servers?

  2. How will source linking work wrt to versioning? Do we need to specify a specific commit or tag for each repo we're source-linking to?

@ctaggart

This comment has been minimized.

ctaggart commented Jan 29, 2018

Great write-up @diverdan92. Thanks for sharing the plan. I'm trying to better understand the advantages of using a symbol server instead of just including the .pdb files in a normal .nupkg.

If a .pdb is included in a normal .nupkg, NuGet and NuGet.org will not do any special validation or indexing of the .pdbs. Debugging of the .nupkg will work in a limited way as the symbols will be cached next to the .dll, but it will not be indexed and downloaded from a symbol server. The only way to ensure indexing on the symbol server is to include the .snupkg. If the symbols are not indexed, certain scenarios will not work, such as crash dump debugging and attaching to an application.

So you can't do crash dump debugging or attach to an application with just .pdb files on the filesystem distributed via the normal .nupkg???

@jnm2

This comment has been minimized.

jnm2 commented Jan 29, 2018

I had a similar thought- all that complication to save a bit of disk space? Hey, if you're willing to do that, and keep it simple for both authors and consumers, more power to you!
The news about the UI is particularly encouraging. Educating people and lowering the barrier to stepping into library source will bring us closer to a world in which every library comes with symbols and all consumers make use of them. I expect the quality of bug reports to go up when it's no longer a black box and a curious mind can figure out a minimal repro so much more easily.

@dotMorten

This comment has been minimized.

dotMorten commented Jan 29, 2018

I had a similar thought- all that complication to save a bit of disk space?

My PDBs are over 500mb each, and there's 3 CPU architectures, meaning over 1.5gb of PDBs uncompressed! (about 400mb compressed in the nupkg)

Currently there's a 250mb limit on nuget.org.

@diverdan92

This comment has been minimized.

diverdan92 commented Jan 29, 2018

@dotMorten

Excellent questions!

  1. Yes, you should be able to host your own symbol server. You can setup your NuGet server to do the processing of .snupkgs manually, or you can just spin it up as a regular server to distribute .pdbs (e.g. upload directly to your symbol server, rather than to your own NuGet server).
  2. The policy we are imposing is specific to the NuGet.org server. You are free to decide which policies you would like to apply to your own private symbol servers.
    3 + 4. @tmat is the expert here. Can you chime in?

@ctaggart - Great feedback. The issue is trying to debug a crash dump for a .dll you don't necessarily have restored/installed. If it's hosted on a Symbol server and you have it correctly configured, the tooling will automatic fetch the symbols to get meaningful debugging information. If you have everything installed and can locate the Symbols for the debugger, you can still get this to work including the Symbols as a part of the package, it's just not as ideal as it could be.

One of the open issues we need to close on is "How do we support preemptively caching symbols for offline scenarios". We do recognize a need for preemptive caching and are looking at the best ways to solve this. Any suggestions?

@ctaggart

This comment has been minimized.

ctaggart commented Jan 30, 2018

The issue is trying to debug a crash dump for a .dll you don't necessarily have restored/installed. If it's hosted on a Symbol server and you have it correctly configured, the tooling will automatic fetch the symbols to get meaningful debugging information. If you have everything installed and can locate the Symbols for the debugger, you can still get this to work including the Symbols as a part of the package, it's just not as ideal as it could be.

@diverdan92, so if you include the .pdb files in the regular .nupkg, are there debugging scenarios that do not work? Wouldn't eliminating the need for Symbol Servers for a large portion of debugging scenarios simplify things for those scenarios? Why not make adding .pdb files to the regular .nupkg the default?

@diverdan92

This comment has been minimized.

diverdan92 commented Jan 30, 2018

@ctaggart we don't want to bloat the entire NuGet ecosystem unnecessarily. It impacts nearly everything - from Visual Studio install time, project creation time (for most projects), restore times, and more. We also have evidence that a majority of users won't debug into third party components, so promoting a model where the .pdbs are included in the .nupkg by default, we impact the performance for everyone to support a scenario for more advanced users.

@ctaggart

This comment has been minimized.

ctaggart commented Jan 30, 2018

@diverdan92 As a compromise, can you support a scenario to restore the symbols upfront at the same time that we restore the packages? A RestoreSymbols target was suggested: msbuild /t:RestoreSymbols MySolution.sln. A property could work too.

@onovotny

This comment has been minimized.

onovotny commented Jan 30, 2018

@ctaggart I was proposing a setting in the nuget.config file for this.

I also suggest having the symbol sever endpoint being part of the v3 protocol spec, being part of the feed info itself. Then each NuGet client can discover the symbol server endpoint info. You could enable the per-feed symbols with an on/off setting on the nuget.config feed item element easily then.

So you may want symbols from your myget feeds, but not nuget.org. And you'd have a setting in the nuget.config file to pull them on restore as well that's repo-wide. That way you can have a consistent team experience.

@diverdan92

This comment has been minimized.

diverdan92 commented Jan 30, 2018

@ctaggart that's where the option issue came from 😊. We will do something for this scenario to support the ability to download symbols upfront. The exact mechanism is TBD.

@ctaggart

This comment has been minimized.

ctaggart commented Jan 30, 2018

I see this is labelled Priority:1. Would this be in the VS 2017 15.7 time-frame?

@diverdan92

This comment has been minimized.

diverdan92 commented Jan 30, 2018

Still to be determined. There are several moving pieces (compiler work to embed hashes of .pdbs in .dlls, client changes, spinning up the service, etc...). I would expect it after 15.7, but we may be able to start lighting up certain scenarios between 15.7 and whatever is next. Can't make any promises just yet.

@tmat

This comment has been minimized.

tmat commented Jan 30, 2018

Will source linking support multiple repos (at the very least one per .dll), and will you support on-premise git servers?

Yes. The implementation we are working on will support git submodules if that's what you mean by multiple repos. You can even have one dll built from multiple submodules. It will also support sources included from NuGet source packages.

How will source linking work wrt to versioning? Do we need to specify a specific commit or tag for each repo we're source-linking to?

The build task will automatically determine the commit sha from the repository using Git API (or changeset number for TFVC repositories). No need to specify anything.

I have done some prototyping of integrating source control features to build.
You can find a short doc here: https://github.com/tmat/repository-info/blob/master/docs/Readme.md
I'm now working on turning the prototype to real features and packages.

@mungojam

This comment has been minimized.

mungojam commented Apr 25, 2018

We've discovered what was happening, Resharper was decompiling the code that we were debugging into or getting it from somewhere else rather than from TFS. When we made sure that VS was pulling the sources we were then able to observe all variables in the debugged code.

Thanks again, the requirement to create debug versions of our packages may have just disappeared for us :) Wish I knew about this setting 5 years ago!

@mungojam

This comment has been minimized.

mungojam commented May 8, 2018

It seems there is a big regression with the new project-system (not when just changing to PackageReference). We include our .pdb files inside the appropriate lib folder of our packages. With the old project system, they get automatically output to the bin directory of our tools for easy debugging.

With any tools using the new project-system none of them are not getting output, just the pdb from the tools themselves.

Is this a known functionality change?

now raised: dotnet/project-system#3524

@jnm2

This comment has been minimized.

jnm2 commented May 8, 2018

@mungojam There is an existing issue: dotnet/sdk#1458 (workaround).

@krispenner

This comment has been minimized.

krispenner commented May 25, 2018

Additional client restore tooling to support a flag on the consuming side that would attempt to restore both Debug/Release packages on package restore, and load the different assemblies at runtime based on project configuration(s)

@diverdan92 I would like to see this in VS - is there a discussion with the VS team for this that you are aware of?

Currently, you cannot step-in to an async method built in release mode - some optimization prevents this, it just steps over it every time. Your suggestion would make this very easy when debugging referenced packages with async methods which is pretty common. See dotnet/roslyn#13850

One thing you can do with PackageReference is create both a debug and release version of your package (e.g. Foo.nupkg and Foo.Debug.nupkg), and then conditionally include those packages depending on the configuration of your project.

I was initially thinking along these lines as well and came across this issue when googling what others were doing to support debug builds in packages. Will probably go this path until better support for debug builds is added. The editing my csproj files by hand to add conditional package references is a drawback for me.

@dotMorten

This comment has been minimized.

dotMorten commented Jul 13, 2018

@tmat

Yes. The implementation we are working on will support git submodules if that's what you mean by multiple repos. You can even have one dll built from multiple submodules. It will also support sources included from NuGet source packages.

No I did not mean submodules. Each repo builds different DLLs, and the nuget packaging assembles them all. In my case one repo for one of C++ DLLs (multiple architectures) a another repo for a .NET AnyCPU DLL + a C++ (Multi-architecture) DLL. We don't use submodules, since that requires us to daily update the commit ID, so the dependency can be referenced by master daily or log onto a tag when we start stabilizing.

@tmat

This comment has been minimized.

tmat commented Jul 13, 2018

@dotMorten In that case each repo is responsible for generating SourceLink for the binaries it produces and embedding it into the PDBs corresponding to these binaries.

@dotMorten

This comment has been minimized.

dotMorten commented Jul 18, 2018

@tmat SourceLink appears to be a Roslyn feature and doesn't help PDBs for C++ libraries.

@tmat

This comment has been minimized.

tmat commented Jul 18, 2018

@dotMorten We have implemented support for Source Link in C++ as well. Let me check when is the feature expected to be released.

@tmat

This comment has been minimized.

tmat commented Jul 18, 2018

@dotMorten VS 2017 Update 8.

@rrelyea

This comment has been minimized.

Contributor

rrelyea commented Sep 21, 2018

@diverdan92 - nuget client has finished the work for 15.9 related to this.

trying to clean up this milestone. can we close this issue, or move to future milestone, or other?

@diverdan92

This comment has been minimized.

diverdan92 commented Sep 21, 2018

@rrelyea - we should wait until the e2e scenario ships to close the issue.

@drauch

This comment has been minimized.

drauch commented Nov 17, 2018

I'm new to this proposal, just read on it on the blog. The thing I don't get: what's wrong with adding the PDBs to the original NuGet package? We've done that for years and haven't got any problems with it. Is this discouraged now?

@dotMorten

This comment has been minimized.

dotMorten commented Nov 17, 2018

You're bloating the package for no reason. Also forcing a larger deployment on others who might not care about 3rd party symbols but are only interested in symbols for their own code.

@JamesNK

This comment has been minimized.

JamesNK commented Nov 18, 2018

How do snupkg files relate to package signing? If the main package is signed then should both be signed?

Also should this guidance be changed - https://docs.microsoft.com/en-us/dotnet/standard/library-guidance/nuget#symbol-packages

Is there any reason to still have PDB files in the main package?

@onovotny

This comment has been minimized.

onovotny commented Nov 19, 2018

IMO, PDB's should be in the main package, always. It's much faster to have the pdb's around instead of downloading on-demand. Also, some things like .NET Native need the symbols to be on disk in advance in order to munge the PDB's to match the native output. Symbol server does not work for that scenario.

I believe snupkg should only be used for full pdb's, which are bigger, but that's the one type of symbol they don't support. PPDB's are small. Better yet, use embedded as the symbol type and it'll be compressed and put directly in the DLL. No extra symbols to keep track of! :)

@JamesNK

This comment has been minimized.

JamesNK commented Nov 19, 2018

Interesting. I'm going to create a PR in the library guidance. I'll advertise it and see what direction the discussion and consensus goes.

@onovotny

This comment has been minimized.

onovotny commented Nov 19, 2018

@JamesNK Can you please post the issue link here when you do?

@JamesNK

This comment has been minimized.

JamesNK commented Nov 19, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment