Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Possible to populate Win64 COFF objects with CodeView dbginfo? #167

Closed
TurkeyMan opened this Issue Sep 17, 2012 · 38 comments

Comments

Projects
None yet
8 participants

Since LLVM doesn't look like it will support PDB any time soon, and DWARF isn't particularly useful to most windows dev environments, is it possible to populate the COFF objects with CodeView debug info?

MSVC has the (legacy) /Z7 option, which populates the objects with CV debuginfo, which the MS link.exe is still capable if parsing and inserting into the final PDB while linking...

If LLVM/LDC were able to populate the windows objects with CV debuginfo, they may be much more useful when used in typical MSVC dev environments.

Owner

klickverbot commented Sep 17, 2012

This is a question for @redstar – these sorts of issues are exactly where my Windows dev knowledge is lacking…

This has a key limitation though; you can't link VC and LDC objects together, since VC produces a PDB for the stuff it compiles, and then cv2pdb overwrites it. The resulting debug info is mutually exclusive.

It would certainly be much better if the objects had CV debuginfo, and let link.exe do what it's supposed to do.

Owner

redstar commented Sep 18, 2012

As already discussed on the LLVM list the way to go is to convert the dwarf debugging information to CV or PDB format while writing the COFF file.
That's a point on my TODO list but with lower priority then the exception handling.

Okay cool. Yeah that's where the LLVM guys ended up too. It seems very indirect.
I wonder if code from cv2pdb can be integrated into LDC directly so it can be done in one pass...

Perhaps worth mentioning that Walter has preliminary CV8 support written in DMD-Win64 already, and is continuing to work on it. Perhaps this is a good reference for the debuginfo format.
I'm using the MS linker to build the PDB file, and can do the same when using LDC. So PDB format is not necessary, just the CV8 data in the objects.

Nice documents! I hope Walter knows about all of those... or have the 2 of you been talking? :)

Owner

redstar commented Nov 5, 2012

I started to implement this. It seems to be easier then expected. The construction of DWARF debug symbols is encapsulated in class DwarfDebug, which is used in the AsmPrinter. I defined the interface in a new abstract base class and introduced a new CodeViewDebug class. Looks like I can output the first bytes into a .debug$S section :-)

If you are curious then you can look at this diff. But there is still a lot to do - the debug symbol encoding is anything than simple.

Awesome, nice work.
I recommend checking out Walter's recent work too in DMD, he has CV8 debuginfo in COFF objects from DMD working pretty nicely now. Might be of use, since it's all so poorly documented.

Owner

redstar commented Nov 18, 2012

I still fight with the results of a hard disk failure... Nevertheless I can report some progress here. I started to write the first bits to `.debug$T´ section.

And yes, I am aware of the code produced by Walter and I am studying it in case of questions.

Awesome! I really think this will do a lot of good for Windows users!
DMD is great, I'm really happy to have usable access to the reference compiler on Windows now, but LDC will offer a backend with far better optimisation, maturity, and portability, which will be of unquestionable value to Windows users.

Owner

redstar commented Dec 7, 2012

As promise I posted first code to the LLVM commit list. I can already generate the right type information for simple signatures like long add(long a, long b). The generated .debug$T section is identical to the one emitted by DMD.

I now try to emit line information. Then it should be possible to really 'see' something in a debugger.

pravic commented Dec 7, 2012

@redstar Great and really needed work!

You sir, are rocking a full jar of awesome sauce! :P .. Walter had some trouble with the call stack too, how's that looking?

Contributor

Trass3r commented May 11, 2014

What's the current state of this?

Owner

redstar commented May 13, 2014

There is only one improvement. The last missing piece (a special relocation) has been added to LLVM 2 month ago by the Google people. I'll restart work on this as soon as I have some more time.

BTW: LLVM has now COFF line number support. I don't know if this helps much...

That motion from Clang to output objects that were fully compatible with MSVC, how is that going?
I thought they'd have to deal with this problem in Clang to meet that goal of theirs?

Will we see MSVC compatible objects from LDC any time soon? :)

Contributor

Trass3r commented Jul 1, 2014

http://clang.llvm.org/docs/MSVCCompatibility.html
Only line tables. But yeah eventually they will have to solve that problem too.

@redstar redstar pushed a commit that referenced this issue Sep 27, 2014

@jmdavis jmdavis Merge pull request #167 from dawgfoto/newline
std.string.newline is deprecated
998d8af

Any recent motion on this issue?
I think this is the last significant issue standing between D and many many windows users.
I think it's safe to say a lot of potential Windows users will be turned away by DMD's inferior codegen. People don't often use native languages if weak performance is an option.

Contributor

Trass3r commented Oct 7, 2014

That and EH. No news on dbginfo.

I don't use EH, but some libraries (including phobos) use it. Personally, I can work-around EH. Although I'm sure it'll be a massive win for Windows too, I think the debuginfo is the biggest blocker. Nobody I've ever met would dev on Windows without the debugger. People are utterly accustomed to visual studio :)
Most people in my industry would equally avoid EH as me...

Contributor

Trass3r commented Oct 7, 2014

Sure, users can avoid EH, but it is a big development issue cause it results in most druntime/phobos unittests crashing which masks potential real ldc problems.

Well naturally, the VS debugger kicks ass. I don't know any other one that comes even close.
A good thing you can do is push for it on cfe-dev. Their priorities are a bit off, working on 100% msvc ABI compatibility for features that could easily be worked around rather than fundamental blockers. But I already got them to pull EH up in the list.

Contributor

Trass3r commented Oct 22, 2014

Ok now that EH is working this is the number 1 roadblock.

Contributor

Trass3r commented Oct 22, 2014

lists.cs.uiuc.edu/pipermail/cfe-dev/2014-January/034747.html
-.-

Ilod commented Oct 23, 2014

Hello,
FYI, since this thread was my principal source of information on this subject, I recently started to look in debug information generation for Visual Studio with LLVM, as I need it for C++ development. Not sure I will be able to commit some useful stuff, as I'm not really familiar with this, but with the resource listed in this thread I was finally able today to add functionnal breakpoints to the line table generation in really simple programs. I will probably start a thread on LLVM ML/start working on some patch next week.

Contributor

Trass3r commented Oct 23, 2014

Are you able to get source code interleaved with disassembly?
Some of the links above are dead by now.

https://android.googlesource.com/toolchain/yasm/+/master/modules/dbgfmts/codeview/cv8.txt

Ilod commented Oct 23, 2014

Yeah, I may have some small problems as I still have to setup function prolog/epilog correctly, but I can display disassembly and it seems to correspond to the code. There was not much missing to the existing line table generation to do this in fact, they were juste missing the symbol function declaration.

Contributor

Trass3r commented Oct 24, 2014

The function symbols have just landed in llvm: http://reviews.llvm.org/D5772
This enables basic source-level debugging (no variables of course)!

@vtjnash vtjnash referenced this issue in JuliaLang/julia Dec 27, 2014

Merged

RFC: MSVC continued #7761

Owner

redstar commented Jan 27, 2015

There is still work on this topic in LLVM: http://reviews.llvm.org/D7153

Owner

redstar commented Nov 1, 2015

Some more news: Microsoft is going to provide CodeView symbols to LLVM.
See the announcement: http://permalink.gmane.org/gmane.comp.compilers.clang.devel/45575
LLVM: http://reviews.llvm.org/D14209
clang: http://reviews.llvm.org/D14210

Oh, god almighty, thank you! :P
I see that their patch comes with a 'Clang portion' along with an 'LLVM portion'. I guess this implies an LDC portion. How complex is that?

Owner

redstar commented Nov 1, 2015

I just discover the announcement. It says that they want to introduce new metadata nodes. This could mean that we have to create a 2nd DIBuilder. But I hope that a more generic design emerges. Debug info generation is centralized in the DIBuilder class which should therefore the only class which needs to be enhanced.

Reading those articles, it looks like they generate CV metadata in Clang independently from DWARF. Kinda looks like the Clang patch is significant, and also requires full duplication in LDC... :/

Owner

redstar commented Nov 1, 2015

Yes, that's the approach of the RFC. But the first comments in the discussion about the RFC points out that it is better to have an uniform solution.

That's nice, I didn't read that far.
I saw he made a comment where he made a change to the Microsoft mangler to support some stuff; this means he's making changes to the MS codebase (Visual Studio?) to complete this work. If you can get in on details required to nicely support D; ie, things like local functions, which MS might not support, then perhaps you can influence him to add support for these cases to MS's CodeView description while he's at it...
Ie, being involved with this guy while he's on this task might lead to influence in the MS toolchain, and that means we might be able to slip technology in there required by D (local functions? D closures?) that he would have otherwise ignored ;)

Do you know what this means for de-mangling? Does VS use the prettyName field he refers to to display function names, which we can populate correctly, or would some work be required by MS to have good support?

Owner

redstar commented Nov 1, 2015

I have a different understanding here: he changed the Microsoft Mangler in clang, see http://reviews.llvm.org/D14210#e4b48f2a.

Owner

redstar commented Aug 27, 2016

See #1716 for current state.

Member

kinke commented Sep 2, 2016

Let's close this then and continue with #1716, as (incomplete) debuginfos are already generated.

@kinke kinke closed this Sep 2, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment