DWARF support for macOS and Linux

[joelreymont]

DWARF v5 Debugging Support for OCaml Native Compiler

This PR adds DWARF v5 debug information to the OCaml native compiler, allowing proper source-level debugging in GDB and LLDB.

What's Implemented

Core DWARF Support

• Implements DWARF v5 using inline strings (DW_FORM_string) to avoid linker issues.
• Multi-compilation unit (CU) support with string table deduplication.
• Section-relative relocations for portability.
• Supported architectures: AMD64 and ARM64. 32-bit platforms are not supported.

Debug Information

• Function-level debugging: breakpoints by function name.
• Line-level debugging: breakpoints by file and line.
• Parameter tracking: parameters visible in debuggers with source names.
• Local variable tracking: let-bound variables with correct scopes and locations.
• Lexical blocks: nested scopes mapped correctly in DWARF.
• Type information: basic OCaml types (int, float, addr, val) with proper DW_AT_type references.

Platform Support

• Linux/ELF: full DWARF support with section-relative offsets.
• macOS/Mach-O: full support using ARM64_RELOC_SUBTRACTOR relocations for multi-object builds.
• Windows/Other: explicitly disabled with a clear error message.

Tooling

• LLDB plug-in (tools/lldb/ocaml_lldb.py) parses DWARF data, formats OCaml values, and provides the ocaml print command.
• Test suite: 9 tests covering basic functionality, function/line debugging, and type visibility.
• Validation: checks for architecture, relocations, and DWARF section correctness.

Command-Line Interface

ocamlopt -g program.ml          # Enable DWARF debug info
ocamlopt -gdwarf program.ml     # Explicit DWARF flag (future use)

Usage Example

$ ocamlopt -g example.ml
$ lldb ./a.out
(lldb) br set -n camlExample__add  # Breakpoint by function name
(lldb) br set -f example.ml -l 5   # Breakpoint by line
(lldb) run
(lldb) ocaml print x               # Print OCaml value

Implementation Details

• 37 commits, each adding part of the DWARF infrastructure.
• Multi-object linking fixes DW_AT_stmt_list offsets via external symbol references.
• Integration with register allocation updates variable locations after allocation.
• No runtime overhead when debug info is disabled (only in .o/.dSYM).

Testing

All existing tests pass. Additional DWARF tests verify:

• DWARF structure (DW_TAG_compile_unit, DW_TAG_subprogram).
• Breakpoints by function and line in both GDB and LLDB.
• Type information and variable visibility.
• Correct multi-object linking.
• Platform-specific relocation handling.

[tmcgilchrist]

Bits of un-used functions that should be hooked into something.

[yallop]

(If your work is so heavily derived from Mark's work, maybe it would make sense to credit him in the PR description, or in the webpage you created specifically to advertise for this work?)

The webpage credits another author:

Native binary debugging for OCaml (written by Claude!)

@joelreymont, could you please explain where you obtained the code in this PR?

[joelreymont]

It’s not where I obtained this PR but how.

Claude Sonnet 4.5 (Claude Code) wrote most of it with ChatGPT 5 (Codex) reviewing and Claude addressing issues in each review. Codex wrote the last 10% or so when Claude kept getting stuck.

I did not write a single line of code but carefully shepherded AI over the course of several days and kept it on the straight and narrow.

• AI: I need to keep track of variables moving across registers. This is too hard, let’s go shopping…
• Me: Hey, don’t any no shortcuts!

My work was just directing, shaping, cajoling and reviewing.

[yallop]

Why do the file headers credit Mark Shinwell? Did you (@joelreymont) incorporate Mark's code, or did Claude and ChatGPT and Codex add those headers?

[gasche]

I have several high-level criticisms of this PR and the overall contribution dynamics:

• There is an obvious problem with copyright if you reuse large amounts of people's code. The fact that the tool that produced the code attributes its copyright to a real human is a clear sign that something is an issue. (You might also wonder if that human agrees with being described as the author of code that you yourself introduce in the compiler.) At this point, merging the code in the compiler would be a legal liability.

• There hasn't been a design discussion for which approach should be chosen to integrate this feature. We would appreciate people discussing design before they dump 13K-lines PRs on us. In fact, it's a bit worse in this case: people that are expert on this topic and actively working on this feature ( @tmcgilchrist above ) have given you design advice and made recommendations for how to contribute in a way that they felt would be productive. As far as I can tell you did not take this feedback into account.

• This humongous amount of code is hard to review, and very lightly tested. (You are only testing that basic functionality works.) Inevitably the code will be full of problems, and we (the maintainers of the compiler) will have to pay the cost of fixing them. But maintaining large pieces of plausible-in-general-but-weird-in-the-details code is a large burden.

• As you might know, the OCaml compiler codebase suffers from a lack of people available to do code reviews and maintenance. See this topic for background: Maintenance bottlenecks in the compiler distribution. Your approach of submitting very large relatively-low-effort PRs creates a very real risk of bringing the Pull-Request system to a halt, especially given that, in my personal experience, reviewing AI-written code is more taxing that reviewing human-written code.
  You may think that the answer to that is to also automate the review process, or (more plausibly) to lower our quality standards: we can accept PRs based on simple/lightweight tests (themselves AI-generated), and if users find issues we can quickly use automated tools to fix them, basically having our users perform the testing work that is missing. But so far we have not decided to work in this way (and this has very real costs for users of the compiler), so these potential solutions are not available.

• Finally, you seem to not give a thought about the fact that your contributions demand work of other people. The fact that you were able to generate large amount of code that passes test is interesting, but that's only 20% of the work, the other 80% are to get the feature discussed, reviewed and integrated, and this work will be paid by you and others. But you only focus on the initial writing phase and you personal success, over-communicate on this, and do not appear to realize that this has very real costs on others. For example, a few days ago you sent a PR that was a complete waste of our collective maintenance time (Add line number breakpoints to ocamldebug #14350 ; you had us review and discuss the implementation of a feature that was in fact already available in the tool), we wasted this time, and you never apologized.

If your intent is to demonstrate that you personally know how to generate large amount of code that provide useful functionality and pass basic test, in a complex software project, then I think you have succeeded; congratulations. But if you are also hoping to demonstrate that they can be successfully integrated by upstream software projects, following the existing collaboration practices of the project, this is a complete failure. Maybe being a bit less tone-deaf and discussing things with people before you throw hundreds or thousands of lines at them would be a first step. But personally at this point I wish that you would do your "scientific experiments" (your words) on another software project, and I have personally decided that I will not invest any more time looking at your PRs for the next six months.

[joelreymont]

Here's the AI-written copyright analysis...

Executive Summary

After thorough investigation comparing the DWARF implementation in this repository with the oxcaml repository's DWARF code (authored by Mark Shinwell at Jane Street), I conclude that no code was copied from oxcaml. The implementations are fundamentally different in design, architecture, and implementation approach.

Investigation Methodology

1. Cloned oxcaml repository to ~/Work/oxcaml
2. Located DWARF implementation files in backend/debug/dwarf/
3. Compared key files across both implementations:
   • dwarf_operator.ml
   • dwarf_tag.ml
   • dwarf_world.ml
   • proto_die.ml
4. Analyzed naming conventions, type systems, architecture patterns, and code complexity

Key Findings

1. Naming Conventions

OxCaml (Mark Shinwell):

type t =
  | DW_op_addr
  | DW_op_lit0
  | Array_type
  | Formal_parameter

Current Implementation:

type t =
  | DW_OP_addr         (* uppercase, with underscore *)
  | DW_OP_lit0
  | DW_TAG_array_type  (* prefixed with DW_TAG_ *)
  | DW_TAG_formal_parameter

The naming conventions are completely different. OxCaml uses lowercase with a single underscore, while the current implementation uses uppercase with DWARF standard prefixes.

2. Type System Architecture

OxCaml:

type implicit_value =
  | Int of Targetint.t
  | Symbol of Asm_symbol.t

type t =
  | DW_op_breg0 of { offset_in_bytes : Targetint.t }
  | DW_op_breg1 of { offset_in_bytes : Targetint.t }
  | DW_op_fbreg of { offset_in_bytes : Targetint.t }

Uses parameterized types with record fields for operators that carry data.

Current Implementation:

type t =
  | DW_OP_breg0
  | DW_OP_breg1
  | DW_OP_fbreg

Uses simple enumerations without parameters. Data is handled separately.

3. Design Philosophy - Proto DIE Module

OxCaml proto_die.ml (96 lines in dwarf_world.ml):

• Mutable, imperative style
• Complex parent-child relationships with physical equality checks
• Uses Asm_label.t and Asm_symbol.t for references
• Sorting by priority with Int.Map
• Children stored in mutable maps: mutable children_by_sort_priority : t list Int.Map.t

type t =
  { parent : t option;
    mutable children_by_sort_priority : t list Int.Map.t;
    tag : Dwarf_tag.t;
    mutable attribute_values : AV.t ASS.Map.t;
    label : Asm_label.t;
    mutable name : Asm_symbol.t option;
    location_list_in_debug_loc_table : Dwarf_4_location_list.t option
  }

Current Implementation (566 lines in dwarf_world.ml):

• Immutable, functional style
• Simple list-based structures
• No parent pointers
• No mutable state
• Straightforward attribute and children lists

type attribute = {
  attr : Dwarf_attributes.t;
  value : Dwarf_value.t;
  form : Dwarf_form.t;
}

type t = {
  tag : Dwarf_tag.t;
  attributes : attribute list;
  children : t list;
  has_children : bool;
}

4. Module Dependencies

OxCaml:

open! Int_replace_polymorphic_compare [@@ocaml.warning "-66"]
open Asm_targets
module Int8 = Numbers.Int8
module Int16 = Numbers.Int16
module Uint8 = Numbers.Uint8
module Uint16 = Numbers.Uint16
module Uint32 = Numbers.Uint32
module Uint64 = Numbers.Uint64
module I = Dwarf_int
module V = Dwarf_value

Heavy use of specialized number types and assembly target abstractions.

Current Implementation:

[@@@ocaml.warning "+a-4-30-40-41-42"]

Minimal dependencies, simpler type system.

5. File Complexity Comparison

File	OxCaml Lines	Current Lines	Ratio
dwarf_world.ml	96	566	5.9x larger
dwarf_operator.ml	~200	~150	Similar
proto_die.ml	~100	~150	Similar

The implementations have completely different modularity strategies. OxCaml delegates heavily to other modules, while the current implementation is more self-contained.

6. Architecture Patterns

OxCaml Approach:

• Heavy use of functors and module composition
• Profile instrumentation throughout
• Complex label and symbol management system
• Multiple indirection layers (Debug_info_section, Abbreviations_table, etc.)

Current Implementation:

• Direct, straightforward code
• Simpler label system
• Fewer abstraction layers
• More inline logic

File-by-File Analysis

dwarf_operator.ml

Line Count:

• OxCaml: Approximately 200 lines
• Current: Approximately 150 lines

Key Differences:

• Completely different type definitions
• OxCaml uses parameterized constructors, current uses simple enums
• Different helper function approaches
• Current has to_code and to_string as simple pattern matches

dwarf_tag.ml

Constructor Names:

• OxCaml: Array_type, Class_type, Formal_parameter
• Current: DW_TAG_array_type, DW_TAG_class_type, DW_TAG_formal_parameter

Complete naming divergence indicates independent implementation.

dwarf_world.ml

Architecture:

• OxCaml: Thin emission layer delegating to specialized modules
• Current: Self-contained world model with compilation unit management

Functionality:

• OxCaml: emit0 function coordinates multiple specialized modules
• Current: Direct DWARF structure building and emission

proto_die.ml

State Management:

• OxCaml: Mutable records with complex parent-child graph
• Current: Immutable records with simple tree structure

Reference System:

• OxCaml: Assembly labels and symbols with bidirectional links
• Current: Simple offset-based references

Conclusion

The DWARF implementation in this repository is independently generated code that does not derive from Mark Shinwell's oxcaml implementation. While both implement the DWARF specification and thus share conceptual similarities (as they must to comply with the standard), the actual code, design patterns, naming conventions, and architectural approaches are fundamentally different.

Evidence Summary:

1. Different naming conventions (DW_OP_* vs DW_op_*)
2. Different type system (simple enums vs parameterized types)
3. Different design philosophy (functional vs imperative)
4. Different complexity and modularity patterns
5. Different module dependency structures

[gasche]

I will go ahead and close this PR.

• There is nothing wrong with further discussion of this topic, closing does not prevent that. (I would rather avoid large AI-generated comments if you can, please.)
• On the other hand, none of the people who could plausibly be interested in reviewing and supporting a DWARF-support PR seem willing to consider doing the work for that one iteration, so I think that this has no chance of being merged.

This is not a value judgment of the code that is being proposed (I haven't looked at it in enough details to comment), nor on the usefulness of the feature. It is perfectly possible that this development approach provides good value for some software projects (for example those that are entirely your own), when it aligns with the current code-integration and maintenance practices of the project. Currently github/ocaml as it is organized is not able to work in this way. (This may change in the future and then one could reconsider.)

[joelreymont]

• There hasn't been a design discussion for which approach should be chosen to integrate this feature.

I agree with you. My goal here was to get it done and verify that it works.

• This humongous amount of code is hard to review, and very lightly tested. (You are only testing that basic functionality works.)

I would disagree with you here. AI has a very deep understanding of how this code works. Please challenge me on this.

It was able to tell me that it needed to track local variables moving across registers to be able to add enough DWARF information to display the variables. I have the whole session logs of Codex reviewing Claude and the insights are deep!

• As you might know, the OCaml compiler codebase suffers from a lack of people available to do code reviews and maintenance. See this topic for background: Maintenance bottlenecks in the compiler distribution.

I’ll gladly help with code reviews and maintenance here but I suffer from a lack of funding.

Your approach of submitting very large relatively-low-effort PRs creates a very real risk of bringing the Pull-Request system to a halt, especially given that, in my personal experience, reviewing AI-written code is more taxing that reviewing human-written code.

I do not intend to submit any more PRs of this kind. This was a proof of concept and an attempt to push AI as far as it would go. I believe that it has succeeded brilliantly! Also, I would not call this a low-effort PR.

You may think that the answer to that is to also automate the review process, or (more plausibly) to lower our quality standards: we can accept PRs based on simple/lightweight tests (themselves AI-generated), and if users find issues we can quickly use automated tools to fix them, basically having our users perform the testing work that is missing.

No, I do not believe in lowering standards, neither here nor at all! I believe in maintaining high quality standards and its observation, based on experience, that AI could help.

AI can generate all kinds of tests, from lightweight to heavyweight, it all depends on who’s shepherding it. If you feel that this PR needs more tests then please let me know in what areas and I’ll present the tests, as deep as you want them to be.

For example, a few days ago you sent a PR that was a complete waste of our collective maintenance time (Add line number breakpoints to ocamldebug #14350 ; you had us review and discuss the implementation of a feature that was in fact already available in the tool), we wasted this time, and you never apologized.

I apologize for wasting your time with this and apologize for never apologizing. I beg to differ, though, that the bulk of the PR, the actual editing of the command line with no dependencies on outside libraries, was already available in the tool.

If your intent to demonstrate that you personally know how to generate large amount of code that provide useful functionality and pass basic test, in a complex software project, then I think you have succeeded; congratulations.

My intent is to demonstrate that AI can write my (our?) next compiler, to the highest standards of quality. I’m doing this experiment in a brand new project as, indeed, that’s the right place for it.

Once again, my intent was never to waste your time but to show what’s possible. I apologize for any time wasted and assure you that I won’t be submitting any similar PRs any time in the near future.

[gasche]

You are welcome. I am certainly not trying to suggest that your work has no value (I do agree that getting good native-debugging support in the OCaml compiler would be very nice; we have in fact known this for years and the difficulty is to find a way to do it that maintainers agree comes at a reasonable maintenance burden). I think that it is a case of different-to-the-point-of-being-incompatible software development processes (rather than a given process being fundamentally right or wrong), and I think that the uncertainty here is in part caused by our lack, on the upstream side, of a clear policy for what we expect regarding AI-assisted code contributions.

[bluddy]

Looking over this PR, the vast majority of the code is a DWARF library by itself. This should really not live in the compiler, nor should it become a maintenance burden for the core devs. Ideally, the DWARF library would be completely separate, and the compiler would only use its API to output what it needs after piping the relevant data through the different ASTs. This becomes complicated by bootstrapping issues, but perhaps it could be integrated as an optional, dynamic dependency to deal with this.

Assuming that the oxcaml version is similar, my comment would still be relevant to that effort.

[joelreymont]

we have in fact known this for years and the difficulty is to find a way to do it that maintainers agree comes at a reasonable maintenance burden).

I’m not a compiler developer by trade, although I’ve done all sorts of development over the years. I’m approaching this strictly as a user, perhaps a power user. I used to look at my needs and wants, and sulk because they were not addressed.

Damn, I can’t debug OCaml on my Mac because there’s no DWARF info.

Oh, wow, Jane St released OxCaml! Yay, native debugging on the Mac! Darn, all kinds of package hell is breaking loose. Let me offer my help… Tarides takes care of maintenance of these bits? I’ll talk to them, maybe they can hire me to fix it.

Alas, I’m still here and my needs are not addressed. But, hey, there’s AI and it seems to one-shot fairly complex stuff in different languages, from just a Github issue. Maybe I can try it…

Wow, oh wow! My needs are finally taken care of!

The code seems clean and well-written. I can understand what AI it’s doing and why. All tests pass, documentation and comments are in order. I can definitely use this!

I think that it is a case of different-to-the-point-of-being-incompatible software development processes (rather than a given process being fundamentally right or wrong), and I think that the uncertainty here is in part caused by our lack, on the upstream side, of a clear policy for what we expect regarding AI-assisted code contributions.

That is something I’ve been pondering myself. I tried approaching several projects this way, trying to take care of things that bother me. The reaction is similar across the board. Folks want a nuanced and thorough discussion, as well as buy-in, before an implementation is submitted.

This is incompatible with what I found to be the most efficient way of using AI, though. It doesn’t need that much input. I can kick off a project just by telling it to add DWARF debugging information to OCaml, such that breakpoints and source code listings, as well as variable printing, works in lldb and gdb. I tell it to follow the practices of the OCaml code base, and to make sure new tests are compatible with existing test infrastructure. Then I add that I want all new code to be thoroughly tested.

AI goes away and does the work, asking me for input. I review what it's doing and make sure it doesn't take shortcuts. This is more art than science at this point. Different models produce different work so you need to know which model to use. For example, some models are great at code review and some are better at coding. You also need to carefully steer the AI and make sure it stays on track.

I don’t know of a single project that’s ready for this kind of development process. It often gets emotions high (is artisanal coding dead? will AI take my job?) and creates a lot of friction.

This is why I decided to have AI write me a new Lisp compiler, targeting small binaries and bare metal targets. I’ve been cooking it for the last few days and the results are beyond awesome! Next, I will have AI write me a graph database and a Lisp version of Slint. I’m 100% sure it will work and work well, and I will have it in the next few weeks. The best part is that I won’t have to bother anyone with my forced contributions and can just showcase the end results.

To summarize, I love the new AI sausage and, having visited the sausage factory and done a thorough investigation, I’m not concerned with how the sausage is made. I won’t be forcing my sausage-making processes on anyone and will go make my own sausage!

[joelreymont]

Assuming that the oxcaml version is similar, my comment would still be relevant to that effort.

I may be mistaken but I think the OxCaml implementation is very much tied to the compiler.

[dra27]

@joelreymont wrote:
AI has a very deep understanding of how this code works. Please challenge me on this.

With pleasure. s/AI/LLM. There is no "deep understanding" by the model of how this code works - the fact that these models and the platforms built over them can produce meaningful code is a marvel of engineering, but to use these tools in the belief that they "understand" what they're producing conveys a critical lack of even a high-level understanding of what they're actually doing.

@gasche wrote:
We would appreciate people discussing design before they dump 13K-lines PRs on us

I wholeheartedly agree with this! I'd also add that some core developers (who are compiler developers by trade) are actually experimenting with and using these same tools too, but haven't suddenly turned into 13kLoC PR-opening AI superheroes.

[joelreymont]

With pleasure. s/AI/LLM. There is no "deep understanding" by the model of how this code works - the fact that these models and the platforms built over them can produce meaningful code is a marvel of engineering, but to use these tools in the belief that they "understand" what they're producing conveys a critical lack of even a high-level understanding of what they're actually doing.

That's not what I meant. My challenge is to question the AI about the codebase. Ask intricate questions and I'll tell you what it comes up with.

Why? To prove that, however the sausage is made, the end product is no worse that what I (perhaps you too?) would have written.

[yallop]

Here's my question: why did the files that you submitted name Mark Shinwell as the author?

[joelreymont]

Here's my question: why did the files that you submitted name Mark Shinwell as the author?

Beats me. AI decided to do so and I didn't question it.

I did ask AI to look at the OxCaml implementation in the beginning.

[tmcgilchrist]

Looking over this PR, the vast majority of the code is a DWARF library by itself. This should really not live in the compiler, nor should it become a maintenance burden for the core devs. Ideally, the DWARF library would be completely separate, and the compiler would only use its API to output what it needs after piping the relevant data through the different ASTs. This becomes complicated by bootstrapping issues, but perhaps it could be integrated as an optional, dynamic dependency to deal with this.

Assuming that the oxcaml version is similar, my comment would still be relevant to that effort.

@bluddy
The data types, serialisation code, and de-serialisation code is all pretty generic. You could reasonably write a library to do that, I've been writing one myself (tmcgilchrist /durin). I'm not sure it's such a burden as the DWARF 5 specification is fixed and it really isn't that much code. The interesting part which needs to be compiler specific is how you describe OCaml types as DWARF information and tracking enough information so you can accurately describe things like inlining, register spills, and line number mappings. Having this code is the cost of having good debugging support.

[joelreymont]

May I bring some of your attention back to the facts that

• this implementation is fully working,
• it looks well-written and
• it's well-tested

I think this counts for something!

[gasche]

What does it print when asked to show a value of an algebraic data type, for example a binary tree?

[joelreymont]

What does it print when asked to show a value of an algebraic data type, for example a binary tree?

What do you mean?

What does lldb show when built using this PR?

[joelreymont]

I'm gonna post the pretty-printing examples shortly.

[gasche]

I have the impression that this conversation is becoming less productive, so I will move ahead and "lock" the topic. The Discuss forum already has a topic about this PR where further discussion can happen -- assuming that it does stay reasonably on track.