Snarky flamechart

[mrmr1993]

This PR

• creates the Flamechart module, which generates JSON trace events (spec) that can be shown as a flamechart in chrome://tracing/
• adds a Snarky_log submodule, containing the module functor Constraints (Snarky : Snarky.Snark_intf.Basic)
  • this hooks into constraint_count (which has a new optional log parameter) and the newly-exposed conv function from the Snarky library
• pulls in an example (elections) from the Snarky module, adapting it as an example/test for generating a flamechart.

Trivial example:

open Snarky
module Snark = Snark.Make (Backends.Bn128.Default)
open Snark
module Constraints = Flamechart.Snarky_log.Constraints (Snark)

let () = Flamechart.to_file "output.json" @@
  Constraints.log_func ~input:Data_spec.[Field.typ; Field.typ] Field.Checked.mul
    ~apply_args:(fun mul -> mul Field.one Field.one)

Checklist:

• Tests were added for the new behavior
• All tests pass (CI will check this if you didn't)
• Does this close issues? List them:

[bkase]

Can you show a screenshot of what an example trace looks like?

Also @cmr built out some infrastructure with chrome tracing as well see src/app/trace-tool and the o1trace library. Not sure if flamecharts use a different structure and/or if we want to move some of the logic that is currently in Rust into OCaml

[mrmr1993]

example trace

Not very exciting, but Izaak said it might be useful for him:

see src/app/trace-tool and the o1trace library.

Yes, it looks like they are doing much the same work, thanks. I'll try and refactor the common stuff into its own library (webkit_event_trace?) and call that from my snarky lib and o1trace.

Separately, it looks like o1trace shares this buffer across threads: do we not need to worry about threads racing and potentially clobbering it?

I'm not officially working this week, but I'm sure I can find some time to put something together.

[bkase]

A lot of us are out this week (including me), don't worry about doing anything this week!

Separately, it looks like o1trace shares this buffer across threads: do we not need to worry about threads racing and potentially clobbering it?

My understanding of Async's runtime is that the threads are green and mapped to a single OS-thread. Also see https://dev.realworldocaml.org/concurrent-programming.html . So this is safe as written.

Let me know if you see something I'm missing here though.

webkit_event_trace seems good

[mrmr1993]

I've updated this so that the new library (now called Snarky_log) and O1trace both use the same library (Webkit_trace_event) for creating the events. Webkit_trace_event has 2 output modes:

• Webkit_trace_event.Output.Binary, which gives output that src/app/trace-tool consumes
  • this is what O1trace uses, to keep things as similar to how they were as possible
  • trace-tool uses some custom event types to set state information (e.g. pid, thread names, etc.), and relies on parsing all the messages in one go to get the right information for all of its output
• Webkit_trace_event.Output.JSON, which gives Yojson compatible JSON output (but doesn't depend on Yojson explicitly)
  • this outputs JSON using only the information stored in each event, so that it can stream output if we need it
  • this JSON is necessarily different to the JSON that trace-tool will generate in its full-pass

I've held off touching or reimplementing trace-tool, but I can do this if it seems like a good idea.

Sample code for Snarky_log is barely different to before:

open Snarky
module Snark = Snark.Make (Backends.Bn128.Default)
open Snark
module Constraints = Snarky_log.Constraints (Snark)

let () = Snarky_log.to_file "output.json" @@
  Constraints.log_func ~input:Data_spec.[Field.typ; Field.typ] Field.Checked.mul
    ~apply_args:(fun mul -> mul Field.one Field.one)

[emberian]

👍 to Webkit_trace_event. To further elaborate on what @bkase said, it's safe as long as the code using the buffer is synchronous (doesn't do anything with Deferred etc), which it is. The final submission of the buffer with finish causes a synchronous copy into a buffer internal to async_unix, at which point the buffer is free to be used again.

[emberian]

trace-tool being implemented in Rust is a bit of a holdover from when I was generating hundreds of MBs of trace events, but now that traces are substantially smaller it being in ocaml would be fine too.