cmd/trace: view large trace all at once

[josharian]

Here's an example.

x_test.go:

package p

import (
    "sync"
    "testing"
)

func BenchmarkBlocking(b *testing.B) {
    var mu sync.Mutex
    b.RunParallel(func(pb *testing.PB) {
        var x int
        for pb.Next() {
            for i := 0; i < 10000; i++ {
                x *= 2
                x /= 2
            }
            mu.Lock()
            for i := 0; i < 1000; i++ {
                x *= 2
                x /= 2
            }
            mu.Unlock()
        }
    })
}

Do this:

$ go test -bench=. -trace=trace.out
BenchmarkBlocking-8   500000          3401 ns/op
$ go tool trace p.test trace.out

Then click on View trace.

Result: Browser tab crashes.

@egonelbre did an initial analysis on a related trace that came from real code (thanks, Egon!):

Preliminary analysis seems to indicate that the trace-viewer is blowing some sort of internal memory limit in Chrome. i.e. the trace json file is 133MB and chrome dies after allocating 800MB of RAM.
Found somewhat related issue google/trace-viewer#298

I did some experimenting and was able to load 1.07 million events, but not 1.08M. It seems that trace-viewer can load around ~1M events. The whole dataset is ~1.5M events.
From DOM/JavaScript perspective it is quite impressive that it can handle that much and the tool stays quite responsive.
I think it might be better to do this at the go trace tool level, enforce the limit that you cannot look over 1M events. And then from the selection part you can select the range you wish to see.

Note that there is also a possibility of removing some particular counters from the trace.json they make up over half of the events.
Alternatively we can reduce counters using some similarity: https://play.golang.org/p/GnJIyzmsA5

Given how easy it is to create giant (and thus unusable) traces, I think we should handle them gracefully, since the trace-viewer can't.

One option (perhaps not great): 'go tool trace' could allow you to specify a time window to display, and refuse to send windows that are too large to the trace viewer.

/cc @dvyukov @egonelbre

[egonelbre]

After thinking about it, the core of the trace-viewer isn't complicated, and decided to put together a quick prototype in JS: https://github.com/egonelbre/trace-spector

This version can load the large file and can be improved with regards to memory usage (currently stabilizes at ~0.5GB with 1.5M events). Currently, trace-viewer AFAIK uses at least one object per event which means 1.5M objects. But I have a feeling that no matter how far we engineer it, the trace-viewer-format based solution is not going to be sufficient for Go traces. I think we should be able to do much better by pre-processing and packing.

Essentially, we are hitting the browser limit rather than anything else. I think using Go + SDL/GL (whatever) would be better.

I'll try to finish up the nanoseconds support in trace-viewer, I think it will be sufficient ATM. Once I have some time left over from other things I'll start on the go based trace inspector (unless someone already beats me to it). And by no means it will be as feature-complete as trace-viewer. And no promises this happening any time soon, I'm already hitting my daily limit with regards to typing too much.

[josharian]

Essentially, we are hitting the browser limit rather than anything else. I
think using Go + SDL/GL (whatever) would be better.

Another possibility would be to do like (say) Google maps and drop
detail when zoomed out. It'd make zooming expensive where it is now
cheap, but then in theory you should be able to hand arbitrary traces.

My main concern at this point, though, is to make sure that folks'
initial perceptions of the tracing tool are positive, since initial
reactions are hard to change. That might just be a matter of getting
the messaging right.

Since you're versed in the guts of the trace-viewer, maybe it's worth
adding a simple check in the trace viewer to replace the current crash
with a "trace too big" alert? (We should also fix 'go tool trace', of
course, but redundant error checking is ok.)

[egonelbre]

Another possibility would be to do like (say) Google maps and drop
detail when zoomed out. It'd make zooming expensive where it is now
cheap, but then in theory you should be able to hand arbitrary traces.

The problem isn't in performance. It performs quite well. The main problem is memory usage and the way that trace events are stored in memory.

LOD would be nice, but it adds a lot of machinery. Even if we implement this in trace-viewer, I'm not sure that such wide-scale change would be accepted.

I'll look into whether creating a custom importer would give some improvement - but I feel it still will need to unpack into separate JavaScript objects to make it usable with trace-viewer, which essentially takes us back to the beginning.

Since you're versed in the guts of the trace-viewer, maybe it's worth adding a simple check in the trace viewer to replace the current crash with a "trace too big" alert? (We should also fix 'go tool trace', of course, but redundant error checking is ok.)

I'm not sure how well it can be detected or on what basis it should be done. From Go side we create a limited subset of possible traces - the conditions that hold in Go may not necessarily translate to other traces e.g. Chrome, Android. Generic detection for crash would be problematic, if doable at all - hence it's not a good solution for trace-viewer.

I need to inspect why exactly is it crashing at the moment and maybe it can be mitigated instead. Although with long traces we might still hit a memory limit.

[egonelbre]

I did some further inspecting and testing.

Loading the huge file works in Chrome Canary (45.0.2448.0), but still uses ~1.5GB during importing. So this seems to be a Chrome issue that has been fixed.

[josharian]

That's great! Thanks, @egonelbre.

I think we should leave this open, because we should probably still have a way for users to manage large traces. (Perhaps a trace from a live server or just running many benchmarks could end up needing 15GB instead of 1.5GB?) Dropping to Unplanned though. Hopefully we can do something useful for Go 1.6. There could also be third party tools to trim traces before sending them to the viewer, such as @egonelbre started.

[dvyukov]

@egonelbre Having our own specialized viewer would be great. Then we can improve input format, make it consume less memory, improve visual appearance, improve navigation (e.g. go to a goroutine trace from full trace), fix remaining bugs (e.g. some elements don't display stack traces), etc.
Also, the impression that I got is that making changes to trace-viewer is a very slow process. And they silently break interfaces multiple times (e.g. at some point arrows not starting inside of a slice disappeared from UI, and now you say that lean config is broken).

[egonelbre]

@dvyukov do you have a nice list of things that need to be visualized? Any preferences for tech used - i.e. browser vs native+gl vs native+something? I would like to make it such that it can visualize 1M events with 30FPS while streaming them in. Basically you should be able to view some portion of the trace live while program is emitting it.

impression that I got is that making changes to trace-viewer is a very slow process.

Partly, yes, partly, no... I.e. someone working full time would probably get the patches, features much faster in than I. For me there's a quite large context switch to modify it and I am able, at best, to take 0.5 day from week to work on it.

[egonelbre]

Any preferences for tech used...

Anyways, after a bit of experimenting with gl, the cgo overhead currently is significant i.e. ~300ns per call. I wouldn't want to spend most of the CPU time for cgo. Implementing a gl command-buffer (same principle that Chromium uses) to remove the overhead would be one possibility - and something that rest of people could significantly benefit from.

Whole viewer will only need simple primitives: rect, line, bezier, triangle, text, bitblt, maskblt, alphablend. Implementing such canvas in Go is mostly trivial, since content should be pixel-aligned anyways and we don't have to worry about aliasing that much (with the exception of bezier and triangle).

Thoughts?

[dvyukov]

@egonelbre I think the way to go is html+browser. There are currently no native GUI tools in std lib and no packages to do it. I feel that a native app can have more portability issues across operating systems, devices, graphics cards, etc. All current tools use html/svg+browser.

[egonelbre]

@dvyukov you've got to start building GUI tools somewhere :). The Go canvas approach would have very few requirements - i.e. capture mouse and keyboard input and display a bitmap. But sure, I can see the reasoning and will do the HTML+browser version first. PS. do you have a nice list of things that you need to be visualized.

[dvyukov]

@egonelbre Do you mean "logical" things (like syscall or goroutine unblocking) or "physical" things (like slice and arrow)?

[dvyukov]

@egonelbre regarding html vs canvas, frankly I have no idea how it will look like and what consequences will be. If you want to use canvas, please ask on golang-dev.