Size limit for the JSON trace event format?

[flx42]

Hello, love the UI, very clean and responsive!

I'm wondering if there is still a size limit on the size of uploaded JSON files in the trace event format. My understanding is that chrome://tracing was limited to 256MB: google/trace-viewer#627

I tried to load a 900MB trace file into ui.perfetto.dev, and I got a crash:

To assist with debugging please attach or link to the trace you were viewing.

Viewed on: https://ui.perfetto.dev

Uncaught Error: RuntimeError: abort(undefined). Build with -s ASSERTIONS=1 for more info.
RuntimeError: abort(undefined). Build with -s ASSERTIONS=1 for more info.
    at abort (https://ui.perfetto.dev/v12.1.269/engine_bundle.js:26:11478)
    at _abort (https://ui.perfetto.dev/v12.1.269/engine_bundle.js:26:76163)
    at <anonymous>:wasm-function[42]:0x3131
    at <anonymous>:wasm-function[85]:0x6333
    at <anonymous>:wasm-function[3598]:0x20680d
    at <anonymous>:wasm-function[3581]:0x204ae7
    at <anonymous>:wasm-function[2178]:0x157818
    at <anonymous>:wasm-function[1839]:0x12986b
    at <anonymous>:wasm-function[785]:0x728eb
    at <anonymous>:wasm-function[3429]:0x1f047d

stderr: 
undefined

v12.1.269 a1386eaf2121de029e239a8d9b414ebcdd4c8257
UA: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36

The README mentions:

web-based UI to visualize and explore multi-GB traces.

But I figure it might only apply for the Perfetto protobuf format, is that correct?

If yes, is there an existing way to convert from the trace event json format to the Perfetto protobuf format?

Thanks!

[chromy]

Hey! Happy to hear you are enjoying the UI 😄 So by default ui.perfetto.dev is limited the maximum amount memory your browser will allow a WASM application to use. This is normally 2-4gb. When ui.perfetto.dev parses the trace file (whether .json or .proto or another format) it normally ends up using more memory than the underlying trace file, mostly because we trade some memory for better query performance. The exact expansion factor varies based on the format and the content of the trace. In this case it sounds like the 900mb ends up using more than the 2-4gb when loaded.

The bad news is we don't have a way to convert .json traces to .proto traces, and even if we did it wouldn't help with this.

The good news is we have a mode designed exactly for analysing very large trace files in the UI! The steps are here: https://perfetto.dev/docs/quickstart/trace-analysis#setup but in brief:

• Build or download the trace_processor_shell binary
• You run trace_processor_shell --httpd yourtrace.json on your computer
• Open ui.perfetto.dev
• It detects that the trace_processor is running locally and offers to connect to it:

• Click Yes, use loaded trace

You should then be able to load traces as large as however much RAM you have + full native performance.

The UI should detect OOM conditions and pop up a dialog explaining the --http mode instead of crashing with the error message you saw ...but unfortunately we broke this feature in the most recent UI release. Hopefully this should be fixed in the next UI release.

Hopefully that helps! Let me know if you have any further questions.

[flx42]

@chromy thanks for the detailed explanation!

My main workstation is a Windows 10 desktop, and I ssh to a headless Linux server to collect trace event traces. I tried running the trace processor under WSL2 on this desktop, using netsh to proxy to port 9001 inside the Ubuntu WSL2 VM, but it didn't detect the trace processor for some reason when opening ui.perfetto.dev inside Chrome on Windows.

I then tried on a Linux laptop (trace_processor + Chrome browser) and then it worked like a charm, thanks! I'm wondering, does the trace processor need to sit on localhost? (e.g. as a security limitations?). If not, it might be useful to add an option to configure the URL of the trace_processor server.

Last question: on the Linux laptop, I see 10GB of resident memory when loading this 900MB trace. Does that mean the WASM version of the import would also need 10GB of RAM or are the RAM requirements not directly related?

[LalitMaganti]

I've used ssh port forwarding to proxy the port and it has worked correctly so I'd double check that this is working on Windows. Unfortunately, we cannot support arbitrary pointing to arbitrary URLs because of CORS restrictions - we only support localhost as it's safe for all users. With proxying though, you should always be able to port forward and that should work.

10GB of resident memory is pretty out there. We'd usually expect a 2x increase in memory compared to the raw trace but seems like something has gone wrong here as this is more than 10x. We've observed this in some internal bugs as well so seems like a real issue.

I'm currently working on some other projects so I will be unable to look into this for a bit but I will come back to this once those settle a bit more.

[primiano]

Note that with r.android.com/1539396 * trace_processor_shell can be built as a native windows executable. If you want I can build it for you and put the exe somewhere.
Also i am pretty sure I tried it in WSL2 and it worked without having to do any port forwarding. TCP ports opened on 0.0.0.0 on Linux should be reachable from the Windows side (unless you use some custom network config). Would be curious to understand more what's happening here.

• Unfortunately it got reverted in r.android.com/1608095 because broke chrome win32, I'll reland in the next weeks

[flx42]

Thanks @LalitMaganti and @primiano for the detailed answers!

If you want I can build it for you and put the exe somewhere.

I'm good for now with my setup, no need to do that just for me. But if you do publish the binary under get.perfetto.dev, I will try.

I'll also take another stab at WSL2 later. The server seemed accessible since I was able to access nginx running under WSL2 from my browser. Trying to access localhost:9001 from my browser yielded connection reset instead of unable to connect, so I suppose I'm able to reach the RPC server (but obviously it didn't like my HTTP GET request).

[flx42]

Wait, now it works fine from WSL2. Perhaps a PEBKAC previously, sorry about that.

[LalitMaganti]

The above change should fix the very high memory usage on JSON traces.

[flx42]

Thanks @LalitMaganti!