Export to an analysis friendly Data Structure?

[mdrakiburrahman]

Thank you for this wonderful library, it's super fast!

What are your thoughts on outputting to an analysis friendly data structure, e.g. JSON files (per type of schema, such as threads etc.)?

The use-case is, I'd like to run hprof-slurp on a folder full of dumps, have it spit out JSON files, then run something like Apache Spark on the JSON files, to plot out trends.

[agourlay]

Thanks for the kind words 👋

Exporting the result in a structured fashion makes a lot of sense.
I think JSON would be a good starting point.

Do you maybe have some ideas regarding the actual schema?
Maybe we could adopt the same schema as existing toolings for interoperability.

[mdrakiburrahman]

I was basically thinking of taking the txt file the CLI dumps out, and seeing if there's a way to "JSON-ize" it instead :)

(E.g. right now, I wrote up a Reg-Ex based text parser to load the data into a DataFrame, it's a bit hacky)

Perhaps something simple, like using serde to dump out the underlying data structure collected into JSON could suffice?

I understand some of the thread stacks are a bit gross to store in a JSON field, but if it's newline encoded, it should be fine.

One option is, a "giant JSON" file that stores everything with k:v.
Another thought is, "each type of data" is exported into a different JSON file.
(E.g. the top-n goes into one file, the threads goes into another).

Perhaps the giant JSON would be a good start?

[agourlay]

I have added the export of the memory stats as JSON in v0.6.0

Please give it a try and provide feedback!

[mdrakiburrahman]

Amazing! Thank you @agourlay - this looks great. I'll close this issue, and if any further enhancement is required, will open a more specific one. Cheers :)

[mdrakiburrahman]

@agourlay - thanks again for this wonderful tool. I wrote about how I immensely benefited from your work here, as an FYI:

https://www.rakirahman.me/spark-otel-plugin/#examining-heap-dump-data-as-a-time-series-with-hprof-slurp

All the other hprof analysis tools are GUI based and near impossible to automate, hprof-slurp needs a lot more recognition!

[agourlay]

This is a great write up, I really appreciate the shout-out!

All the other hprof analysis tools are GUI based and near impossible to automate

It seems that hprof-slurp is a perfect match for this kind of automated analysis without GUI 👍

hprof-slurp needs a lot more recognition!

Yes! 😸

Keep me posted me if you need more features.

[mdrakiburrahman]

Will do :)