Add support for event tracing for visualizing where time is spent during execution

[andygrove]

What is the problem the feature request solves?

TIL I learned about Chrome's tracing log format. I built a very quick POC with Comet. It would be nice to have the ability to produce these visualizations to help track down performance issues in queries. I used a script from this repo to generate the image.

Describe the potential solution

No response

Additional context

No response

[andygrove]

@alamb This may also be interesting to explore for DataFusion

[alamb]

@geoffreyclaude added some hooks recently in DataFusion to support various tracing libraries. See the really nice example here:

• https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/tracing.rs

For local test, I have also had great luck with https://github.com/mstange/samply which also uses the chrome tracing UI

[geoffreyclaude]

@geoffreyclaude added some hooks recently in DataFusion to support various tracing libraries. See the really nice example here:

• https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/tracing.rs

For local test, I have also had great luck with https://github.com/mstange/samply which also uses the chrome tracing UI

I've also recently released our internal tracing code to https://github.com/datafusion-contrib/datafusion-tracing. It won't show execution time explicitly (just one big span per execution node) but maybe there's something to be done here.