chore: Add Java Flight Recorder profiling to TPC benchmarks#3597
chore: Add Java Flight Recorder profiling to TPC benchmarks#3597andygrove merged 3 commits intoapache:mainfrom
Conversation
andygrove
changed the title
andygrove
changed the title
Add --jfr and --jfr-dir flags to run.py for capturing JFR profiles from Spark driver and executors. Use disk=true for reliable recordings in Docker environments. Add stop_grace_period to worker containers for graceful JVM shutdown.
andygrove
force-pushed
the
jfr-profiling
branch
from
7d5718c to
b5b0647
Compare
andygrove
changed the title
|
One thing to note when running on K8s is to make sure the executor pods are not removed by spark before you can access the jfr file. Also for production cases, one can use the profiling feature available in Spark - https://github.com/apache/spark/tree/b3703755d80585297367d539de9fa8c5783b1c6b/connector/profiler |
parthchandra
left a comment
parthchandra
left a comment
There was a problem hiding this comment.
Added a minor comment, but this lgtm
Thanks. I started a draft PR to add k8s support to these scripts, so will add this there.
I will look at this too. |
2 participants
Which issue does this PR close?
N/A
Rationale for this change
Make it easier to run the TPC benchmarks in the repository with JFR.
What changes are included in this PR?
--jfrand--jfr-dirflags torun.pythat enable JFR profiling for both Spark driver and executorsHow are these changes tested?
I tested on macOS with standalone mode and in docker-compose