Skip to content

Add SparkPlanInfo constructor compatible with Databricks' Spark fork #9888

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Nov 6, 2025
Merged

Add SparkPlanInfo constructor compatible with Databricks' Spark fork #9888

merged 2 commits into from
Nov 6, 2025

Conversation

@charlesmyu
Copy link
Contributor

@charlesmyu charlesmyu commented Oct 30, 2025 •
edited
Loading

What Does This Do

The primary addition in this PR is support for the Databricks-specific SparkPlanInfo constructor. The available public methods and constructors on Databricks 17.3 LTS (based on Spark 4.0.0), for example, look like this:

def <init>(
	nodeName: String, 
	simpleString: String, 
	children: Seq[org.apache.spark.sql.execution.SparkPlanInfo], 
	metadata: Map[String,String], 
	metrics: Seq[org.apache.spark.sql.execution.metric.SQLMetricInfo], 
	estRowCount: Option[BigInt], 
	rddScopeId: String, 
	explainId: Option[Int]
): org.apache.spark.sql.execution.SparkPlanInfo;
def <init>(
	nodeName: String, 
	simpleString: String, 
	children: Seq[org.apache.spark.sql.execution.SparkPlanInfo], 
	metadata: Map[String,String], 
	metrics: Seq[org.apache.spark.sql.execution.metric.SQLMetricInfo], 
	estRowCount: Option[BigInt], 
	rddScopeId: String
): org.apache.spark.sql.execution.SparkPlanInfo;

val nodeName: <?>;
val simpleString: <?>;
val children: <?>;
val metadata: <?>;
val metrics: <?>;
val estRowCount: <?>;
val rddScopeId: <?>;
val explainId: <?>;

The last three methods (estRowCount, rddScopeId, explainId) are non-standard, and thus must be accounted for in these specific constructors.

Note that we attempt the Databricks constructor first since it has more info, and then the standard constructor. Both are unlikely to exist simultaneously, but this is just to be defensive. In addition, we try to take advantage of the MethodHandles class where possible for reflection, but can't use it to load any of the getters in SparkPlanInfoUtils as all the methods are inherited (MethodHandles uses getDeclaredMethod which excludes inherited methods).

Additional, smaller changes worth noting:

  • Moves the SparkPlanInfo constructor logic into a dedicated class, SparkPlanInfoUtils.
  • Makes a small change in AbstractSparkPlanSerializer to improve how we reflect the simpleString method, specifically preferring the .invoke method provided by MethodHandlers as a nice wrapper for exception handling logic.
    • This shouldn't change any behaviour of how we get simpleString.
  • Swaps us from matching on parameter count to matching on explicit parameter types to avoid false matches, especially as we handle a more diverse set of constructors.
  • We also explicitly reflect all getters (i.e. nodeName(), simpleString(), children(), and metrics()) to avoid version incompatibilities causing muzzle to reject the entire Spark tracer.
    • e.g. in Scala 2.13.0 , scala.Seq (used by children and metrics) was migrated from scala.collections.Seq to scala.collections.immutable.Seq.

Motivation

We would like to support Spark jobs for column-level metadata extraction, but they use a different signature for their SparkPlanInfo constructors. This PR fixes that.

Additional Notes

Tested in Databricks and we can see the fields show up as expected (link).

Contributor Checklist

Jira ticket: DJM-974

@datadog-datadog-prod-us1
Copy link
Contributor

datadog-datadog-prod-us1 bot commented Oct 30, 2025 •
edited by datadog-official bot
Loading

🎯 Code Coverage
• Patch Coverage: 100.00%
• Total Coverage: 59.67% (+0.05%)

View detailed report

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 0bfaa06 | Docs | Datadog PR Page | Was this helpful? Give us feedback!

@pr-commenter
Copy link

pr-commenter bot commented Oct 30, 2025 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master charles.yu/djm-974/spark-plan-databricks
git_commit_date 1762435787 1762442890
git_commit_sha 19d774d 0bfaa06
release_version 1.56.0-SNAPSHOT~19d774d48c 1.56.0-SNAPSHOT~0bfaa064cd
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1762444859 1762444859
ci_job_id 1219866538 1219866538
ci_pipeline_id 81464370 81464370
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-3-1oo6797q 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-3-1oo6797q 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 53 metrics, 12 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.56.0-SNAPSHOT~0bfaa064cd, baseline=1.56.0-SNAPSHOT~19d774d48c

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.049 s) : 0, 1048510
Total [baseline] (8.624 s) : 0, 8623512
Agent [candidate] (1.05 s) : 0, 1049915
Total [candidate] (8.64 s) : 0, 8639525
section iast
Agent [baseline] (1.178 s) : 0, 1177852
Total [baseline] (9.24 s) : 0, 9240210
Agent [candidate] (1.186 s) : 0, 1185664
Total [candidate] (9.302 s) : 0, 9301981
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.049 s -
Agent iast 1.178 s 129.343 ms (12.3%)
Total tracing 8.624 s -
Total iast 9.24 s 616.699 ms (7.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.05 s -
Agent iast 1.186 s 135.749 ms (12.9%)
Total tracing 8.64 s -
Total iast 9.302 s 662.456 ms (7.7%) 
gantt
    title insecure-bank - break down per module: candidate=1.56.0-SNAPSHOT~0bfaa064cd, baseline=1.56.0-SNAPSHOT~19d774d48c

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.452 ms) : 0, 1452
crashtracking [candidate] (1.457 ms) : 0, 1457
BytebuddyAgent [baseline] (706.503 ms) : 0, 706503
BytebuddyAgent [candidate] (706.747 ms) : 0, 706747
GlobalTracer [baseline] (245.688 ms) : 0, 245688
GlobalTracer [candidate] (246.804 ms) : 0, 246804
AppSec [baseline] (32.349 ms) : 0, 32349
AppSec [candidate] (32.256 ms) : 0, 32256
Debugger [baseline] (6.403 ms) : 0, 6403
Debugger [candidate] (6.394 ms) : 0, 6394
Remote Config [baseline] (711.826 µs) : 0, 712
Remote Config [candidate] (709.23 µs) : 0, 709
Telemetry [baseline] (15.725 ms) : 0, 15725
Telemetry [candidate] (14.399 ms) : 0, 14399
Flare Poller [baseline] (5.02 ms) : 0, 5020
Flare Poller [candidate] (6.521 ms) : 0, 6521
section iast
crashtracking [baseline] (1.456 ms) : 0, 1456
crashtracking [candidate] (1.462 ms) : 0, 1462
BytebuddyAgent [baseline] (827.595 ms) : 0, 827595
BytebuddyAgent [candidate] (833.33 ms) : 0, 833330
GlobalTracer [baseline] (233.976 ms) : 0, 233976
GlobalTracer [candidate] (234.97 ms) : 0, 234970
IAST [baseline] (33.36 ms) : 0, 33360
IAST [candidate] (32.615 ms) : 0, 32615
AppSec [baseline] (27.822 ms) : 0, 27822
AppSec [candidate] (28.926 ms) : 0, 28926
Debugger [baseline] (5.98 ms) : 0, 5980
Debugger [candidate] (6.157 ms) : 0, 6157
Remote Config [baseline] (588.215 µs) : 0, 588
Remote Config [candidate] (598.135 µs) : 0, 598
Telemetry [baseline] (8.32 ms) : 0, 8320
Telemetry [candidate] (8.591 ms) : 0, 8591
Flare Poller [baseline] (4.113 ms) : 0, 4113
Flare Poller [candidate] (4.198 ms) : 0, 4198
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.56.0-SNAPSHOT~0bfaa064cd, baseline=1.56.0-SNAPSHOT~19d774d48c

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.045 s) : 0, 1045133
Total [baseline] (10.696 s) : 0, 10695737
Agent [candidate] (1.044 s) : 0, 1044268
Total [candidate] (10.749 s) : 0, 10749334
section appsec
Agent [baseline] (1.219 s) : 0, 1219486
Total [baseline] (10.858 s) : 0, 10858327
Agent [candidate] (1.237 s) : 0, 1237166
Total [candidate] (10.888 s) : 0, 10887965
section iast
Agent [baseline] (1.177 s) : 0, 1176842
Total [baseline] (11.123 s) : 0, 11122812
Agent [candidate] (1.187 s) : 0, 1187221
Total [candidate] (11.192 s) : 0, 11192478
section profiling
Agent [baseline] (1.192 s) : 0, 1191827
Total [baseline] (10.922 s) : 0, 10922131
Agent [candidate] (1.195 s) : 0, 1195450
Total [candidate] (10.987 s) : 0, 10986948
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.045 s -
Agent appsec 1.219 s 174.353 ms (16.7%)
Agent iast 1.177 s 131.709 ms (12.6%)
Agent profiling 1.192 s 146.694 ms (14.0%)
Total tracing 10.696 s -
Total appsec 10.858 s 162.589 ms (1.5%)
Total iast 11.123 s 427.074 ms (4.0%)
Total profiling 10.922 s 226.393 ms (2.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.044 s -
Agent appsec 1.237 s 192.898 ms (18.5%)
Agent iast 1.187 s 142.952 ms (13.7%)
Agent profiling 1.195 s 151.181 ms (14.5%)
Total tracing 10.749 s -
Total appsec 10.888 s 138.631 ms (1.3%)
Total iast 11.192 s 443.143 ms (4.1%)
Total profiling 10.987 s 237.614 ms (2.2%) 
gantt
    title petclinic - break down per module: candidate=1.56.0-SNAPSHOT~0bfaa064cd, baseline=1.56.0-SNAPSHOT~19d774d48c

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.458 ms) : 0, 1458
crashtracking [candidate] (1.457 ms) : 0, 1457
BytebuddyAgent [baseline] (703.535 ms) : 0, 703535
BytebuddyAgent [candidate] (702.361 ms) : 0, 702361
GlobalTracer [baseline] (245.136 ms) : 0, 245136
GlobalTracer [candidate] (245.739 ms) : 0, 245739
AppSec [baseline] (32.394 ms) : 0, 32394
AppSec [candidate] (32.284 ms) : 0, 32284
Debugger [baseline] (6.398 ms) : 0, 6398
Debugger [candidate] (6.366 ms) : 0, 6366
Remote Config [baseline] (713.721 µs) : 0, 714
Remote Config [candidate] (714.891 µs) : 0, 715
Telemetry [baseline] (14.303 ms) : 0, 14303
Telemetry [candidate] (15.015 ms) : 0, 15015
Flare Poller [baseline] (6.553 ms) : 0, 6553
Flare Poller [candidate] (5.809 ms) : 0, 5809
section appsec
crashtracking [baseline] (1.449 ms) : 0, 1449
crashtracking [candidate] (1.481 ms) : 0, 1481
BytebuddyAgent [baseline] (727.367 ms) : 0, 727367
BytebuddyAgent [candidate] (740.492 ms) : 0, 740492
GlobalTracer [baseline] (237.045 ms) : 0, 237045
GlobalTracer [candidate] (240.1 ms) : 0, 240100
IAST [baseline] (24.828 ms) : 0, 24828
IAST [candidate] (25.288 ms) : 0, 25288
AppSec [baseline] (174.797 ms) : 0, 174797
AppSec [candidate] (175.182 ms) : 0, 175182
Debugger [baseline] (5.999 ms) : 0, 5999
Debugger [candidate] (6.075 ms) : 0, 6075
Remote Config [baseline] (645.572 µs) : 0, 646
Remote Config [candidate] (669.15 µs) : 0, 669
Telemetry [baseline] (8.483 ms) : 0, 8483
Telemetry [candidate] (8.665 ms) : 0, 8665
Flare Poller [baseline] (3.989 ms) : 0, 3989
Flare Poller [candidate] (4.01 ms) : 0, 4010
section iast
crashtracking [baseline] (1.443 ms) : 0, 1443
crashtracking [candidate] (1.47 ms) : 0, 1470
BytebuddyAgent [baseline] (825.846 ms) : 0, 825846
BytebuddyAgent [candidate] (833.907 ms) : 0, 833907
GlobalTracer [baseline] (234.21 ms) : 0, 234210
GlobalTracer [candidate] (235.733 ms) : 0, 235733
IAST [baseline] (30.125 ms) : 0, 30125
IAST [candidate] (30.218 ms) : 0, 30218
AppSec [baseline] (31.224 ms) : 0, 31224
AppSec [candidate] (31.565 ms) : 0, 31565
Debugger [baseline] (5.994 ms) : 0, 5994
Debugger [candidate] (6.071 ms) : 0, 6071
Remote Config [baseline] (598.733 µs) : 0, 599
Remote Config [candidate] (603.966 µs) : 0, 604
Telemetry [baseline] (8.556 ms) : 0, 8556
Telemetry [candidate] (8.585 ms) : 0, 8585
Flare Poller [baseline] (4.19 ms) : 0, 4190
Flare Poller [candidate] (4.202 ms) : 0, 4202
section profiling
crashtracking [baseline] (1.462 ms) : 0, 1462
crashtracking [candidate] (1.459 ms) : 0, 1459
BytebuddyAgent [baseline] (728.58 ms) : 0, 728580
BytebuddyAgent [candidate] (731.628 ms) : 0, 731628
GlobalTracer [baseline] (221.354 ms) : 0, 221354
GlobalTracer [candidate] (222.308 ms) : 0, 222308
AppSec [baseline] (32.171 ms) : 0, 32171
AppSec [candidate] (32.191 ms) : 0, 32191
Debugger [baseline] (9.066 ms) : 0, 9066
Debugger [candidate] (10.625 ms) : 0, 10625
Remote Config [baseline] (691.999 µs) : 0, 692
Remote Config [candidate] (1.548 ms) : 0, 1548
Telemetry [baseline] (13.704 ms) : 0, 13704
Telemetry [candidate] (11.348 ms) : 0, 11348
Flare Poller [baseline] (4.125 ms) : 0, 4125
Flare Poller [candidate] (4.112 ms) : 0, 4112
ProfilingAgent [baseline] (110.992 ms) : 0, 110992
ProfilingAgent [candidate] (111.026 ms) : 0, 111026
Profiling [baseline] (111.665 ms) : 0, 111665
Profiling [candidate] (111.702 ms) : 0, 111702
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master charles.yu/djm-974/spark-plan-databricks
git_commit_date 1762435787 1762442890
git_commit_sha 19d774d 0bfaa06
release_version 1.56.0-SNAPSHOT~19d774d48c 1.56.0-SNAPSHOT~0bfaa064cd
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1762445353 1762445353
ci_job_id 1219866541 1219866541
ci_pipeline_id 81464370 81464370
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-4-bshu75to 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-4-bshu75to 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 2 performance improvements and 3 performance regressions! Performance is the same for 7 metrics, 12 unstable metrics.

scenario Δ mean http_req_duration Δ mean throughput candidate mean http_req_duration candidate mean throughput baseline mean http_req_duration baseline mean throughput
scenario:load:insecure-bank:tracing:high_load worse
[+52.195µs; +85.056µs] or [+2.925%; +4.766%]		 unstable
[-344.333op/s; +165.833op/s] or [-13.676%; +6.587%]		 1.853ms 2428.500op/s 1.784ms 2517.750op/s
scenario:load:insecure-bank:profiling:high_load worse
[+44.923µs; +82.683µs] or [+2.325%; +4.279%]		 unstable
[-331.021op/s; +188.709op/s] or [-14.197%; +8.093%]		 1.996ms 2260.500op/s 1.932ms 2331.656op/s
scenario:load:petclinic:iast:high_load worse
[+1.400ms; +1.800ms] or [+7.960%; +10.237%]		 unstable
[-46.222op/s; +2.097op/s] or [-17.424%; +0.790%]		 19.188ms 243.219op/s 17.587ms 265.281op/s
scenario:load:petclinic:profiling:high_load better
[-1.488ms; -1.066ms] or [-7.467%; -5.351%]		 unstable
[-7.249op/s; +39.124op/s] or [-3.093%; +16.693%]		 18.646ms 250.312op/s 19.923ms 234.375op/s
scenario:load:petclinic:no_agent:high_load better
[-1.394ms; -0.978ms] or [-7.224%; -5.064%]		 unstable
[-10.041op/s; +41.666op/s] or [-4.154%; +17.235%]		 18.118ms 257.562op/s 19.304ms 241.750op/s
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.56.0-SNAPSHOT~0bfaa064cd, baseline=1.56.0-SNAPSHOT~19d774d48c
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.209 ms) : 1196, 1221
.   : milestone, 1209,
iast (3.297 ms) : 3257, 3336
.   : milestone, 3297,
iast_FULL (5.912 ms) : 5851, 5972
.   : milestone, 5912,
iast_GLOBAL (3.687 ms) : 3633, 3741
.   : milestone, 3687,
profiling (1.932 ms) : 1916, 1949
.   : milestone, 1932,
tracing (1.784 ms) : 1770, 1799
.   : milestone, 1784,
section candidate
no_agent (1.229 ms) : 1217, 1241
.   : milestone, 1229,
iast (3.215 ms) : 3174, 3257
.   : milestone, 3215,
iast_FULL (5.791 ms) : 5734, 5847
.   : milestone, 5791,
iast_GLOBAL (3.565 ms) : 3505, 3625
.   : milestone, 3565,
profiling (1.996 ms) : 1978, 2014
.   : milestone, 1996,
tracing (1.853 ms) : 1837, 1869
.   : milestone, 1853,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.209 ms [1.196 ms, 1.221 ms] -
iast 3.297 ms [3.257 ms, 3.336 ms] 2.088 ms (172.8%)
iast_FULL 5.912 ms [5.851 ms, 5.972 ms] 4.703 ms (389.2%)
iast_GLOBAL 3.687 ms [3.633 ms, 3.741 ms] 2.479 ms (205.1%)
profiling 1.932 ms [1.916 ms, 1.949 ms] 723.844 µs (59.9%)
tracing 1.784 ms [1.77 ms, 1.799 ms] 575.948 µs (47.7%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.229 ms [1.217 ms, 1.241 ms] -
iast 3.215 ms [3.174 ms, 3.257 ms] 1.986 ms (161.6%)
iast_FULL 5.791 ms [5.734 ms, 5.847 ms] 4.561 ms (371.1%)
iast_GLOBAL 3.565 ms [3.505 ms, 3.625 ms] 2.336 ms (190.0%)
profiling 1.996 ms [1.978 ms, 2.014 ms] 767.027 µs (62.4%)
tracing 1.853 ms [1.837 ms, 1.869 ms] 623.954 µs (50.8%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.56.0-SNAPSHOT~0bfaa064cd, baseline=1.56.0-SNAPSHOT~19d774d48c
    dateFormat X
    axisFormat %s
section baseline
no_agent (19.304 ms) : 19106, 19503
.   : milestone, 19304,
appsec (18.418 ms) : 18229, 18607
.   : milestone, 18418,
code_origins (17.891 ms) : 17713, 18069
.   : milestone, 17891,
iast (17.587 ms) : 17414, 17761
.   : milestone, 17587,
profiling (19.923 ms) : 19718, 20127
.   : milestone, 19923,
tracing (18.343 ms) : 18163, 18522
.   : milestone, 18343,
section candidate
no_agent (18.118 ms) : 17930, 18307
.   : milestone, 18118,
appsec (18.767 ms) : 18578, 18956
.   : milestone, 18767,
code_origins (17.86 ms) : 17681, 18039
.   : milestone, 17860,
iast (19.188 ms) : 18990, 19385
.   : milestone, 19188,
profiling (18.646 ms) : 18459, 18833
.   : milestone, 18646,
tracing (18.6 ms) : 18411, 18789
.   : milestone, 18600,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 19.304 ms [19.106 ms, 19.503 ms] -
appsec 18.418 ms [18.229 ms, 18.607 ms] -885.964 µs (-4.6%)
code_origins 17.891 ms [17.713 ms, 18.069 ms] -1.413 ms (-7.3%)
iast 17.587 ms [17.414 ms, 17.761 ms] -1.717 ms (-8.9%)
profiling 19.923 ms [19.718 ms, 20.127 ms] 618.35 µs (3.2%)
tracing 18.343 ms [18.163 ms, 18.522 ms] -961.737 µs (-5.0%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 18.118 ms [17.93 ms, 18.307 ms] -
appsec 18.767 ms [18.578 ms, 18.956 ms] 648.549 µs (3.6%)
code_origins 17.86 ms [17.681 ms, 18.039 ms] -258.502 µs (-1.4%)
iast 19.188 ms [18.99 ms, 19.385 ms] 1.069 ms (5.9%)
profiling 18.646 ms [18.459 ms, 18.833 ms] 527.484 µs (2.9%)
tracing 18.6 ms [18.411 ms, 18.789 ms] 481.615 µs (2.7%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master charles.yu/djm-974/spark-plan-databricks
git_commit_date 1762435787 1762442890
git_commit_sha 19d774d 0bfaa06
release_version 1.56.0-SNAPSHOT~19d774d48c 1.56.0-SNAPSHOT~0bfaa064cd
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1762445055 1762445055
ci_job_id 1219866542 1219866542
ci_pipeline_id 81464370 81464370
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-4-11h7kx66 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-4-11h7kx66 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 2 unstable metrics.

Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.56.0-SNAPSHOT~0bfaa064cd, baseline=1.56.0-SNAPSHOT~19d774d48c
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.03 s) : 15030000, 15030000
.   : milestone, 15030000,
appsec (14.559 s) : 14559000, 14559000
.   : milestone, 14559000,
iast (18.458 s) : 18458000, 18458000
.   : milestone, 18458000,
iast_GLOBAL (17.931 s) : 17931000, 17931000
.   : milestone, 17931000,
profiling (14.98 s) : 14980000, 14980000
.   : milestone, 14980000,
tracing (14.728 s) : 14728000, 14728000
.   : milestone, 14728000,
section candidate
no_agent (15.727 s) : 15727000, 15727000
.   : milestone, 15727000,
appsec (15.202 s) : 15202000, 15202000
.   : milestone, 15202000,
iast (18.336 s) : 18336000, 18336000
.   : milestone, 18336000,
iast_GLOBAL (17.992 s) : 17992000, 17992000
.   : milestone, 17992000,
profiling (14.983 s) : 14983000, 14983000
.   : milestone, 14983000,
tracing (14.873 s) : 14873000, 14873000
.   : milestone, 14873000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.03 s [15.03 s, 15.03 s] -
appsec 14.559 s [14.559 s, 14.559 s] -471.0 ms (-3.1%)
iast 18.458 s [18.458 s, 18.458 s] 3.428 s (22.8%)
iast_GLOBAL 17.931 s [17.931 s, 17.931 s] 2.901 s (19.3%)
profiling 14.98 s [14.98 s, 14.98 s] -50.0 ms (-0.3%)
tracing 14.728 s [14.728 s, 14.728 s] -302.0 ms (-2.0%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.727 s [15.727 s, 15.727 s] -
appsec 15.202 s [15.202 s, 15.202 s] -525.0 ms (-3.3%)
iast 18.336 s [18.336 s, 18.336 s] 2.609 s (16.6%)
iast_GLOBAL 17.992 s [17.992 s, 17.992 s] 2.265 s (14.4%)
profiling 14.983 s [14.983 s, 14.983 s] -744.0 ms (-4.7%)
tracing 14.873 s [14.873 s, 14.873 s] -854.0 ms (-5.4%)
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.56.0-SNAPSHOT~0bfaa064cd, baseline=1.56.0-SNAPSHOT~19d774d48c
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.471 ms) : 1459, 1482
.   : milestone, 1471,
appsec (3.64 ms) : 3425, 3855
.   : milestone, 3640,
iast (2.203 ms) : 2139, 2267
.   : milestone, 2203,
iast_GLOBAL (2.253 ms) : 2188, 2317
.   : milestone, 2253,
profiling (2.475 ms) : 2252, 2697
.   : milestone, 2475,
tracing (2.024 ms) : 1974, 2073
.   : milestone, 2024,
section candidate
no_agent (1.472 ms) : 1460, 1483
.   : milestone, 1472,
appsec (3.676 ms) : 3458, 3895
.   : milestone, 3676,
iast (2.205 ms) : 2141, 2269
.   : milestone, 2205,
iast_GLOBAL (2.245 ms) : 2181, 2310
.   : milestone, 2245,
profiling (2.062 ms) : 2010, 2115
.   : milestone, 2062,
tracing (2.014 ms) : 1964, 2063
.   : milestone, 2014,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.471 ms [1.459 ms, 1.482 ms] -
appsec 3.64 ms [3.425 ms, 3.855 ms] 2.169 ms (147.5%)
iast 2.203 ms [2.139 ms, 2.267 ms] 732.083 µs (49.8%)
iast_GLOBAL 2.253 ms [2.188 ms, 2.317 ms] 782.128 µs (53.2%)
profiling 2.475 ms [2.252 ms, 2.697 ms] 1.004 ms (68.3%)
tracing 2.024 ms [1.974 ms, 2.073 ms] 552.963 µs (37.6%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.472 ms [1.46 ms, 1.483 ms] -
appsec 3.676 ms [3.458 ms, 3.895 ms] 2.204 ms (149.8%)
iast 2.205 ms [2.141 ms, 2.269 ms] 733.231 µs (49.8%)
iast_GLOBAL 2.245 ms [2.181 ms, 2.31 ms] 773.404 µs (52.5%)
profiling 2.062 ms [2.01 ms, 2.115 ms] 590.487 µs (40.1%)
tracing 2.014 ms [1.964 ms, 2.063 ms] 541.84 µs (36.8%)

@charlesmyu charlesmyu force-pushed the charles.yu/djm-974/spark-plan-databricks branch 2 times, most recently from 06f8fee to f649787 Compare October 30, 2025 22:09
@charlesmyu charlesmyu added the inst: apache spark Apache Spark instrumentation label Oct 30, 2025
@charlesmyu charlesmyu force-pushed the charles.yu/djm-974/spark-plan-databricks branch from f649787 to 03b891a Compare October 30, 2025 22:33
Base automatically changed from charles.yu/djm-974/reflect-spi-constructor to adrien.boitreaud/fix-muzzle-mismatch October 31, 2025 08:53
pawel-big-lebowski
pawel-big-lebowski reviewed Oct 31, 2025
View reviewed changes
...trumentation/spark/src/main/java/datadog/trace/instrumentation/spark/SparkPlanInfoUtils.java Outdated
SparkPlanInfo.class,
String.class,
String.class,
scala.collection.immutable.Seq.class,
Copy link
Contributor

@pawel-big-lebowski pawel-big-lebowski Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One of the most painful discrepancies between Scala 2.12 and 2.13 is in scala.Seq. And these bugs are hard to tackle.

I think it makes sense having two separate Spark212PlanInfoUtils2 and Spark213PlanInfoUtils classes, at least for the constructors and databricksConstructor part. This assures code is compiled with the same Scala version as it's run.

Copy link
Contributor Author

@charlesmyu charlesmyu Oct 31, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this makes sense - I was worried initially it was Spark version incompatibilities but if it's Scala there isn't much reason to not have two separate files. I ended up putting them in completely separate files with no common abstraction since I couldn't find a clean way of separating them for now, but I imagine down the line when we need to add more constructors that may become more appealing. Let me know what you think!

e82dd20 (#9888)

Edit: ended up adding an abstract class after some more testing. Let me know your thoughts!

0bfaa06 (#9888)

@charlesmyu charlesmyu force-pushed the charles.yu/djm-974/spark-plan-databricks branch from 03b891a to 078c8db Compare October 31, 2025 13:49
Base automatically changed from adrien.boitreaud/fix-muzzle-mismatch to master October 31, 2025 14:35
@charlesmyu charlesmyu force-pushed the charles.yu/djm-974/spark-plan-databricks branch 7 times, most recently from 61f7bdb to fce0d86 Compare November 5, 2025 20:07
@charlesmyu charlesmyu force-pushed the charles.yu/djm-974/spark-plan-databricks branch from fce0d86 to e82dd20 Compare November 5, 2025 20:39
@charlesmyu charlesmyu force-pushed the charles.yu/djm-974/spark-plan-databricks branch from 8cff397 to 0bfaa06 Compare November 6, 2025 15:28
@charlesmyu charlesmyu marked this pull request as ready for review November 6, 2025 16:08
@charlesmyu charlesmyu requested a review from a team as a code owner November 6, 2025 16:08
@github-actions
Copy link
Contributor

github-actions bot commented Nov 6, 2025

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

  • Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

@charlesmyu charlesmyu added the type: enhancement Enhancements and improvements label Nov 6, 2025
@charlesmyu charlesmyu merged commit 5743cd5 into master Nov 6, 2025
544 of 545 checks passed
@charlesmyu charlesmyu deleted the charles.yu/djm-974/spark-plan-databricks branch November 6, 2025 19:43
@github-actions github-actions bot added this to the 1.56.0 milestone Nov 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pawel-big-lebowski pawel-big-lebowski pawel-big-lebowski left review comments

@mobuchowski mobuchowski mobuchowski approved these changes

Assignees

No one assigned

Labels

inst: apache spark Apache Spark instrumentation type: enhancement Enhancements and improvements

Projects

None yet

Milestone

1.56.0

Development

Successfully merging this pull request may close these issues.

4 participants

@charlesmyu @pawel-big-lebowski @mobuchowski