Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Finish pyspark-shell trace using SparkListenerApplicationEnd #6956

Merged
merged 2 commits into from
Apr 30, 2024
Merged

Finish pyspark-shell trace using SparkListenerApplicationEnd #6956

merged 2 commits into from
Apr 30, 2024

Conversation

paul-laffon-dd
Copy link
Contributor

@paul-laffon-dd paul-laffon-dd commented Apr 26, 2024 •
edited
Loading

What Does This Do

Finish pyspark-shell application span when the spark listener is receiving the SparkListenerApplicationEnd event

The application span is currently finished when exiting the runMain method to capture JVM errors (from this PR), however this function is not exited as expected (System.exit(0)) when executing as pyspark-shell

Motivation

Pyspark application can be launched using multiple method, mainly:

  • spark-submit script.py
  • python script.py

The spark instrumentation is working as expected in the spark-submit case, because this is java/scala code launched behind the scene

However, the python case, the listener is being injected as expected, but the spark.application span is not being closed properly because the JVM is launched as PythonGatewayServer which is exited using System.exit(0), leading to the exit advice not being called

Additional Notes

@pr-commenter
Copy link

pr-commenter bot commented Apr 26, 2024 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master paul.laffon/finish-pyspark-spans
git_commit_date 1714482408 1714483538
git_commit_sha fcfaf54 afe92c9
release_version 1.34.0-SNAPSHOT~fcfaf54d06 1.34.0-SNAPSHOT~afe92c9006
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1714486331 1714486331
ci_job_id 500797962 500797962
ci_pipeline_id 33249794 33249794
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 49 metrics, 14 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.34.0-SNAPSHOT~afe92c9006, baseline=1.34.0-SNAPSHOT~fcfaf54d06

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.077 s) : 0, 1076902
Total [baseline] (8.543 s) : 0, 8543320
Agent [candidate] (1.084 s) : 0, 1083984
Total [candidate] (8.547 s) : 0, 8546777
section iast
Agent [baseline] (1.204 s) : 0, 1203580
Total [baseline] (9.024 s) : 0, 9024460
Agent [candidate] (1.207 s) : 0, 1207268
Total [candidate] (9.004 s) : 0, 9003712
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.201 s) : 0, 1200727
Total [baseline] (8.954 s) : 0, 8954326
Agent [candidate] (1.202 s) : 0, 1201677
Total [candidate] (8.984 s) : 0, 8983804
section iast_TELEMETRY_OFF
Agent [baseline] (1.198 s) : 0, 1198388
Total [baseline] (9.001 s) : 0, 9001421
Agent [candidate] (1.195 s) : 0, 1195336
Total [candidate] (9.003 s) : 0, 9002689
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.077 s -
Agent iast 1.204 s 126.677 ms (11.8%)
Agent iast_HARDCODED_SECRET_DISABLED 1.201 s 123.825 ms (11.5%)
Agent iast_TELEMETRY_OFF 1.198 s 121.486 ms (11.3%)
Total tracing 8.543 s -
Total iast 9.024 s 481.14 ms (5.6%)
Total iast_HARDCODED_SECRET_DISABLED 8.954 s 411.006 ms (4.8%)
Total iast_TELEMETRY_OFF 9.001 s 458.101 ms (5.4%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.084 s -
Agent iast 1.207 s 123.284 ms (11.4%)
Agent iast_HARDCODED_SECRET_DISABLED 1.202 s 117.693 ms (10.9%)
Agent iast_TELEMETRY_OFF 1.195 s 111.352 ms (10.3%)
Total tracing 8.547 s -
Total iast 9.004 s 456.935 ms (5.3%)
Total iast_HARDCODED_SECRET_DISABLED 8.984 s 437.027 ms (5.1%)
Total iast_TELEMETRY_OFF 9.003 s 455.911 ms (5.3%) 
gantt
    title insecure-bank - break down per module: candidate=1.34.0-SNAPSHOT~afe92c9006, baseline=1.34.0-SNAPSHOT~fcfaf54d06

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (675.476 ms) : 0, 675476
BytebuddyAgent [candidate] (679.001 ms) : 0, 679001
GlobalTracer [baseline] (309.395 ms) : 0, 309395
GlobalTracer [candidate] (312.438 ms) : 0, 312438
AppSec [baseline] (49.437 ms) : 0, 49437
AppSec [candidate] (49.618 ms) : 0, 49618
Remote Config [baseline] (653.462 µs) : 0, 653
Remote Config [candidate] (671.467 µs) : 0, 671
Telemetry [baseline] (7.531 ms) : 0, 7531
Telemetry [candidate] (7.595 ms) : 0, 7595
section iast
BytebuddyAgent [baseline] (798.807 ms) : 0, 798807
BytebuddyAgent [candidate] (799.654 ms) : 0, 799654
GlobalTracer [baseline] (287.728 ms) : 0, 287728
GlobalTracer [candidate] (290.29 ms) : 0, 290290
AppSec [baseline] (48.021 ms) : 0, 48021
AppSec [candidate] (50.561 ms) : 0, 50561
Remote Config [baseline] (1.328 ms) : 0, 1328
Remote Config [candidate] (1.366 ms) : 0, 1366
Telemetry [baseline] (6.65 ms) : 0, 6650
Telemetry [candidate] (6.681 ms) : 0, 6681
IAST [baseline] (26.566 ms) : 0, 26566
IAST [candidate] (24.03 ms) : 0, 24030
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (794.721 ms) : 0, 794721
BytebuddyAgent [candidate] (796.238 ms) : 0, 796238
GlobalTracer [baseline] (288.838 ms) : 0, 288838
GlobalTracer [candidate] (288.711 ms) : 0, 288711
AppSec [baseline] (51.586 ms) : 0, 51586
AppSec [candidate] (51.701 ms) : 0, 51701
Remote Config [baseline] (565.681 µs) : 0, 566
Remote Config [candidate] (579.617 µs) : 0, 580
Telemetry [baseline] (7.346 ms) : 0, 7346
Telemetry [candidate] (7.438 ms) : 0, 7438
IAST [baseline] (23.361 ms) : 0, 23361
IAST [candidate] (22.536 ms) : 0, 22536
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (793.221 ms) : 0, 793221
BytebuddyAgent [candidate] (791.03 ms) : 0, 791030
GlobalTracer [baseline] (288.537 ms) : 0, 288537
GlobalTracer [candidate] (287.912 ms) : 0, 287912
AppSec [baseline] (49.123 ms) : 0, 49123
AppSec [candidate] (48.157 ms) : 0, 48157
Remote Config [baseline] (575.316 µs) : 0, 575
Remote Config [candidate] (576.466 µs) : 0, 576
Telemetry [baseline] (8.049 ms) : 0, 8049
Telemetry [candidate] (7.274 ms) : 0, 7274
IAST [baseline] (24.505 ms) : 0, 24505
IAST [candidate] (26.073 ms) : 0, 26073
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.34.0-SNAPSHOT~afe92c9006, baseline=1.34.0-SNAPSHOT~fcfaf54d06

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.078 s) : 0, 1078279
Total [baseline] (10.465 s) : 0, 10464968
Agent [candidate] (1.077 s) : 0, 1076645
Total [candidate] (10.38 s) : 0, 10379543
section appsec
Agent [baseline] (1.19 s) : 0, 1190003
Total [baseline] (10.447 s) : 0, 10447446
Agent [candidate] (1.188 s) : 0, 1188435
Total [candidate] (10.467 s) : 0, 10467015
section iast
Agent [baseline] (1.209 s) : 0, 1209362
Total [baseline] (10.74 s) : 0, 10740146
Agent [candidate] (1.201 s) : 0, 1200783
Total [candidate] (10.718 s) : 0, 10717661
section profiling
Agent [baseline] (1.27 s) : 0, 1270304
Total [baseline] (10.653 s) : 0, 10653389
Agent [candidate] (1.276 s) : 0, 1276018
Total [candidate] (10.701 s) : 0, 10701113
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.078 s -
Agent appsec 1.19 s 111.725 ms (10.4%)
Agent iast 1.209 s 131.083 ms (12.2%)
Agent profiling 1.27 s 192.026 ms (17.8%)
Total tracing 10.465 s -
Total appsec 10.447 s -17.523 ms (-0.2%)
Total iast 10.74 s 275.178 ms (2.6%)
Total profiling 10.653 s 188.421 ms (1.8%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.077 s -
Agent appsec 1.188 s 111.79 ms (10.4%)
Agent iast 1.201 s 124.138 ms (11.5%)
Agent profiling 1.276 s 199.374 ms (18.5%)
Total tracing 10.38 s -
Total appsec 10.467 s 87.472 ms (0.8%)
Total iast 10.718 s 338.118 ms (3.3%)
Total profiling 10.701 s 321.569 ms (3.1%) 
gantt
    title petclinic - break down per module: candidate=1.34.0-SNAPSHOT~afe92c9006, baseline=1.34.0-SNAPSHOT~fcfaf54d06

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (675.894 ms) : 0, 675894
BytebuddyAgent [candidate] (673.652 ms) : 0, 673652
GlobalTracer [baseline] (310.109 ms) : 0, 310109
GlobalTracer [candidate] (310.59 ms) : 0, 310590
AppSec [baseline] (49.511 ms) : 0, 49511
AppSec [candidate] (49.741 ms) : 0, 49741
Remote Config [baseline] (666.724 µs) : 0, 667
Remote Config [candidate] (703.652 µs) : 0, 704
Telemetry [baseline] (7.669 ms) : 0, 7669
Telemetry [candidate] (7.632 ms) : 0, 7632
section appsec
BytebuddyAgent [baseline] (696.13 ms) : 0, 696130
BytebuddyAgent [candidate] (694.994 ms) : 0, 694994
GlobalTracer [baseline] (291.876 ms) : 0, 291876
GlobalTracer [candidate] (291.053 ms) : 0, 291053
AppSec [baseline] (148.933 ms) : 0, 148933
AppSec [candidate] (149.321 ms) : 0, 149321
Remote Config [baseline] (620.673 µs) : 0, 621
Remote Config [candidate] (616.814 µs) : 0, 617
Telemetry [baseline] (8.482 ms) : 0, 8482
Telemetry [candidate] (8.483 ms) : 0, 8483
IAST [baseline] (19.219 ms) : 0, 19219
IAST [candidate] (19.316 ms) : 0, 19316
section iast
BytebuddyAgent [baseline] (801.555 ms) : 0, 801555
BytebuddyAgent [candidate] (795.8 ms) : 0, 795800
GlobalTracer [baseline] (290.316 ms) : 0, 290316
GlobalTracer [candidate] (288.795 ms) : 0, 288795
AppSec [baseline] (50.698 ms) : 0, 50698
AppSec [candidate] (50.156 ms) : 0, 50156
Remote Config [baseline] (568.204 µs) : 0, 568
Remote Config [candidate] (576.815 µs) : 0, 577
Telemetry [baseline] (6.626 ms) : 0, 6626
Telemetry [candidate] (8.882 ms) : 0, 8882
IAST [baseline] (24.981 ms) : 0, 24981
IAST [candidate] (22.032 ms) : 0, 22032
section profiling
BytebuddyAgent [baseline] (678.339 ms) : 0, 678339
BytebuddyAgent [candidate] (681.283 ms) : 0, 681283
GlobalTracer [baseline] (380.568 ms) : 0, 380568
GlobalTracer [candidate] (382.456 ms) : 0, 382456
AppSec [baseline] (50.27 ms) : 0, 50270
AppSec [candidate] (50.399 ms) : 0, 50399
Remote Config [baseline] (709.913 µs) : 0, 710
Remote Config [candidate] (704.162 µs) : 0, 704
Telemetry [baseline] (7.471 ms) : 0, 7471
Telemetry [candidate] (7.535 ms) : 0, 7535
ProfilingAgent [baseline] (96.571 ms) : 0, 96571
ProfilingAgent [candidate] (96.83 ms) : 0, 96830
Profiling [baseline] (96.596 ms) : 0, 96596
Profiling [candidate] (96.854 ms) : 0, 96854
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-04-30T13:45:13 2024-04-30T13:52:02
git_branch master paul.laffon/finish-pyspark-spans
git_commit_date 1714482408 1714483538
git_commit_sha fcfaf54 afe92c9
release_version 1.34.0-SNAPSHOT~fcfaf54d06 1.34.0-SNAPSHOT~afe92c9006
start_time 2024-04-30T13:44:59 2024-04-30T13:51:49
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1714485471 1714485471
ci_job_id 500797963 500797963
ci_pipeline_id 33249794 33249794
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 1 performance regressions! Performance is the same for 12 metrics, 15 unstable metrics.

scenario Δ mean http_req_duration Δ mean throughput candidate mean http_req_duration candidate mean throughput baseline mean http_req_duration baseline mean throughput
scenario:load:petclinic:profiling worse
[+70.615µs; +125.253µs] or [+4.747%; +8.420%]		 unstable
[-764.251op/s; +290.177op/s] or [-23.883%; +9.068%]		 1.586ms 2962.963op/s 1.488ms 3200.000op/s
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.34.0-SNAPSHOT~afe92c9006, baseline=1.34.0-SNAPSHOT~fcfaf54d06
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.374 ms) : 1354, 1394
.   : milestone, 1374,
appsec (1.731 ms) : 1708, 1755
.   : milestone, 1731,
appsec_no_iast (1.714 ms) : 1689, 1738
.   : milestone, 1714,
iast (1.513 ms) : 1490, 1536
.   : milestone, 1513,
profiling (1.488 ms) : 1463, 1512
.   : milestone, 1488,
tracing (1.487 ms) : 1463, 1511
.   : milestone, 1487,
section candidate
no_agent (1.363 ms) : 1344, 1382
.   : milestone, 1363,
appsec (1.729 ms) : 1704, 1754
.   : milestone, 1729,
appsec_no_iast (1.744 ms) : 1720, 1767
.   : milestone, 1744,
iast (1.504 ms) : 1481, 1528
.   : milestone, 1504,
profiling (1.586 ms) : 1560, 1612
.   : milestone, 1586,
tracing (1.469 ms) : 1444, 1494
.   : milestone, 1469,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.374 ms [1.354 ms, 1.394 ms] -
appsec 1.731 ms [1.708 ms, 1.755 ms] 357.684 µs (26.0%)
appsec_no_iast 1.714 ms [1.689 ms, 1.738 ms] 340.246 µs (24.8%)
iast 1.513 ms [1.49 ms, 1.536 ms] 139.627 µs (10.2%)
profiling 1.488 ms [1.463 ms, 1.512 ms] 113.88 µs (8.3%)
tracing 1.487 ms [1.463 ms, 1.511 ms] 113.413 µs (8.3%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.363 ms [1.344 ms, 1.382 ms] -
appsec 1.729 ms [1.704 ms, 1.754 ms] 365.755 µs (26.8%)
appsec_no_iast 1.744 ms [1.72 ms, 1.767 ms] 380.457 µs (27.9%)
iast 1.504 ms [1.481 ms, 1.528 ms] 141.209 µs (10.4%)
profiling 1.586 ms [1.56 ms, 1.612 ms] 222.31 µs (16.3%)
tracing 1.469 ms [1.444 ms, 1.494 ms] 105.606 µs (7.7%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.34.0-SNAPSHOT~afe92c9006, baseline=1.34.0-SNAPSHOT~fcfaf54d06
    dateFormat X
    axisFormat %s
section baseline
no_agent (376.439 µs) : 356, 396
.   : milestone, 376,
iast (484.746 µs) : 464, 505
.   : milestone, 485,
iast_FULL (540.293 µs) : 520, 561
.   : milestone, 540,
iast_GLOBAL (503.497 µs) : 482, 525
.   : milestone, 503,
iast_HARDCODED_SECRET_DISABLED (477.952 µs) : 456, 500
.   : milestone, 478,
iast_INACTIVE (453.27 µs) : 433, 474
.   : milestone, 453,
iast_TELEMETRY_OFF (481.162 µs) : 460, 503
.   : milestone, 481,
tracing (444.16 µs) : 424, 464
.   : milestone, 444,
section candidate
no_agent (370.132 µs) : 350, 390
.   : milestone, 370,
iast (474.626 µs) : 454, 496
.   : milestone, 475,
iast_FULL (543.846 µs) : 523, 564
.   : milestone, 544,
iast_GLOBAL (510.402 µs) : 488, 533
.   : milestone, 510,
iast_HARDCODED_SECRET_DISABLED (478.913 µs) : 458, 500
.   : milestone, 479,
iast_INACTIVE (459.479 µs) : 439, 480
.   : milestone, 459,
iast_TELEMETRY_OFF (475.277 µs) : 455, 496
.   : milestone, 475,
tracing (453.289 µs) : 432, 474
.   : milestone, 453,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 376.439 µs [356.467 µs, 396.412 µs] -
iast 484.746 µs [464.086 µs, 505.406 µs] 108.306 µs (28.8%)
iast_FULL 540.293 µs [519.827 µs, 560.759 µs] 163.853 µs (43.5%)
iast_GLOBAL 503.497 µs [482.262 µs, 524.733 µs] 127.058 µs (33.8%)
iast_HARDCODED_SECRET_DISABLED 477.952 µs [456.175 µs, 499.728 µs] 101.512 µs (27.0%)
iast_INACTIVE 453.27 µs [432.626 µs, 473.913 µs] 76.83 µs (20.4%)
iast_TELEMETRY_OFF 481.162 µs [459.59 µs, 502.734 µs] 104.722 µs (27.8%)
tracing 444.16 µs [423.97 µs, 464.349 µs] 67.72 µs (18.0%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 370.132 µs [350.323 µs, 389.94 µs] -
iast 474.626 µs [453.514 µs, 495.739 µs] 104.495 µs (28.2%)
iast_FULL 543.846 µs [523.218 µs, 564.474 µs] 173.715 µs (46.9%)
iast_GLOBAL 510.402 µs [488.227 µs, 532.577 µs] 140.271 µs (37.9%)
iast_HARDCODED_SECRET_DISABLED 478.913 µs [457.912 µs, 499.914 µs] 108.781 µs (29.4%)
iast_INACTIVE 459.479 µs [438.644 µs, 480.314 µs] 89.347 µs (24.1%)
iast_TELEMETRY_OFF 475.277 µs [454.604 µs, 495.949 µs] 105.145 µs (28.4%)
tracing 453.289 µs [432.337 µs, 474.241 µs] 83.158 µs (22.5%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master paul.laffon/finish-pyspark-spans
git_commit_date 1714482408 1714483538
git_commit_sha fcfaf54 afe92c9
release_version 1.34.0-SNAPSHOT~fcfaf54d06 1.34.0-SNAPSHOT~afe92c9006
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1714486015 1714486015
ci_job_id 500797964 500797964
ci_pipeline_id 33249794 33249794
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.34.0-SNAPSHOT~afe92c9006, baseline=1.34.0-SNAPSHOT~fcfaf54d06
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.468 ms) : 1457, 1480
.   : milestone, 1468,
appsec (2.216 ms) : 2182, 2251
.   : milestone, 2216,
iast (1.889 ms) : 1854, 1924
.   : milestone, 1889,
iast_GLOBAL (1.92 ms) : 1885, 1956
.   : milestone, 1920,
profiling (1.853 ms) : 1819, 1886
.   : milestone, 1853,
tracing (1.835 ms) : 1803, 1867
.   : milestone, 1835,
section candidate
no_agent (1.468 ms) : 1457, 1480
.   : milestone, 1468,
appsec (2.2 ms) : 2166, 2233
.   : milestone, 2200,
iast (1.888 ms) : 1852, 1923
.   : milestone, 1888,
iast_GLOBAL (1.917 ms) : 1882, 1951
.   : milestone, 1917,
profiling (1.85 ms) : 1817, 1883
.   : milestone, 1850,
tracing (1.832 ms) : 1801, 1864
.   : milestone, 1832,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.468 ms [1.457 ms, 1.48 ms] -
appsec 2.216 ms [2.182 ms, 2.251 ms] 747.887 µs (50.9%)
iast 1.889 ms [1.854 ms, 1.924 ms] 420.813 µs (28.7%)
iast_GLOBAL 1.92 ms [1.885 ms, 1.956 ms] 452.047 µs (30.8%)
profiling 1.853 ms [1.819 ms, 1.886 ms] 384.583 µs (26.2%)
tracing 1.835 ms [1.803 ms, 1.867 ms] 366.514 µs (25.0%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.468 ms [1.457 ms, 1.48 ms] -
appsec 2.2 ms [2.166 ms, 2.233 ms] 731.286 µs (49.8%)
iast 1.888 ms [1.852 ms, 1.923 ms] 419.298 µs (28.6%)
iast_GLOBAL 1.917 ms [1.882 ms, 1.951 ms] 448.491 µs (30.5%)
profiling 1.85 ms [1.817 ms, 1.883 ms] 381.665 µs (26.0%)
tracing 1.832 ms [1.801 ms, 1.864 ms] 364.221 µs (24.8%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.34.0-SNAPSHOT~afe92c9006, baseline=1.34.0-SNAPSHOT~fcfaf54d06
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.363 s) : 15363000, 15363000
.   : milestone, 15363000,
appsec (14.975 s) : 14975000, 14975000
.   : milestone, 14975000,
iast (18.668 s) : 18668000, 18668000
.   : milestone, 18668000,
iast_GLOBAL (17.903 s) : 17903000, 17903000
.   : milestone, 17903000,
profiling (15.126 s) : 15126000, 15126000
.   : milestone, 15126000,
tracing (15.047 s) : 15047000, 15047000
.   : milestone, 15047000,
section candidate
no_agent (14.845 s) : 14845000, 14845000
.   : milestone, 14845000,
appsec (15.195 s) : 15195000, 15195000
.   : milestone, 15195000,
iast (18.91 s) : 18910000, 18910000
.   : milestone, 18910000,
iast_GLOBAL (17.95 s) : 17950000, 17950000
.   : milestone, 17950000,
profiling (15.408 s) : 15408000, 15408000
.   : milestone, 15408000,
tracing (15.417 s) : 15417000, 15417000
.   : milestone, 15417000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.363 s [15.363 s, 15.363 s] -
appsec 14.975 s [14.975 s, 14.975 s] -388.0 ms (-2.5%)
iast 18.668 s [18.668 s, 18.668 s] 3.305 s (21.5%)
iast_GLOBAL 17.903 s [17.903 s, 17.903 s] 2.54 s (16.5%)
profiling 15.126 s [15.126 s, 15.126 s] -237.0 ms (-1.5%)
tracing 15.047 s [15.047 s, 15.047 s] -316.0 ms (-2.1%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.845 s [14.845 s, 14.845 s] -
appsec 15.195 s [15.195 s, 15.195 s] 350.0 ms (2.4%)
iast 18.91 s [18.91 s, 18.91 s] 4.065 s (27.4%)
iast_GLOBAL 17.95 s [17.95 s, 17.95 s] 3.105 s (20.9%)
profiling 15.408 s [15.408 s, 15.408 s] 563.0 ms (3.8%)
tracing 15.417 s [15.417 s, 15.417 s] 572.0 ms (3.9%)

@paul-laffon-dd paul-laffon-dd force-pushed the paul.laffon/finish-pyspark-spans branch from 00d352d to 34c252a Compare April 29, 2024 09:24
@paul-laffon-dd paul-laffon-dd added the inst: apache spark Apache Spark instrumentation label Apr 29, 2024
@paul-laffon-dd paul-laffon-dd marked this pull request as ready for review April 29, 2024 12:59
@paul-laffon-dd paul-laffon-dd requested a review from a team as a code owner April 29, 2024 12:59
amarziali
amarziali reviewed Apr 30, 2024
View reviewed changes
...on/spark/src/main/java/datadog/trace/instrumentation/spark/AbstractSparkInstrumentation.java Outdated
import static net.bytebuddy.matcher.ElementMatchers.isDeclaredBy;
import static net.bytebuddy.matcher.ElementMatchers.isMethod;
import static net.bytebuddy.matcher.ElementMatchers.nameEndsWith;
import static net.bytebuddy.matcher.ElementMatchers.*;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The star imports are discouraged. Could you please reconfigure the IDE according to the contributing doc and format again?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, thanks, fixed the imports

amarziali
amarziali reviewed Apr 30, 2024
View reviewed changes
...on/spark/src/main/java/datadog/trace/instrumentation/spark/AbstractSparkInstrumentation.java
public static class RunMainAdvice {
@Advice.OnMethodEnter(suppress = Throwable.class)
public static void enter() {
AbstractDatadogSparkListener.finishTraceOnApplicationEnd = false;
if (!AbstractDatadogSparkListener.isPysparkShell) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is probably not needed since it's false by default

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is false by default, but if runMain is called after prepareSubmitEnvironment, we still want to have when using pyspark-shell finishTraceOnApplicationEnd=true

amarziali
amarziali approved these changes Apr 30, 2024
View reviewed changes
Copy link
Collaborator

@amarziali amarziali left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I left a couple of comments. I strongly encourage to apply the right formatting before merging

@paul-laffon-dd paul-laffon-dd merged commit f015cfc into master Apr 30, 2024
80 checks passed
@paul-laffon-dd paul-laffon-dd deleted the paul.laffon/finish-pyspark-spans branch April 30, 2024 17:09
@github-actions github-actions bot added this to the 1.34.0 milestone Apr 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tabgok tabgok tabgok approved these changes

@amarziali amarziali amarziali approved these changes

@yiliangzhou yiliangzhou Awaiting requested review from yiliangzhou

Assignees
No one assigned
Labels
inst: apache spark Apache Spark instrumentation
Projects
None yet
Milestone
1.34.0
Development

Successfully merging this pull request may close these issues.
3 participants
@paul-laffon-dd @amarziali @tabgok