Run all tests on Github Merge Queue #10578
Closed
+9
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
PerfectSlayer
left a comment
PerfectSlayer
left a comment
There was a problem hiding this comment.
❔ question: Do we have to change the check-pull-requests script?
GH merge queues won’t create PR, right? So there should be nothing to check 🤔
If this workflow is blocking the GH merge queues, let me know and I will try to tweak the related protection rule.
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 63 metrics, 8 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.60.0-SNAPSHOT~5041298e48, baseline=1.60.0-SNAPSHOT~2ebc964340
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.068 s) : 0, 1067679
Total [baseline] (10.955 s) : 0, 10954900
Agent [candidate] (1.065 s) : 0, 1064514
Total [candidate] (10.982 s) : 0, 10982111
section appsec
Agent [baseline] (1.241 s) : 0, 1241286
Total [baseline] (11.105 s) : 0, 11104765
Agent [candidate] (1.239 s) : 0, 1239396
Total [candidate] (11.059 s) : 0, 11059146
section iast
Agent [baseline] (1.253 s) : 0, 1252880
Total [baseline] (11.33 s) : 0, 11329566
Agent [candidate] (1.233 s) : 0, 1232698
Total [candidate] (11.242 s) : 0, 11242062
section profiling
Agent [baseline] (1.193 s) : 0, 1193073
Total [baseline] (11.087 s) : 0, 11087026
Agent [candidate] (1.199 s) : 0, 1198840
Total [candidate] (11.102 s) : 0, 11102251
gantt
title petclinic - break down per module: candidate=1.60.0-SNAPSHOT~5041298e48, baseline=1.60.0-SNAPSHOT~2ebc964340
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.188 ms) : 0, 1188
crashtracking [candidate] (1.182 ms) : 0, 1182
BytebuddyAgent [baseline] (629.053 ms) : 0, 629053
BytebuddyAgent [candidate] (629.255 ms) : 0, 629255
AgentMeter [baseline] (28.979 ms) : 0, 28979
AgentMeter [candidate] (28.979 ms) : 0, 28979
GlobalTracer [baseline] (257.834 ms) : 0, 257834
GlobalTracer [candidate] (257.659 ms) : 0, 257659
AppSec [baseline] (32.822 ms) : 0, 32822
AppSec [candidate] (32.882 ms) : 0, 32882
Debugger [baseline] (62.309 ms) : 0, 62309
Debugger [candidate] (62.636 ms) : 0, 62636
Remote Config [baseline] (631.407 µs) : 0, 631
Remote Config [candidate] (623.014 µs) : 0, 623
Telemetry [baseline] (12.45 ms) : 0, 12450
Telemetry [candidate] (11.415 ms) : 0, 11415
Flare Poller [baseline] (6.956 ms) : 0, 6956
Flare Poller [candidate] (4.537 ms) : 0, 4537
section appsec
crashtracking [baseline] (1.193 ms) : 0, 1193
crashtracking [candidate] (1.181 ms) : 0, 1181
BytebuddyAgent [baseline] (658.276 ms) : 0, 658276
BytebuddyAgent [candidate] (658.291 ms) : 0, 658291
AgentMeter [baseline] (11.996 ms) : 0, 11996
AgentMeter [candidate] (11.974 ms) : 0, 11974
GlobalTracer [baseline] (258.914 ms) : 0, 258914
GlobalTracer [candidate] (258.632 ms) : 0, 258632
AppSec [baseline] (167.98 ms) : 0, 167980
AppSec [candidate] (168.765 ms) : 0, 168765
Debugger [baseline] (67.958 ms) : 0, 67958
Debugger [candidate] (66.002 ms) : 0, 66002
Remote Config [baseline] (672.419 µs) : 0, 672
Remote Config [candidate] (658.839 µs) : 0, 659
Telemetry [baseline] (9.962 ms) : 0, 9962
Telemetry [candidate] (9.548 ms) : 0, 9548
Flare Poller [baseline] (3.769 ms) : 0, 3769
Flare Poller [candidate] (3.705 ms) : 0, 3705
IAST [baseline] (25.109 ms) : 0, 25109
IAST [candidate] (25.25 ms) : 0, 25250
section iast
crashtracking [baseline] (1.204 ms) : 0, 1204
crashtracking [candidate] (1.188 ms) : 0, 1188
BytebuddyAgent [baseline] (809.076 ms) : 0, 809076
BytebuddyAgent [candidate] (796.058 ms) : 0, 796058
AgentMeter [baseline] (11.605 ms) : 0, 11605
AgentMeter [candidate] (11.261 ms) : 0, 11261
GlobalTracer [baseline] (251.983 ms) : 0, 251983
GlobalTracer [candidate] (248.873 ms) : 0, 248873
AppSec [baseline] (33.193 ms) : 0, 33193
AppSec [candidate] (33.945 ms) : 0, 33945
Debugger [baseline] (69.89 ms) : 0, 69890
Debugger [candidate] (66.593 ms) : 0, 66593
Remote Config [baseline] (545.926 µs) : 0, 546
Remote Config [candidate] (538.1 µs) : 0, 538
Telemetry [baseline] (8.672 ms) : 0, 8672
Telemetry [candidate] (8.598 ms) : 0, 8598
Flare Poller [baseline] (3.54 ms) : 0, 3540
Flare Poller [candidate] (3.485 ms) : 0, 3485
IAST [baseline] (27.526 ms) : 0, 27526
IAST [candidate] (26.806 ms) : 0, 26806
section profiling
crashtracking [baseline] (1.219 ms) : 0, 1219
crashtracking [candidate] (1.22 ms) : 0, 1220
BytebuddyAgent [baseline] (683.341 ms) : 0, 683341
BytebuddyAgent [candidate] (687.488 ms) : 0, 687488
AgentMeter [baseline] (8.657 ms) : 0, 8657
AgentMeter [candidate] (8.663 ms) : 0, 8663
GlobalTracer [baseline] (216.422 ms) : 0, 216422
GlobalTracer [candidate] (217.344 ms) : 0, 217344
AppSec [baseline] (32.734 ms) : 0, 32734
AppSec [candidate] (32.848 ms) : 0, 32848
Debugger [baseline] (67.662 ms) : 0, 67662
Debugger [candidate] (67.641 ms) : 0, 67641
Remote Config [baseline] (633.022 µs) : 0, 633
Remote Config [candidate] (625.855 µs) : 0, 626
Telemetry [baseline] (8.917 ms) : 0, 8917
Telemetry [candidate] (9.003 ms) : 0, 9003
Flare Poller [baseline] (3.811 ms) : 0, 3811
Flare Poller [candidate] (3.812 ms) : 0, 3812
ProfilingAgent [baseline] (99.567 ms) : 0, 99567
ProfilingAgent [candidate] (99.857 ms) : 0, 99857
Profiling [baseline] (100.144 ms) : 0, 100144
Profiling [candidate] (100.438 ms) : 0, 100438
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.60.0-SNAPSHOT~5041298e48, baseline=1.60.0-SNAPSHOT~2ebc964340
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.073 s) : 0, 1072867
Total [baseline] (8.813 s) : 0, 8812796
Agent [candidate] (1.066 s) : 0, 1065609
Total [candidate] (8.759 s) : 0, 8759464
section iast
Agent [baseline] (1.242 s) : 0, 1242017
Total [baseline] (9.399 s) : 0, 9399036
Agent [candidate] (1.231 s) : 0, 1230808
Total [candidate] (9.369 s) : 0, 9369408
gantt
title insecure-bank - break down per module: candidate=1.60.0-SNAPSHOT~5041298e48, baseline=1.60.0-SNAPSHOT~2ebc964340
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.193 ms) : 0, 1193
crashtracking [candidate] (1.179 ms) : 0, 1179
BytebuddyAgent [baseline] (632.269 ms) : 0, 632269
BytebuddyAgent [candidate] (628.39 ms) : 0, 628390
AgentMeter [baseline] (29.518 ms) : 0, 29518
AgentMeter [candidate] (28.947 ms) : 0, 28947
GlobalTracer [baseline] (259.471 ms) : 0, 259471
GlobalTracer [candidate] (258.138 ms) : 0, 258138
AppSec [baseline] (33.215 ms) : 0, 33215
AppSec [candidate] (32.797 ms) : 0, 32797
Debugger [baseline] (63.46 ms) : 0, 63460
Debugger [candidate] (59.986 ms) : 0, 59986
Remote Config [baseline] (613.481 µs) : 0, 613
Remote Config [candidate] (634.09 µs) : 0, 634
Telemetry [baseline] (10.679 ms) : 0, 10679
Telemetry [candidate] (12.381 ms) : 0, 12381
Flare Poller [baseline] (6.929 ms) : 0, 6929
Flare Poller [candidate] (7.774 ms) : 0, 7774
section iast
crashtracking [baseline] (1.201 ms) : 0, 1201
crashtracking [candidate] (1.177 ms) : 0, 1177
BytebuddyAgent [baseline] (803.315 ms) : 0, 803315
BytebuddyAgent [candidate] (795.277 ms) : 0, 795277
AgentMeter [baseline] (11.457 ms) : 0, 11457
AgentMeter [candidate] (11.254 ms) : 0, 11254
GlobalTracer [baseline] (249.316 ms) : 0, 249316
GlobalTracer [candidate] (248.433 ms) : 0, 248433
AppSec [baseline] (35.185 ms) : 0, 35185
AppSec [candidate] (33.948 ms) : 0, 33948
Debugger [baseline] (65.564 ms) : 0, 65564
Debugger [candidate] (65.907 ms) : 0, 65907
Remote Config [baseline] (554.068 µs) : 0, 554
Remote Config [candidate] (542.665 µs) : 0, 543
Telemetry [baseline] (8.79 ms) : 0, 8790
Telemetry [candidate] (8.583 ms) : 0, 8583
Flare Poller [baseline] (3.556 ms) : 0, 3556
Flare Poller [candidate] (3.481 ms) : 0, 3481
IAST [baseline] (27.476 ms) : 0, 27476
IAST [candidate] (26.846 ms) : 0, 26846
LoadParameters
See matching parameters
SummaryFound 1 performance improvements and 1 performance regressions! Performance is the same for 19 metrics, 15 unstable metrics.
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~5041298e48, baseline=1.60.0-SNAPSHOT~2ebc964340
dateFormat X
axisFormat %s
section baseline
no_agent (1.197 ms) : 1185, 1208
. : milestone, 1197,
iast (3.1 ms) : 3062, 3137
. : milestone, 3100,
iast_FULL (6.064 ms) : 6002, 6125
. : milestone, 6064,
iast_GLOBAL (3.608 ms) : 3550, 3665
. : milestone, 3608,
profiling (2.056 ms) : 2037, 2076
. : milestone, 2056,
tracing (1.823 ms) : 1806, 1840
. : milestone, 1823,
section candidate
no_agent (1.172 ms) : 1161, 1183
. : milestone, 1172,
iast (3.169 ms) : 3127, 3212
. : milestone, 3169,
iast_FULL (5.838 ms) : 5779, 5897
. : milestone, 5838,
iast_GLOBAL (3.612 ms) : 3559, 3664
. : milestone, 3612,
profiling (2.22 ms) : 2198, 2243
. : milestone, 2220,
tracing (1.786 ms) : 1771, 1801
. : milestone, 1786,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~5041298e48, baseline=1.60.0-SNAPSHOT~2ebc964340
dateFormat X
axisFormat %s
section baseline
no_agent (19.146 ms) : 18958, 19333
. : milestone, 19146,
appsec (18.725 ms) : 18534, 18917
. : milestone, 18725,
code_origins (17.602 ms) : 17429, 17774
. : milestone, 17602,
iast (17.665 ms) : 17491, 17840
. : milestone, 17665,
profiling (18.449 ms) : 18265, 18633
. : milestone, 18449,
tracing (17.601 ms) : 17429, 17774
. : milestone, 17601,
section candidate
no_agent (18.166 ms) : 17978, 18353
. : milestone, 18166,
appsec (18.751 ms) : 18560, 18942
. : milestone, 18751,
code_origins (17.374 ms) : 17203, 17544
. : milestone, 17374,
iast (17.979 ms) : 17802, 18156
. : milestone, 17979,
profiling (18.719 ms) : 18533, 18906
. : milestone, 18719,
tracing (17.64 ms) : 17467, 17812
. : milestone, 17640,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~5041298e48, baseline=1.60.0-SNAPSHOT~2ebc964340
dateFormat X
axisFormat %s
section baseline
no_agent (1.486 ms) : 1475, 1498
. : milestone, 1486,
appsec (2.891 ms) : 2801, 2981
. : milestone, 2891,
iast (2.663 ms) : 2553, 2774
. : milestone, 2663,
iast_GLOBAL (2.71 ms) : 2599, 2821
. : milestone, 2710,
profiling (2.876 ms) : 2630, 3121
. : milestone, 2876,
tracing (2.432 ms) : 2342, 2521
. : milestone, 2432,
section candidate
no_agent (1.485 ms) : 1474, 1497
. : milestone, 1485,
appsec (2.849 ms) : 2760, 2939
. : milestone, 2849,
iast (2.658 ms) : 2548, 2768
. : milestone, 2658,
iast_GLOBAL (2.709 ms) : 2598, 2820
. : milestone, 2709,
profiling (2.456 ms) : 2365, 2548
. : milestone, 2456,
tracing (2.403 ms) : 2314, 2491
. : milestone, 2403,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~5041298e48, baseline=1.60.0-SNAPSHOT~2ebc964340
dateFormat X
axisFormat %s
section baseline
no_agent (14.916 s) : 14916000, 14916000
. : milestone, 14916000,
appsec (14.182 s) : 14182000, 14182000
. : milestone, 14182000,
iast (17.687 s) : 17687000, 17687000
. : milestone, 17687000,
iast_GLOBAL (16.995 s) : 16995000, 16995000
. : milestone, 16995000,
profiling (14.497 s) : 14497000, 14497000
. : milestone, 14497000,
tracing (14.615 s) : 14615000, 14615000
. : milestone, 14615000,
section candidate
no_agent (15.16 s) : 15160000, 15160000
. : milestone, 15160000,
appsec (14.616 s) : 14616000, 14616000
. : milestone, 14616000,
iast (17.651 s) : 17651000, 17651000
. : milestone, 17651000,
iast_GLOBAL (17.233 s) : 17233000, 17233000
. : milestone, 17233000,
profiling (14.352 s) : 14352000, 14352000
. : milestone, 14352000,
tracing (14.064 s) : 14064000, 14064000
. : milestone, 14064000,
|
Contributor
Author
@PerfectSlayer GH merge queues won't create a PR, correct. I'm not entirely certain, but from this documentation, it says "If your repository uses GitHub Actions to perform required checks on pull requests in your repository, you need to update the workflows to include the merge_group event as an additional trigger." From my understanding, |
sarahchen6
requested review from
AlexeyKuznetsov-DD and
randomanderson
and removed request for
a team
Contributor
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
sarahchen6
added
tag: no release notes
Contributor
Author
|
@AlexeyKuznetsov-DD brings up a really good point that GH merge queues do not have auto-retry capabilities (whereas Datadog MQs do). Though our flaky test rate has improved, we still have the occasional flake that would make the developer merge experience really annoying if there were no retries. |
Contributor
So there is a retry on top of the GitLab job retry?
Yes, and we can expect worst flake rate with test coverage increase.
Sure, I was curious to see how everything should assemble overall :) |
Contributor
Author
Yes (doc here)! According to the devflow team, there is one retry per job if the logs contain a known retry-able error. |
Contributor
Author
Right... I think GH merge queues will not be feasible then -- also because the entire pipeline would need to pass in one go in order for the merge queue to pass (no retrying just the failed job) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
Run all tests, in particular NON_DEFAULT_JVMs, on the Github merge queue. Also enable running system tests and skip the required check-PR requirements for merge queues.
Motivation
In comparison to the Datadog merge queue, Github merge queues provide better UI experiences for developers and more granular control over merge queue requirements. Let's try it out!
Additional Notes
Documentation for Github merge queues: https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue
This will be tested by using a merge queue to merge into the
sarahchen6/test-gh-mqbranch.Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]
Note: Once your PR is ready to merge, add it to the merge queue by commenting
/merge./merge -ccancels the queue request./merge -f --reason "reason"skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.