Add GHA workflow to create a release branch and pin system tests #9782
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…it sha after minor release
|
🎯 Code Coverage 🔗 Commit SHA: 8dd3d64 | Docs | Was this helpful? Give us feedback! |
sarahchen6
requested review from
AlexeyKuznetsov-DD
and removed request for
a team
Contributor
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
sarahchen6
added
type: enhancement
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 6 unstable metrics. Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.55.0-SNAPSHOT~8dd3d64f72, baseline=1.55.0-SNAPSHOT~5d0320341a
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.019 s) : 0, 1018815
Total [baseline] (8.673 s) : 0, 8673173
Agent [candidate] (1.016 s) : 0, 1016239
Total [candidate] (8.668 s) : 0, 8667966
section iast
Agent [baseline] (1.15 s) : 0, 1150013
Total [baseline] (9.27 s) : 0, 9269697
Agent [candidate] (1.152 s) : 0, 1152280
Total [candidate] (9.276 s) : 0, 9276181
gantt
title insecure-bank - break down per module: candidate=1.55.0-SNAPSHOT~8dd3d64f72, baseline=1.55.0-SNAPSHOT~5d0320341a
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.471 ms) : 0, 1471
crashtracking [candidate] (1.462 ms) : 0, 1462
BytebuddyAgent [baseline] (694.367 ms) : 0, 694367
BytebuddyAgent [candidate] (692.628 ms) : 0, 692628
GlobalTracer [baseline] (242.833 ms) : 0, 242833
GlobalTracer [candidate] (242.702 ms) : 0, 242702
AppSec [baseline] (32.324 ms) : 0, 32324
AppSec [candidate] (32.069 ms) : 0, 32069
Debugger [baseline] (6.308 ms) : 0, 6308
Debugger [candidate] (6.266 ms) : 0, 6266
Remote Config [baseline] (685.427 µs) : 0, 685
Remote Config [candidate] (684.165 µs) : 0, 684
Telemetry [baseline] (9.383 ms) : 0, 9383
Telemetry [candidate] (9.321 ms) : 0, 9321
Flare Poller [baseline] (10.162 ms) : 0, 10162
Flare Poller [candidate] (10.022 ms) : 0, 10022
section iast
crashtracking [baseline] (1.496 ms) : 0, 1496
crashtracking [candidate] (1.499 ms) : 0, 1499
BytebuddyAgent [baseline] (813.739 ms) : 0, 813739
BytebuddyAgent [candidate] (816.184 ms) : 0, 816184
GlobalTracer [baseline] (231.867 ms) : 0, 231867
GlobalTracer [candidate] (231.663 ms) : 0, 231663
IAST [baseline] (26.831 ms) : 0, 26831
IAST [candidate] (26.622 ms) : 0, 26622
AppSec [baseline] (34.921 ms) : 0, 34921
AppSec [candidate] (35.04 ms) : 0, 35040
Debugger [baseline] (6.174 ms) : 0, 6174
Debugger [candidate] (6.184 ms) : 0, 6184
Remote Config [baseline] (611.145 µs) : 0, 611
Remote Config [candidate] (616.186 µs) : 0, 616
Telemetry [baseline] (8.779 ms) : 0, 8779
Telemetry [candidate] (8.796 ms) : 0, 8796
Flare Poller [baseline] (4.267 ms) : 0, 4267
Flare Poller [candidate] (4.224 ms) : 0, 4224
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.55.0-SNAPSHOT~8dd3d64f72, baseline=1.55.0-SNAPSHOT~5d0320341a
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.021 s) : 0, 1021086
Total [baseline] (10.629 s) : 0, 10629406
Agent [candidate] (1.028 s) : 0, 1028073
Total [candidate] (10.813 s) : 0, 10812719
section appsec
Agent [baseline] (1.197 s) : 0, 1197348
Total [baseline] (10.785 s) : 0, 10785154
Agent [candidate] (1.218 s) : 0, 1217949
Total [candidate] (10.916 s) : 0, 10915520
section iast
Agent [baseline] (1.152 s) : 0, 1152101
Total [baseline] (11.027 s) : 0, 11027219
Agent [candidate] (1.151 s) : 0, 1150737
Total [candidate] (11.074 s) : 0, 11074470
section profiling
Agent [baseline] (1.17 s) : 0, 1170066
Total [baseline] (10.889 s) : 0, 10888954
Agent [candidate] (1.187 s) : 0, 1186650
Total [candidate] (11.02 s) : 0, 11019792
gantt
title petclinic - break down per module: candidate=1.55.0-SNAPSHOT~8dd3d64f72, baseline=1.55.0-SNAPSHOT~5d0320341a
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.471 ms) : 0, 1471
crashtracking [candidate] (1.494 ms) : 0, 1494
BytebuddyAgent [baseline] (694.708 ms) : 0, 694708
BytebuddyAgent [candidate] (701.058 ms) : 0, 701058
GlobalTracer [baseline] (243.444 ms) : 0, 243444
GlobalTracer [candidate] (244.397 ms) : 0, 244397
AppSec [baseline] (32.112 ms) : 0, 32112
AppSec [candidate] (32.421 ms) : 0, 32421
Debugger [baseline] (6.294 ms) : 0, 6294
Debugger [candidate] (6.353 ms) : 0, 6353
Remote Config [baseline] (677.579 µs) : 0, 678
Remote Config [candidate] (688.516 µs) : 0, 689
Telemetry [baseline] (9.347 ms) : 0, 9347
Telemetry [candidate] (9.337 ms) : 0, 9337
Flare Poller [baseline] (11.745 ms) : 0, 11745
Flare Poller [candidate] (10.945 ms) : 0, 10945
section appsec
crashtracking [baseline] (1.472 ms) : 0, 1472
crashtracking [candidate] (1.499 ms) : 0, 1499
BytebuddyAgent [baseline] (719.607 ms) : 0, 719607
BytebuddyAgent [candidate] (733.284 ms) : 0, 733284
GlobalTracer [baseline] (235.6 ms) : 0, 235600
GlobalTracer [candidate] (239.335 ms) : 0, 239335
IAST [baseline] (24.911 ms) : 0, 24911
IAST [candidate] (25.591 ms) : 0, 25591
AppSec [baseline] (175.536 ms) : 0, 175536
AppSec [candidate] (177.102 ms) : 0, 177102
Debugger [baseline] (6.101 ms) : 0, 6101
Debugger [candidate] (6.18 ms) : 0, 6180
Remote Config [baseline] (622.854 µs) : 0, 623
Remote Config [candidate] (643.598 µs) : 0, 644
Telemetry [baseline] (8.425 ms) : 0, 8425
Telemetry [candidate] (8.777 ms) : 0, 8777
Flare Poller [baseline] (3.865 ms) : 0, 3865
Flare Poller [candidate] (4.054 ms) : 0, 4054
section iast
crashtracking [baseline] (1.484 ms) : 0, 1484
crashtracking [candidate] (1.463 ms) : 0, 1463
BytebuddyAgent [baseline] (815.09 ms) : 0, 815090
BytebuddyAgent [candidate] (814.98 ms) : 0, 814980
GlobalTracer [baseline] (231.972 ms) : 0, 231972
GlobalTracer [candidate] (231.664 ms) : 0, 231664
IAST [baseline] (26.934 ms) : 0, 26934
IAST [candidate] (26.489 ms) : 0, 26489
AppSec [baseline] (35.177 ms) : 0, 35177
AppSec [candidate] (35.073 ms) : 0, 35073
Debugger [baseline] (6.167 ms) : 0, 6167
Debugger [candidate] (6.106 ms) : 0, 6106
Remote Config [baseline] (619.568 µs) : 0, 620
Remote Config [candidate] (592.818 µs) : 0, 593
Telemetry [baseline] (8.892 ms) : 0, 8892
Telemetry [candidate] (8.691 ms) : 0, 8691
Flare Poller [baseline] (4.234 ms) : 0, 4234
Flare Poller [candidate] (4.236 ms) : 0, 4236
section profiling
crashtracking [baseline] (1.471 ms) : 0, 1471
crashtracking [candidate] (1.488 ms) : 0, 1488
BytebuddyAgent [baseline] (723.688 ms) : 0, 723688
BytebuddyAgent [candidate] (734.08 ms) : 0, 734080
GlobalTracer [baseline] (219.994 ms) : 0, 219994
GlobalTracer [candidate] (222.47 ms) : 0, 222470
AppSec [baseline] (32.607 ms) : 0, 32607
AppSec [candidate] (32.986 ms) : 0, 32986
Debugger [baseline] (7.486 ms) : 0, 7486
Debugger [candidate] (7.678 ms) : 0, 7678
Remote Config [baseline] (720.365 µs) : 0, 720
Remote Config [candidate] (1.485 ms) : 0, 1485
Telemetry [baseline] (15.249 ms) : 0, 15249
Telemetry [candidate] (13.223 ms) : 0, 13223
Flare Poller [baseline] (4.157 ms) : 0, 4157
Flare Poller [candidate] (5.886 ms) : 0, 5886
ProfilingAgent [baseline] (110.565 ms) : 0, 110565
ProfilingAgent [candidate] (112.593 ms) : 0, 112593
Profiling [baseline] (111.212 ms) : 0, 111212
Profiling [candidate] (113.22 ms) : 0, 113220
LoadParameters
See matching parameters
SummaryFound 3 performance improvements and 2 performance regressions! Performance is the same for 7 metrics, 12 unstable metrics.
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.55.0-SNAPSHOT~8dd3d64f72, baseline=1.55.0-SNAPSHOT~5d0320341a
dateFormat X
axisFormat %s
section baseline
no_agent (37.139 ms) : 36832, 37445
. : milestone, 37139,
appsec (47.802 ms) : 47367, 48236
. : milestone, 47802,
code_origins (43.307 ms) : 42927, 43687
. : milestone, 43307,
iast (43.726 ms) : 43366, 44086
. : milestone, 43726,
profiling (49.719 ms) : 49221, 50216
. : milestone, 49719,
tracing (44.503 ms) : 44134, 44871
. : milestone, 44503,
section candidate
no_agent (37.445 ms) : 37131, 37758
. : milestone, 37445,
appsec (46.91 ms) : 46510, 47310
. : milestone, 46910,
code_origins (43.406 ms) : 43032, 43780
. : milestone, 43406,
iast (42.994 ms) : 42613, 43376
. : milestone, 42994,
profiling (47.111 ms) : 46681, 47541
. : milestone, 47111,
tracing (43.896 ms) : 43515, 44277
. : milestone, 43896,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.55.0-SNAPSHOT~8dd3d64f72, baseline=1.55.0-SNAPSHOT~5d0320341a
dateFormat X
axisFormat %s
section baseline
no_agent (4.249 ms) : 4192, 4306
. : milestone, 4249,
iast (9.627 ms) : 9469, 9785
. : milestone, 9627,
iast_FULL (15.389 ms) : 15080, 15698
. : milestone, 15389,
iast_GLOBAL (10.832 ms) : 10630, 11035
. : milestone, 10832,
profiling (8.866 ms) : 8711, 9020
. : milestone, 8866,
tracing (7.829 ms) : 7710, 7947
. : milestone, 7829,
section candidate
no_agent (4.367 ms) : 4317, 4417
. : milestone, 4367,
iast (10.119 ms) : 9944, 10295
. : milestone, 10119,
iast_FULL (13.792 ms) : 13518, 14066
. : milestone, 13792,
iast_GLOBAL (9.876 ms) : 9704, 10048
. : milestone, 9876,
profiling (9.315 ms) : 9167, 9464
. : milestone, 9315,
tracing (7.83 ms) : 7708, 7952
. : milestone, 7830,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.55.0-SNAPSHOT~8dd3d64f72, baseline=1.55.0-SNAPSHOT~5d0320341a
dateFormat X
axisFormat %s
section baseline
no_agent (1.474 ms) : 1462, 1485
. : milestone, 1474,
appsec (3.713 ms) : 3495, 3930
. : milestone, 3713,
iast (2.207 ms) : 2143, 2270
. : milestone, 2207,
iast_GLOBAL (2.24 ms) : 2176, 2304
. : milestone, 2240,
profiling (2.053 ms) : 2001, 2105
. : milestone, 2053,
tracing (2.017 ms) : 1967, 2066
. : milestone, 2017,
section candidate
no_agent (1.472 ms) : 1460, 1483
. : milestone, 1472,
appsec (3.75 ms) : 3526, 3973
. : milestone, 3750,
iast (2.209 ms) : 2145, 2272
. : milestone, 2209,
iast_GLOBAL (2.252 ms) : 2188, 2316
. : milestone, 2252,
profiling (2.042 ms) : 1991, 2093
. : milestone, 2042,
tracing (2.021 ms) : 1972, 2071
. : milestone, 2021,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.55.0-SNAPSHOT~8dd3d64f72, baseline=1.55.0-SNAPSHOT~5d0320341a
dateFormat X
axisFormat %s
section baseline
no_agent (14.893 s) : 14893000, 14893000
. : milestone, 14893000,
appsec (15.046 s) : 15046000, 15046000
. : milestone, 15046000,
iast (18.49 s) : 18490000, 18490000
. : milestone, 18490000,
iast_GLOBAL (18.113 s) : 18113000, 18113000
. : milestone, 18113000,
profiling (15.389 s) : 15389000, 15389000
. : milestone, 15389000,
tracing (15.326 s) : 15326000, 15326000
. : milestone, 15326000,
section candidate
no_agent (14.922 s) : 14922000, 14922000
. : milestone, 14922000,
appsec (14.837 s) : 14837000, 14837000
. : milestone, 14837000,
iast (18.855 s) : 18855000, 18855000
. : milestone, 18855000,
iast_GLOBAL (17.725 s) : 17725000, 17725000
. : milestone, 17725000,
profiling (15.074 s) : 15074000, 15074000
. : milestone, 15074000,
tracing (15.358 s) : 15358000, 15358000
. : milestone, 15358000,
|
| id: define-branch | ||
| run: | | ||
| TAG=${{ steps.determine-tag.outputs.tag }} | ||
| BRANCH=$(echo "$TAG" | sed -E 's/^(v[0-9]+\.[0-9]+)\.0$/release\/\1.x/') |
Contributor
There was a problem hiding this comment.
💭 thought: This \1.x is so confusing 😵
Contributor
Author
There was a problem hiding this comment.
Yeah 😬 I believe it's taking the first matched argument and adding .x instead of .0, but let me see if this can be simplified.
Contributor
Author
There was a problem hiding this comment.
💡 Since I'm already checking that the $TAG is in the correct format, I only need to replace the trailing 0 with x !
| echo "tag=${TAG}" >> "$GITHUB_OUTPUT" | ||
| echo "Processing release tag: ${TAG}" | ||
|
|
||
| - name: Validate tag format |
Contributor
There was a problem hiding this comment.
🎯 suggestion: I would merge determine tag and validate tag format for simplicity.
Comment on lines
70
to
93
| - name: Checkout system-tests to get latest SHA | ||
| if: steps.check-branch.outputs.exists == 'false' | ||
| id: system-test-ref | ||
| uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # 5.0.0 | ||
| with: | ||
| repository: "DataDog/system-tests" | ||
| path: system-tests | ||
| ref: main | ||
|
|
||
| - name: Update reference 1/2 in run-system-tests.yaml | ||
| if: steps.check-branch.outputs.exists == 'false' | ||
| run: .github/scripts/update_system_test_reference.sh | ||
| env: | ||
| TARGET: ".github/workflows/run-system-tests.yaml" | ||
| PATTERN: '(\s*system-tests\.yml@)(\S+)(\s+# system tests.*)' | ||
| REF: ${{ steps.system-test-ref.outputs.commit }} | ||
|
|
||
| - name: Update reference 2/2 in run-system-tests.yaml | ||
| if: steps.check-branch.outputs.exists == 'false' | ||
| run: .github/scripts/update_system_test_reference.sh | ||
| env: | ||
| TARGET: ".github/workflows/run-system-tests.yaml" | ||
| PATTERN: '(\s*ref: )(\S+)(\s+# system tests.*)' | ||
| REF: ${{ steps.system-test-ref.outputs.commit }} |
Contributor
There was a problem hiding this comment.
🎯 suggestion: I would simplify this part too with the following changes:
First I would specialize the script update_system_test_reference.sh to take the desired system test branch as parameter. It would make all the changes needed in the run-system-tests workflow (and even elsewhere if needed). This will also simplify the pattern cleaning thing -- you can implement all the str replace there directly.
We used to have a such script from Santi. It make sense to move it to /tooling too.
Then, you don't need to checkout the system test repo to get the SHA1 of a branch.
You can use git ls-remote https://github.com/DataDog/system-tests <branch-name> and then cut -f 1 like:
COMMIT_REF=$(git ls-remote https://github.com/DataDog/system-tests <branch-name> | cut -f 1)
Contributor
Author
There was a problem hiding this comment.
Oh thanks, good idea. Will do!
PerfectSlayer
approved these changes
Oct 20, 2025
Contributor
PerfectSlayer
left a comment
PerfectSlayer
left a comment
There was a problem hiding this comment.
👏 praise: Looks way simpler now :)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
This PR adds a GHA workflow that is triggered either manually or when a tag is pushed for minor releases. It also adds a corresponding script that updates the commit SHA that we use to run system tests and a
dd-octo-ststrust policy for the workflow.If the release branch does not already exist, this workflow creates a branch that corresponds to the git tag that triggered the workflow (e.g.
v1.54.0should create the branchrelease/v1.54.x). The workflow then updates the system-tests commit SHA inrun-system-tests.yamlto the then-latest SHA available via the script and pushes this with the newly created release branch.Motivation
We want our
masterbranch to run with the latest head of system-tests. However, when making patch releases off of a release branch, the patch should only be tested against the head of system-tests at the time that the minor release was made. Thus, whenever a minor release is made (i.e. a git tag of the patternvX.Y.0is pushed), we can immediately create the release branch that pins system tests to the then-current head. Whenever we go back to this branch to make patch releases, we should now only need to test against this pinned version of system-tests instead of the latest head onmaster.Additional Notes
Unfortunately, since we are using a
dd-octo-ststrust policy that only works offmaster, I cannot test this fully until the policy and workflow are merged. The next minor release isn't planned until Nov 3rd, but in the meantime, I can test this workflow with the manual trigger.Contributor Checklist
type:and (comp:orinst:) labels in addition to any useful labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]