Rewrite the entire Azure DevOps build system #15808

DHowett · 2023-08-08T16:18:03Z

This pull request rewrites the entire Azure DevOps build system.

The guiding principles behind this rewrite are:

No pipeline definitions should contain steps (or tasks) directly.
All jobs should be in template files.
Any set of steps that is reused across multiple jobs must be in
template files.
All artifact names can be customized (via a property called
artifactStem on all templates that produce or consume artifacts).
No compilation happens outside of the "Build" phase, to consolidate
the production and indexing of PDBs.
Building the project produces a bin directory. That bin
directory is therefore the primary currency of the build. Jobs will
either produce or consume bin if they want to do anything with the
build outputs.
All step and job templates are named with step or job first,
which disambiguates them in the templates directory.
Most jobs can be run on different pools, so that we can put
expensive jobs on expensive build agents and cheap jobs on cheap
build agents. Some jobs handle pool selection on their own, however.

Our original build pipelines used the VSBuild task all over the
place. This resulted in Terminal being built in myriad ways, different
for every pipeline. There was an attempt at standardization early on,
where ci.yml consumed jobs and steps templates... but when
release.yml was added, all of that went out the window.

The new pipelines are consistent and focus on a small, well-defined set
of jobs:

job-build-project
- This is the big one!
- Takes a list of build configurations and platforms.
- Produces an artifact named build-PLATFORM-CONFIG for the entire
  matrix of possibilities.
- Optionally signs the output and produces a bill of materials.
- Admittedly has a lot going on.
job-build-package-wpf
- Takes a list of build configurations and platforms.
- Consumes the build- artifact for every config/platform
  possibility, plus one for "Any CPU" (hardcoded; this is where the
  .NET code builds)
- Produces one wpf-nupkg-CONFIG for each configuration, merging
  all platforms.
- Optionally signs the output and produces a bill of materials.
job-merge-msix-into-bundle
- Takes a list of build configurations and platforms.
- Consumes the build- artifact for every config/platform
- Produces one appxbundle-CONFIG for each configuration, merging
  all platforms for that config into one msixbundle.
- Optionally signs the output and produces a bill of materials.
job-package-conpty
- Takes a list of build configurations and platforms.
- Consumes the build- artifact for every config/platform
- Produces one conpty-nupkg-CONFIG for each configuration, merging
  all platforms.
- Optionally signs the output and produces a bill of materials.
job-test-project
- Takes one build config and one platform.
- Consumes build-PLATFORM-CONFIG
- Selects its own pools (hardcoded) because it knows about
  architectures and must choose the right agent arch.
- Runs tests (directly on the build agent).
job-run-pgo-tests
- Just like the above, but runs tests where IsPgo is true
- Collects all of the PGO counts and publishes a pgc-intermediates
  artifact for that platform and configuration.
job-pgo-merge-pgd
- Takes one build config and multiple platforms.
- Consumes build-$platform-CONFIG for each platform.
- Consumes pgc-intermediates-$platform-CONFIG for each platform.
- Merges the pgc files into pgd files
- Produces a new pgd- artifact.
job-pgo-build-nuget-and-publish
- Consumes the pgd- artifact from above.
- Packs it into a nupkg and publishes it.
job-submit-windows-vpack
- Only expected to run against Release.
- Consumes the appxbundle-CONFIG artifact.
- Publishes it to a vpack for Windows to consume.
job-check-code-format
- Does not use artifacts. Runs clang-format.
job-index-github-codenav
- Does not use artifacts.

Fuzz submission is broken due to changes in the onefuzz client.

I have removed the compliance and security build because it is no longer
supported.

Finally, this pull request has some additional benefits:

I've expanded the PGO build phase to cover ARM64!
We can remove everything Helix-related except the WTT parser
- We no longer depend on Helix submission or Helix pools
The WPF control's inner DLLs are now codesigned (The WPF nuget package isn't signed by our CI #15404)
Symbols for the WPF control, both .NET and C++, are published
alongside all other symbols.
The files we submit to ESRP for signing are batched up into a single
step¹

Closes #11874
Closes #11974
Closes #15404

This will have to change if we want to sign the individual
per-architecture .appx files before bundling so that they can be
directly installed. ↩

switch to signing fragments.

…n CI

DHowett · 2023-08-10T15:52:10Z

build/pipelines/fuzz.yml

-        env:
-          target_exe_path: $(Build.ArtifactStagingDirectory)/$(artifactName)/Fuzzing/x64/test/OpenConsoleFuzzer.exe
-          test_name: WriteCharsLegacy
+      - job:


ugh i said no pipeline yml file should ever have steps in it, but here I am violating that rule

DHowett · 2023-08-10T15:53:01Z

build/pipelines/pgo.yml

@@ -1,5 +1,27 @@
 trigger: none
 pr: none
+schedules:
+  - cron: "0 5 * * 2-6" # Run at 05:00 UTC Tuesday through Saturday (Even later than Localization, after the work day in Pacific, Mon-Fri)


I've switched the PGO build to have its own built-in nightly schedule, rather than relying on the triggers from the Azure DevOps side.

DHowett · 2023-08-10T15:53:21Z

build/pipelines/pgo.yml

+  - cron: "0 5 * * 2-6" # Run at 05:00 UTC Tuesday through Saturday (Even later than Localization, after the work day in Pacific, Mon-Fri)
+    displayName: "Nightly Instrumentation Build"
+    branches:
+      include:


In theory, we could automatically PGO release- branches if I had a way to set the branding correctly...

DHowett · 2023-08-10T15:54:51Z

build/pipelines/pgo.yml

+        - main
+    always: false # only run if there's code changes!
+
+parameters:


reviewer note: this file is an excellent overview of how the simple rules fit together.

PGO does this:

builds for x64, arm64 at the same time

after BOTH of those are done, runs PGO on x64 and arm64 at the same time

after both of THOSE are done, runs a single job to take all pgc files for all architectures and merge them into pgd files per architecture

after that, takes all merged PGD files and slaps them into a nuget

DHowett · 2023-08-10T15:55:40Z

build/pipelines/pgo.yml

+          buildConfigurations: [Release]
+          buildEverything: true
+          pgoBuildMode: Instrument
+          artifactStem: -instrumentation


This is a great use of artifactStem. It looks like this in the backend:

DHowett · 2023-08-10T15:56:46Z

build/pipelines/release.yml

+                includePseudoLoc: true
+
+      - ${{ if eq(parameters.buildWPF, true) }}:
+        # Add an Any CPU build flavor for the WPF control bits


This one is interesting. It takes the normal build-project job and runs it AGAIN with a different set of parameters. The most important one is buildPlatforms = [Any CPU]

DHowett · 2023-08-10T15:57:05Z

build/pipelines/release.yml

+          generateSbom: ${{ parameters.generateSbom }}
+          codeSign: ${{ parameters.codeSign }}
+          beforeBuildSteps: # Right before we build, lay down the universal package and localizations
+            - task: PkgESSetupBuild@12


This stuff is specific to the release build, and so it gets inserted by release.yml ONLY

DHowett · 2023-08-10T15:57:57Z

build/pipelines/templates-v2/job-pgo-merge-pgd.yml

+    - pwsh: |-
+        $Arch = '${{ platform }}'
+        $Conf = '${{ parameters.buildConfiguration }}'
+        $PGCDir = '$(Build.SourcesDirectory)/pgc/${{ platform }}/${{ parameters.buildConfiguration }}'
+        $PGDDir = '$(Build.SourcesDirectory)/pgd/${{ platform }}/${{ parameters.buildConfiguration }}'
+        # Flatten the PGD directory
+        Get-ChildItem $PGDDir -Recurse -Filter *.pgd | Move-Item -Destination $PGDDir -Verbose
+        Get-ChildItem $PGCDir -Filter *.pgc |
+          ForEach-Object {
+            $Parts = $_.Name -Split "!";
+            $_ | Add-Member Module $Parts[0] -PassThru
+          } |
+          Group-Object Module |
+          ForEach-Object {
+            & "$(VCToolsInstallDir)\bin\Hostx64\${{ platform }}\pgomgr.exe" /merge $_.Group.FullName "$PGDDir\$($_.Name).pgd"
+          }
+      displayName: Merge PGO Counts for ${{ platform }}


this is the replacement for that weird nuget package called pgo-helpers

DHowett · 2023-08-10T15:58:29Z

build/pipelines/templates-v2/job-run-pgo-tests.yml

+#  - task: PowerShell@2
+#    displayName: 'Convert Test Logs from WTL to xUnit format'
+#    inputs:
+#      targetType: filePath
+#      filePath: build\Helix\ConvertWttLogToXUnit.ps1
+#      arguments: -WttInputPath '${{ parameters.testLogPath }}' -WttSingleRerunInputPath 'unused.wtl' -WttMultipleRerunInputPath 'unused2.wtl' -XUnitOutputPath 'onBuildMachineResults.xml' -TestNamePrefix '$(BuildConfiguration).$(BuildPlatform)'
+#    condition: ne(variables['PGOBuildMode'], 'Instrument')
+#
+#  - task: PublishTestResults@2
+#    displayName: 'Upload converted test logs'
+#    condition: ne(variables['PGOBuildMode'], 'Instrument')
+#    inputs:
+#      testResultsFormat: 'xUnit' # Options: JUnit, NUnit, VSTest, xUnit, cTest
+#      testResultsFiles: '**/onBuildMachineResults.xml'
+#      testRunTitle: 'On Build Machine Tests' # Optional
+#      buildPlatform: $(BuildPlatform) # Optional
+#      buildConfiguration: $(BuildConfiguration) # Optional


Oh, i've gotta delete these commented craps

DHowett · 2023-08-10T16:48:54Z

So @carlos-zamora, you were asking about running things on the build agents. I would direct you to the job-run-pgo-tests job template. It is very simple: download bin for the right architecture, and run stuff inside it

DHowett · 2023-08-10T16:55:37Z

Hey, it's a lot of builds when you do this kind of thing!

DHowett · 2023-08-10T18:32:16Z

@zadjii-msft your tests are failing on all architectures!

DHowett · 2023-08-10T19:13:41Z

PGO publish failed because there's already a package with that version number on the feed (this is expected)

…erge-the-build-rules

lhecker

Yep, yes, of course. Clearly.

build/pipelines/templates-v2/steps-create-signing-config.yml

carlos-zamora

This is awesome! Thanks for doing this.

carlos-zamora · 2023-08-11T17:46:08Z

build/pipelines/feature-flag-ci.yml

@@ -7,6 +7,7 @@ pr:
  paths:
    include:
      - src/features.xml
+      - build/pipelines/feature-flag-ci.yml


Why does this file have to include itself??

So! This file isn't including itself in the sense that it will run yaml contained inside itself twice. This is part of the paths "included" in the trigger for this pipeline. i.e. this pipeline should run when either of these files is changed!

This pull request rewrites the entire Azure DevOps build system. The guiding principles behind this rewrite are: - No pipeline definitions should contain steps (or tasks) directly. - All jobs should be in template files. - Any set of steps that is reused across multiple jobs must be in template files. - All artifact names can be customized (via a property called `artifactStem` on all templates that produce or consume artifacts). - No compilation happens outside of the "Build" phase, to consolidate the production and indexing of PDBs. - **Building the project produces a `bin` directory.** That `bin` directory is therefore the primary currency of the build. Jobs will either produce or consume `bin` if they want to do anything with the build outputs. - All step and job templates are named with `step` or `job` _first_, which disambiguates them in the templates directory. - Most jobs can be run on different `pool`s, so that we can put expensive jobs on expensive build agents and cheap jobs on cheap build agents. Some jobs handle pool selection on their own, however. Our original build pipelines used the `VSBuild` task _all over the place._ This resulted in Terminal being built in myriad ways, different for every pipeline. There was an attempt at standardization early on, where `ci.yml` consumed jobs and steps templates... but when `release.yml` was added, all of that went out the window. The new pipelines are consistent and focus on a small, well-defined set of jobs: - `job-build-project` - This is the big one! - Takes a list of build configurations and platforms. - Produces an artifact named `build-PLATFORM-CONFIG` for the entire matrix of possibilities. - Optionally signs the output and produces a bill of materials. - Admittedly has a lot going on. - `job-build-package-wpf` - Takes a list of build configurations and platforms. - Consumes the `build-` artifact for every config/platform possibility, plus one for "Any CPU" (hardcoded; this is where the .NET code builds) - Produces one `wpf-nupkg-CONFIG` for each configuration, merging all platforms. - Optionally signs the output and produces a bill of materials. - `job-merge-msix-into-bundle` - Takes a list of build configurations and platforms. - Consumes the `build-` artifact for every config/platform - Produces one `appxbundle-CONFIG` for each configuration, merging all platforms for that config into one `msixbundle`. - Optionally signs the output and produces a bill of materials. - `job-package-conpty` - Takes a list of build configurations and platforms. - Consumes the `build-` artifact for every config/platform - Produces one `conpty-nupkg-CONFIG` for each configuration, merging all platforms. - Optionally signs the output and produces a bill of materials. - `job-test-project` - Takes **one** build config and **one** platform. - Consumes `build-PLATFORM-CONFIG` - Selects its own pools (hardcoded) because it knows about architectures and must choose the right agent arch. - Runs tests (directly on the build agent). - `job-run-pgo-tests` - Just like the above, but runs tests where `IsPgo` is `true` - Collects all of the PGO counts and publishes a `pgc-intermediates` artifact for that platform and configuration. - `job-pgo-merge-pgd` - Takes **one** build config and multiple platforms. - Consumes `build-$platform-CONFIG` for each platform. - Consumes `pgc-intermediates-$platform-CONFIG` for each platform. - Merges the `pgc` files into `pgd` files - Produces a new `pgd-` artifact. - `job-pgo-build-nuget-and-publish` - Consumes the `pgd-` artifact from above. - Packs it into a `nupkg` and publishes it. - `job-submit-windows-vpack` - Only expected to run against `Release`. - Consumes the `appxbundle-CONFIG` artifact. - Publishes it to a vpack for Windows to consume. - `job-check-code-format` - Does not use artifacts. Runs `clang-format`. - `job-index-github-codenav` - Does not use artifacts. Fuzz submission is broken due to changes in the `onefuzz` client. I have removed the compliance and security build because it is no longer supported. Finally, this pull request has some additional benefits: - I've expanded the PGO build phase to cover ARM64! - We can remove everything Helix-related except the WTT parser - We no longer depend on Helix submission or Helix pools - The WPF control's inner DLLs are now codesigned (#15404) - Symbols for the WPF control, both .NET and C++, are published alongside all other symbols. - The files we submit to ESRP for signing are batched up into a single step[^1] Closes #11874 Closes #11974 Closes #15404 [^1]: This will have to change if we want to sign the individual per-architecture `.appx` files before bundling so that they can be directly installed. (cherry picked from commit 69eff7e) Service-Card-Id: 90183388 Service-Version: 1.18

This pull request rewrites the entire Azure DevOps build system. The guiding principles behind this rewrite are: - No pipeline definitions should contain steps (or tasks) directly. - All jobs should be in template files. - Any set of steps that is reused across multiple jobs must be in template files. - All artifact names can be customized (via a property called `artifactStem` on all templates that produce or consume artifacts). - No compilation happens outside of the "Build" phase, to consolidate the production and indexing of PDBs. - **Building the project produces a `bin` directory.** That `bin` directory is therefore the primary currency of the build. Jobs will either produce or consume `bin` if they want to do anything with the build outputs. - All step and job templates are named with `step` or `job` _first_, which disambiguates them in the templates directory. - Most jobs can be run on different `pool`s, so that we can put expensive jobs on expensive build agents and cheap jobs on cheap build agents. Some jobs handle pool selection on their own, however. Our original build pipelines used the `VSBuild` task _all over the place._ This resulted in Terminal being built in myriad ways, different for every pipeline. There was an attempt at standardization early on, where `ci.yml` consumed jobs and steps templates... but when `release.yml` was added, all of that went out the window. The new pipelines are consistent and focus on a small, well-defined set of jobs: - `job-build-project` - This is the big one! - Takes a list of build configurations and platforms. - Produces an artifact named `build-PLATFORM-CONFIG` for the entire matrix of possibilities. - Optionally signs the output and produces a bill of materials. - Admittedly has a lot going on. - `job-build-package-wpf` - Takes a list of build configurations and platforms. - Consumes the `build-` artifact for every config/platform possibility, plus one for "Any CPU" (hardcoded; this is where the .NET code builds) - Produces one `wpf-nupkg-CONFIG` for each configuration, merging all platforms. - Optionally signs the output and produces a bill of materials. - `job-merge-msix-into-bundle` - Takes a list of build configurations and platforms. - Consumes the `build-` artifact for every config/platform - Produces one `appxbundle-CONFIG` for each configuration, merging all platforms for that config into one `msixbundle`. - Optionally signs the output and produces a bill of materials. - `job-package-conpty` - Takes a list of build configurations and platforms. - Consumes the `build-` artifact for every config/platform - Produces one `conpty-nupkg-CONFIG` for each configuration, merging all platforms. - Optionally signs the output and produces a bill of materials. - `job-test-project` - Takes **one** build config and **one** platform. - Consumes `build-PLATFORM-CONFIG` - Selects its own pools (hardcoded) because it knows about architectures and must choose the right agent arch. - Runs tests (directly on the build agent). - `job-run-pgo-tests` - Just like the above, but runs tests where `IsPgo` is `true` - Collects all of the PGO counts and publishes a `pgc-intermediates` artifact for that platform and configuration. - `job-pgo-merge-pgd` - Takes **one** build config and multiple platforms. - Consumes `build-$platform-CONFIG` for each platform. - Consumes `pgc-intermediates-$platform-CONFIG` for each platform. - Merges the `pgc` files into `pgd` files - Produces a new `pgd-` artifact. - `job-pgo-build-nuget-and-publish` - Consumes the `pgd-` artifact from above. - Packs it into a `nupkg` and publishes it. - `job-submit-windows-vpack` - Only expected to run against `Release`. - Consumes the `appxbundle-CONFIG` artifact. - Publishes it to a vpack for Windows to consume. - `job-check-code-format` - Does not use artifacts. Runs `clang-format`. - `job-index-github-codenav` - Does not use artifacts. Fuzz submission is broken due to changes in the `onefuzz` client. I have removed the compliance and security build because it is no longer supported. Finally, this pull request has some additional benefits: - I've expanded the PGO build phase to cover ARM64! - We can remove everything Helix-related except the WTT parser - We no longer depend on Helix submission or Helix pools - The WPF control's inner DLLs are now codesigned (#15404) - Symbols for the WPF control, both .NET and C++, are published alongside all other symbols. - The files we submit to ESRP for signing are batched up into a single step[^1] Closes #11874 Closes #11974 Closes #15404 [^1]: This will have to change if we want to sign the individual per-architecture `.appx` files before bundling so that they can be directly installed. (cherry picked from commit 69eff7e) Service-Card-Id: 90183387 Service-Version: 1.17

This pull request rewrites the entire Azure DevOps build system. The guiding principles behind this rewrite are: - No pipeline definitions should contain steps (or tasks) directly. - All jobs should be in template files. - Any set of steps that is reused across multiple jobs must be in template files. - All artifact names can be customized (via a property called `artifactStem` on all templates that produce or consume artifacts). - No compilation happens outside of the "Build" phase, to consolidate the production and indexing of PDBs. - All step and job templates are named with `step` or `job` _first_, which disambiguates them in the templates directory. - Most jobs can be run on different `pool`s, so that we can put expensive jobs on expensive build agents and cheap jobs on cheap build agents. Some jobs handle pool selection on their own, however. Our original build pipelines used the `VSBuild` task _all over the place._ This resulted in PowerToys being built in myriad ways, different for every pipeline. There was an attempt at standardization early on, where `ci.yml` consumed jobs and steps templates... but when `release.yml` was added, all of that went out the window. It's the same story as Terminal (microsoft/terminal#15808). The new pipelines are consistent and focus on a small, well-defined set of jobs: - `job-build-project` - This is the big one! - Takes a list of build configurations and platforms. - Produces an artifact named `build-PLATFORM-CONFIG` for the entire matrix of possibilities. - Builds all of the installers. - Optionally signs the output (all of the output). - Admittedly has a lot going on. - `job-test-project` - Takes **one** build config and **one** platform. - Consumes `build-PLATFORM-CONFIG` - Selects its own pools (hardcoded) because it knows about architectures and must choose the right agent arch. - Runs tests (directly on the build agent). - `job-publish-symbols-using-symbolrequestprod-api` - Consumes `**/*.pdb` from all prior build phases. - Uploads all PDBs in one artifact to Azure DevOps - Uses Microsoft's internal symbol publication REST API to submit stripped symbols to MSDL for public consumption. Finally, this pull request has some additional benefits: - Symbols are published to the private and public feeds at the same time, in the same step. They should be available in the public symbol server for public folks to debug against! - We have all the underpinnings necessary to run tests on ARM64 build agents. - Right now, `ScreenResolutionUtility` is broken - I had to introduce a custom version of `UseDotNet` which would install the right architecture (🤦); see microsoft/azure-pipelines-tasks#20300. - All dotnet and nuget versioning is consolidated into a small set of step templates. - This will provide a great place for us to handle versioning changes later, since all versioning happens in one place.

DHowett and others added 30 commits July 26, 2023 20:47

this cursed language is mankind's punishment

f992b4d

The punishment continues

7ece637

This language is a complete disappointment

83f701d

Okay, this could be the real deal?

d555591

Move around some params

c3ce184

Templates cannot have variables

5f39730

publish shortcut doesn't take artifactName, just artifact(lol)

19d2674

Template doesn't have packVpack

ca46063

oh i need a pool property

9e03153

Generate the code signing config at runtime... add buildEverything.

6b96b26

switch to signing fragments.

ADD A DEBUG THING

cdc1db1

oop forgot the wpf batch config

385ab12

Delete unused PDBs and useless libs from the drop

db23062

Fix the build name

8b47b0f

Maybe we can get down under a gigabyte...

b922070

Actually disable em for Static Lib

ac5301c

Okay, now for the main event: build the msixbundle

a29179b

syntax error

2f0f7e4

oop guard the pool parameter

f5edcbf

need length

b0bdffa

weird, you need to make it an array

7449f55

fix the artifact stem

3d1ab63

add additionalBuildOptions

691797a

Make the msbuild task more readable with a multiline string

365c324

Abstract out before-build steps, try to build and test the same job i…

e8cd067

…n CI

Actually, do audit too. And set the right pool.

66ae925

A value for the additionalBuildOptions etc etc provided

307eb04

Expected a sequence or a mapping

9670fb7

Okay dumbo

34ba91d

forgot a platform

e9af908

DHowett marked this pull request as ready for review August 9, 2023 23:03

DHowett commented Aug 10, 2023

View reviewed changes

h, clean up the pgo job

84201e9

Fix PGO, oops

a7da028

Merge remote-tracking branch 'origin/main' into dev/duhowett/try-to-m…

ec18c05

…erge-the-build-rules

lhecker approved these changes Aug 11, 2023

View reviewed changes

build/pipelines/templates-v2/steps-create-signing-config.yml Show resolved Hide resolved

build/pipelines/templates-v2/steps-create-signing-config.yml Show resolved Hide resolved

carlos-zamora approved these changes Aug 11, 2023

View reviewed changes

DHowett merged commit 69eff7e into main Aug 11, 2023
19 checks passed

DHowett deleted the dev/duhowett/try-to-merge-the-build-rules branch August 11, 2023 19:06

zadjii-msft mentioned this pull request Aug 24, 2023

Re-running just the tests will fail in CI, even if the tests pass #15876

Closed

DHowett mentioned this pull request Sep 24, 2024

Rewrite the entire Azure DevOps build system microsoft/PowerToys#34984

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rewrite the entire Azure DevOps build system #15808

Rewrite the entire Azure DevOps build system #15808

DHowett commented Aug 8, 2023 •

edited

Loading

DHowett Aug 10, 2023

DHowett Aug 10, 2023

DHowett Aug 10, 2023

DHowett Aug 10, 2023

DHowett Aug 10, 2023

DHowett Aug 10, 2023

DHowett Aug 10, 2023

DHowett Aug 10, 2023

DHowett Aug 10, 2023

DHowett commented Aug 10, 2023

DHowett commented Aug 10, 2023

DHowett commented Aug 10, 2023

DHowett commented Aug 10, 2023 •

edited

Loading

lhecker left a comment

carlos-zamora left a comment

carlos-zamora Aug 11, 2023

DHowett Aug 11, 2023

Rewrite the entire Azure DevOps build system #15808

Rewrite the entire Azure DevOps build system #15808

Conversation

DHowett commented Aug 8, 2023 • edited Loading

Footnotes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DHowett commented Aug 10, 2023

DHowett commented Aug 10, 2023

DHowett commented Aug 10, 2023

DHowett commented Aug 10, 2023 • edited Loading

lhecker left a comment

Choose a reason for hiding this comment

carlos-zamora left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DHowett commented Aug 8, 2023 •

edited

Loading

DHowett commented Aug 10, 2023 •

edited

Loading