Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NugetCommand@2 versus GUID@2... can we converge #11874

Closed
miniksa opened this issue Dec 3, 2021 · 0 comments · Fixed by #15808
Closed

NugetCommand@2 versus GUID@2... can we converge #11874

miniksa opened this issue Dec 3, 2021 · 0 comments · Fixed by #15808
Assignees
Labels
Area-Build Issues pertaining to the build system, CI, infrastructure, meta In-PR This issue has a related PR Issue-Task It's a feature request, but it doesn't really need a major design. Needs-Tag-Fix Doesn't match tag requirements Product-Meta The product is the management of the products.

Comments

@miniksa
Copy link
Member

miniksa commented Dec 3, 2021

Why do you have to use the guid here but not in the task immediately under this?

Originally posted by @carlos-zamora in #11819 (comment)

@ghost ghost added Needs-Triage It's a new issue that the core contributor team needs to triage at the next triage meeting Needs-Tag-Fix Doesn't match tag requirements labels Dec 3, 2021
@miniksa miniksa self-assigned this Dec 3, 2021
@miniksa miniksa added this to the Engineering Improvements 2021 milestone Dec 3, 2021
@miniksa miniksa added Area-Build Issues pertaining to the build system, CI, infrastructure, meta Issue-Task It's a feature request, but it doesn't really need a major design. Product-Meta The product is the management of the products. labels Dec 3, 2021
@ghost ghost removed the Needs-Tag-Fix Doesn't match tag requirements label Dec 3, 2021
@zadjii-msft zadjii-msft removed the Needs-Triage It's a new issue that the core contributor team needs to triage at the next triage meeting label Dec 6, 2021
@miniksa miniksa removed their assignment Mar 31, 2022
@DHowett DHowett self-assigned this Aug 8, 2023
DHowett added a commit that referenced this issue Aug 8, 2023
@microsoft-github-policy-service microsoft-github-policy-service bot added the In-PR This issue has a related PR label Aug 10, 2023
DHowett added a commit that referenced this issue Aug 11, 2023
This pull request rewrites the entire Azure DevOps build system.

The guiding principles behind this rewrite are:

- No pipeline definitions should contain steps (or tasks) directly.
- All jobs should be in template files.
- Any set of steps that is reused across multiple jobs must be in
  template files.
- All artifact names can be customized (via a property called
  `artifactStem` on all templates that produce or consume artifacts).
- No compilation happens outside of the "Build" phase, to consolidate
  the production and indexing of PDBs.
- **Building the project produces a `bin` directory.** That `bin`
  directory is therefore the primary currency of the build. Jobs will
  either produce or consume `bin` if they want to do anything with the
  build outputs.
- All step and job templates are named with `step` or `job` _first_,
  which disambiguates them in the templates directory.
- Most jobs can be run on different `pool`s, so that we can put
  expensive jobs on expensive build agents and cheap jobs on cheap
  build agents. Some jobs handle pool selection on their own, however.

Our original build pipelines used the `VSBuild` task _all over the
place._ This resulted in Terminal being built in myriad ways, different
for every pipeline. There was an attempt at standardization early on,
where `ci.yml` consumed jobs and steps templates... but when
`release.yml` was added, all of that went out the window.

The new pipelines are consistent and focus on a small, well-defined set
of jobs:

- `job-build-project`
    - This is the big one!
    - Takes a list of build configurations and platforms.
    - Produces an artifact named `build-PLATFORM-CONFIG` for the entire
      matrix of possibilities.
    - Optionally signs the output and produces a bill of materials.
    - Admittedly has a lot going on.
- `job-build-package-wpf`
    - Takes a list of build configurations and platforms.
    - Consumes the `build-` artifact for every config/platform
      possibility, plus one for "Any CPU" (hardcoded; this is where the
      .NET code builds)
    - Produces one `wpf-nupkg-CONFIG` for each configuration, merging
      all platforms.
    - Optionally signs the output and produces a bill of materials.
- `job-merge-msix-into-bundle`
    - Takes a list of build configurations and platforms.
    - Consumes the `build-` artifact for every config/platform
    - Produces one `appxbundle-CONFIG` for each configuration, merging
      all platforms for that config into one `msixbundle`.
    - Optionally signs the output and produces a bill of materials.
- `job-package-conpty`
    - Takes a list of build configurations and platforms.
    - Consumes the `build-` artifact for every config/platform
    - Produces one `conpty-nupkg-CONFIG` for each configuration, merging
      all platforms.
    - Optionally signs the output and produces a bill of materials.
- `job-test-project`
    - Takes **one** build config and **one** platform.
    - Consumes `build-PLATFORM-CONFIG`
    - Selects its own pools (hardcoded) because it knows about
      architectures and must choose the right agent arch.
    - Runs tests (directly on the build agent).
- `job-run-pgo-tests`
    - Just like the above, but runs tests where `IsPgo` is `true`
    - Collects all of the PGO counts and publishes a `pgc-intermediates`
      artifact for that platform and configuration.
- `job-pgo-merge-pgd`
    - Takes **one** build config and multiple platforms.
    - Consumes `build-$platform-CONFIG` for each platform.
    - Consumes `pgc-intermediates-$platform-CONFIG` for each platform.
    - Merges the `pgc` files into `pgd` files
    - Produces a new `pgd-` artifact.
- `job-pgo-build-nuget-and-publish`
    - Consumes the `pgd-` artifact from above.
    - Packs it into a `nupkg` and publishes it.
- `job-submit-windows-vpack`
    - Only expected to run against `Release`.
    - Consumes the `appxbundle-CONFIG` artifact.
    - Publishes it to a vpack for Windows to consume.
- `job-check-code-format`
    - Does not use artifacts. Runs `clang-format`.
- `job-index-github-codenav`
    - Does not use artifacts.

Fuzz submission is broken due to changes in the `onefuzz` client.

I have removed the compliance and security build because it is no longer
supported.

Finally, this pull request has some additional benefits:

- I've expanded the PGO build phase to cover ARM64!
- We can remove everything Helix-related except the WTT parser
    - We no longer depend on Helix submission or Helix pools
- The WPF control's inner DLLs are now codesigned (#15404)
- Symbols for the WPF control, both .NET and C++, are published
  alongside all other symbols.
- The files we submit to ESRP for signing are batched up into a single
  step[^1]

Closes #11874
Closes #11974
Closes #15404

[^1]: This will have to change if we want to sign the individual
per-architecture `.appx` files before bundling so that they can be
directly installed.
@microsoft-github-policy-service microsoft-github-policy-service bot added the Needs-Tag-Fix Doesn't match tag requirements label Aug 11, 2023
DHowett added a commit that referenced this issue Aug 11, 2023
This pull request rewrites the entire Azure DevOps build system.

The guiding principles behind this rewrite are:

- No pipeline definitions should contain steps (or tasks) directly.
- All jobs should be in template files.
- Any set of steps that is reused across multiple jobs must be in
  template files.
- All artifact names can be customized (via a property called
  `artifactStem` on all templates that produce or consume artifacts).
- No compilation happens outside of the "Build" phase, to consolidate
  the production and indexing of PDBs.
- **Building the project produces a `bin` directory.** That `bin`
  directory is therefore the primary currency of the build. Jobs will
  either produce or consume `bin` if they want to do anything with the
  build outputs.
- All step and job templates are named with `step` or `job` _first_,
  which disambiguates them in the templates directory.
- Most jobs can be run on different `pool`s, so that we can put
  expensive jobs on expensive build agents and cheap jobs on cheap
  build agents. Some jobs handle pool selection on their own, however.

Our original build pipelines used the `VSBuild` task _all over the
place._ This resulted in Terminal being built in myriad ways, different
for every pipeline. There was an attempt at standardization early on,
where `ci.yml` consumed jobs and steps templates... but when
`release.yml` was added, all of that went out the window.

The new pipelines are consistent and focus on a small, well-defined set
of jobs:

- `job-build-project`
    - This is the big one!
    - Takes a list of build configurations and platforms.
    - Produces an artifact named `build-PLATFORM-CONFIG` for the entire
      matrix of possibilities.
    - Optionally signs the output and produces a bill of materials.
    - Admittedly has a lot going on.
- `job-build-package-wpf`
    - Takes a list of build configurations and platforms.
    - Consumes the `build-` artifact for every config/platform
      possibility, plus one for "Any CPU" (hardcoded; this is where the
      .NET code builds)
    - Produces one `wpf-nupkg-CONFIG` for each configuration, merging
      all platforms.
    - Optionally signs the output and produces a bill of materials.
- `job-merge-msix-into-bundle`
    - Takes a list of build configurations and platforms.
    - Consumes the `build-` artifact for every config/platform
    - Produces one `appxbundle-CONFIG` for each configuration, merging
      all platforms for that config into one `msixbundle`.
    - Optionally signs the output and produces a bill of materials.
- `job-package-conpty`
    - Takes a list of build configurations and platforms.
    - Consumes the `build-` artifact for every config/platform
    - Produces one `conpty-nupkg-CONFIG` for each configuration, merging
      all platforms.
    - Optionally signs the output and produces a bill of materials.
- `job-test-project`
    - Takes **one** build config and **one** platform.
    - Consumes `build-PLATFORM-CONFIG`
    - Selects its own pools (hardcoded) because it knows about
      architectures and must choose the right agent arch.
    - Runs tests (directly on the build agent).
- `job-run-pgo-tests`
    - Just like the above, but runs tests where `IsPgo` is `true`
    - Collects all of the PGO counts and publishes a `pgc-intermediates`
      artifact for that platform and configuration.
- `job-pgo-merge-pgd`
    - Takes **one** build config and multiple platforms.
    - Consumes `build-$platform-CONFIG` for each platform.
    - Consumes `pgc-intermediates-$platform-CONFIG` for each platform.
    - Merges the `pgc` files into `pgd` files
    - Produces a new `pgd-` artifact.
- `job-pgo-build-nuget-and-publish`
    - Consumes the `pgd-` artifact from above.
    - Packs it into a `nupkg` and publishes it.
- `job-submit-windows-vpack`
    - Only expected to run against `Release`.
    - Consumes the `appxbundle-CONFIG` artifact.
    - Publishes it to a vpack for Windows to consume.
- `job-check-code-format`
    - Does not use artifacts. Runs `clang-format`.
- `job-index-github-codenav`
    - Does not use artifacts.

Fuzz submission is broken due to changes in the `onefuzz` client.

I have removed the compliance and security build because it is no longer
supported.

Finally, this pull request has some additional benefits:

- I've expanded the PGO build phase to cover ARM64!
- We can remove everything Helix-related except the WTT parser
    - We no longer depend on Helix submission or Helix pools
- The WPF control's inner DLLs are now codesigned (#15404)
- Symbols for the WPF control, both .NET and C++, are published
  alongside all other symbols.
- The files we submit to ESRP for signing are batched up into a single
  step[^1]

Closes #11874
Closes #11974
Closes #15404

[^1]: This will have to change if we want to sign the individual
per-architecture `.appx` files before bundling so that they can be
directly installed.

(cherry picked from commit 69eff7e)
Service-Card-Id: 90183388
Service-Version: 1.18
DHowett added a commit that referenced this issue Aug 11, 2023
This pull request rewrites the entire Azure DevOps build system.

The guiding principles behind this rewrite are:

- No pipeline definitions should contain steps (or tasks) directly.
- All jobs should be in template files.
- Any set of steps that is reused across multiple jobs must be in
  template files.
- All artifact names can be customized (via a property called
  `artifactStem` on all templates that produce or consume artifacts).
- No compilation happens outside of the "Build" phase, to consolidate
  the production and indexing of PDBs.
- **Building the project produces a `bin` directory.** That `bin`
  directory is therefore the primary currency of the build. Jobs will
  either produce or consume `bin` if they want to do anything with the
  build outputs.
- All step and job templates are named with `step` or `job` _first_,
  which disambiguates them in the templates directory.
- Most jobs can be run on different `pool`s, so that we can put
  expensive jobs on expensive build agents and cheap jobs on cheap
  build agents. Some jobs handle pool selection on their own, however.

Our original build pipelines used the `VSBuild` task _all over the
place._ This resulted in Terminal being built in myriad ways, different
for every pipeline. There was an attempt at standardization early on,
where `ci.yml` consumed jobs and steps templates... but when
`release.yml` was added, all of that went out the window.

The new pipelines are consistent and focus on a small, well-defined set
of jobs:

- `job-build-project`
    - This is the big one!
    - Takes a list of build configurations and platforms.
    - Produces an artifact named `build-PLATFORM-CONFIG` for the entire
      matrix of possibilities.
    - Optionally signs the output and produces a bill of materials.
    - Admittedly has a lot going on.
- `job-build-package-wpf`
    - Takes a list of build configurations and platforms.
    - Consumes the `build-` artifact for every config/platform
      possibility, plus one for "Any CPU" (hardcoded; this is where the
      .NET code builds)
    - Produces one `wpf-nupkg-CONFIG` for each configuration, merging
      all platforms.
    - Optionally signs the output and produces a bill of materials.
- `job-merge-msix-into-bundle`
    - Takes a list of build configurations and platforms.
    - Consumes the `build-` artifact for every config/platform
    - Produces one `appxbundle-CONFIG` for each configuration, merging
      all platforms for that config into one `msixbundle`.
    - Optionally signs the output and produces a bill of materials.
- `job-package-conpty`
    - Takes a list of build configurations and platforms.
    - Consumes the `build-` artifact for every config/platform
    - Produces one `conpty-nupkg-CONFIG` for each configuration, merging
      all platforms.
    - Optionally signs the output and produces a bill of materials.
- `job-test-project`
    - Takes **one** build config and **one** platform.
    - Consumes `build-PLATFORM-CONFIG`
    - Selects its own pools (hardcoded) because it knows about
      architectures and must choose the right agent arch.
    - Runs tests (directly on the build agent).
- `job-run-pgo-tests`
    - Just like the above, but runs tests where `IsPgo` is `true`
    - Collects all of the PGO counts and publishes a `pgc-intermediates`
      artifact for that platform and configuration.
- `job-pgo-merge-pgd`
    - Takes **one** build config and multiple platforms.
    - Consumes `build-$platform-CONFIG` for each platform.
    - Consumes `pgc-intermediates-$platform-CONFIG` for each platform.
    - Merges the `pgc` files into `pgd` files
    - Produces a new `pgd-` artifact.
- `job-pgo-build-nuget-and-publish`
    - Consumes the `pgd-` artifact from above.
    - Packs it into a `nupkg` and publishes it.
- `job-submit-windows-vpack`
    - Only expected to run against `Release`.
    - Consumes the `appxbundle-CONFIG` artifact.
    - Publishes it to a vpack for Windows to consume.
- `job-check-code-format`
    - Does not use artifacts. Runs `clang-format`.
- `job-index-github-codenav`
    - Does not use artifacts.

Fuzz submission is broken due to changes in the `onefuzz` client.

I have removed the compliance and security build because it is no longer
supported.

Finally, this pull request has some additional benefits:

- I've expanded the PGO build phase to cover ARM64!
- We can remove everything Helix-related except the WTT parser
    - We no longer depend on Helix submission or Helix pools
- The WPF control's inner DLLs are now codesigned (#15404)
- Symbols for the WPF control, both .NET and C++, are published
  alongside all other symbols.
- The files we submit to ESRP for signing are batched up into a single
  step[^1]

Closes #11874
Closes #11974
Closes #15404

[^1]: This will have to change if we want to sign the individual
per-architecture `.appx` files before bundling so that they can be
directly installed.

(cherry picked from commit 69eff7e)
Service-Card-Id: 90183387
Service-Version: 1.17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area-Build Issues pertaining to the build system, CI, infrastructure, meta In-PR This issue has a related PR Issue-Task It's a feature request, but it doesn't really need a major design. Needs-Tag-Fix Doesn't match tag requirements Product-Meta The product is the management of the products.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants