Skip to content

Commit db9597d

Browse files
anamnaviTravisEz13
anamnavi
and
TravisEz13
authored
Migrate Build pipeline to OneBranch (#810)
* Generated Yaml Pipeline * Generated Yaml Pipeline * rename yml file to what OneBranch set up expects * revert renaming so it matches GH master naming * delete unneeded files and rename mariner2_arm64 to mariner2-arm64 to master GH naming * use update stableReleaseStage.yml that doesnt include mariner2 test-deps image and includes hostVersion value for windowsserver*2022 * fix reference to channelPath variable * use dependencies syntax * in releaseStage.yml refer to prior stage output variable * revert dependency reference to channelPath variable, and add template call for preview and lts releaseStage * comment out preview and lts releaseStage template calls for now * add back preview release stage template call * attempt to move most inline pwsh script into buildHelper module * set var with empty string differently * actually set var with empty string differently * use correct step name * call templates for each channel * remove variable and potentially unneeded Az install step * reduce ipmo lines from 2 to 1 * Revert "reduce ipmo lines from 2 to 1" This reverts commit c9c4c89. * Revert "remove variable and potentially unneeded Az install step" This reverts commit ab84d6e. * Revert "call templates for each channel" This reverts commit 5fc222f. * Revert "use correct step name" This reverts commit ff8bf91. * Revert "actually set var with empty string differently" This reverts commit 4a6c532. * Revert "set var with empty string differently" This reverts commit a6e7db7. * Revert "attempt to move most inline pwsh script into buildHelper module" This reverts commit 3196f1b. * rename releasePhase.yml to releaseJob.yml * conditionally call the channel based template * fix parameter names and clean up setVersionStepsTemplate.yml * fix typo * capture env * define variables * debug why dockernamespace is null * debug why releaseTag is null * Revert "debug why releaseTag is null" This reverts commit a655e6c. * Revert "debug why dockernamespace is null" This reverts commit 97df6ca. * Revert "define variables" This reverts commit 3da7750. * Revert "capture env" This reverts commit 33756f8. * Revert "fix typo" This reverts commit 9db45ac. * Revert "fix parameter names and clean up setVersionStepsTemplate.yml" This reverts commit e924c5e. * Revert "conditionally call the channel based template" This reverts commit e0b6422. * try conditionally calling stableReleaseStage.yml template * see if conditionally setting all channel template calls will bypass size issue * fix typo * rename releasePhase.yml to releaseJob.yml in preview and lts ymls * change param to be for generic version and accept channel param * Fix variable name referred to in setVersionStepsTemplate.yml * add conditional template call for preview and lts * specify hostVersion param for other windows amd64 image * Remove changes to GenerateMatrixJson that were not needed * remove newline added * remove comments * remove releaseStage.yml which is not used anymore * Update .pipelines/PowerShell-Docker-Image-Build-Official.yml Co-authored-by: Travis Plunk <travis.plunk@microsoft.com> * Update .vsts-ci/manifestSteps.yml * Update .vsts-ci/templatesGeneric/setVersionStepsTemplate.yml --------- Co-authored-by: EasyStart-Prod <> Co-authored-by: Travis Plunk <travis.plunk@microsoft.com>
1 parent c2f97e2 commit db9597d

File tree

10 files changed

+526
-541
lines changed

10 files changed

+526
-541
lines changed

‎.config/tsaoptions.json‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
{
2+
"instanceUrl": "https://msazure.visualstudio.com",
3+
"projectName": "One",
4+
"areaPath": "One\\MGMT\\Compute\\Powershell\\Powershell\\PowerShell Core",
5+
"notificationAliases": [
6+
"annavied@microsoft.com",
7+
"tplunk@microsoft.com"
8+
]
9+
}

‎.pipelines/PowerShell-Docker-Image-Build-Official.yml‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
name: PSDocker-Build-$(Build.BuildId)-version_$(releaseVersionTag)_channel_$(releaseChannel)$(Rev:_rr)
2+
trigger: none
3+
parameters:
4+
- name: 'releaseVersionTag'
5+
default: 'v7.4.0-preview.5'
6+
- name: 'releaseChannel'
7+
default: 'preview'
8+
variables:
9+
- name: POWERSHELL_TELEMETRY_OPTOUT
10+
value: 1
11+
- name: DOCKER_RELEASE
12+
value: true
13+
- name: releaseVersionTag
14+
value: ${{ parameters.releaseVersionTag }}
15+
- name: releaseChannel
16+
value: ${{ parameters.releaseChannel }}
17+
- name: releaseChannelPath
18+
value: ''
19+
- name: runCodesignValidationInjection
20+
value: false
21+
- name: DisableDockerDetector
22+
value: true
23+
- name: LinuxContainerImage
24+
value: mcr.microsoft.com/onebranch/cbl-mariner/build:2.0
25+
- name: WindowsContainerImage
26+
value: onebranch.azurecr.io/windows/ltsc2019/vse2022:latest
27+
- name: ob_sdl_sbom_enabled
28+
value: true
29+
- name: ob_signing_setup_enabled
30+
value: true
31+
#CodeQL tasks added manually to workaround signing failures
32+
- name: ob_sdl_codeql_compiled_enabled
33+
value: false
34+
resources:
35+
repositories:
36+
- repository: onebranchTemplates
37+
type: git
38+
name: OneBranch.Pipelines/GovernedTemplates
39+
ref: refs/heads/main
40+
extends:
41+
template: v2/OneBranch.Official.CrossPlat.yml@onebranchTemplates
42+
parameters:
43+
featureFlags:
44+
runHostSDL: true
45+
customTags: 'ES365AIMigrationTooling-BulkMigrated'
46+
stages:
47+
- stage: StageResolveVersionandYaml
48+
displayName: Resolve Versions and Populate Channel Based Yamls
49+
jobs:
50+
- job: JobResolveVersions
51+
displayName: Resolve Versions
52+
condition: succeededOrFailed()
53+
variables:
54+
- name: ob_outputDirectory
55+
value: '$(Build.ArtifactStagingDirectory)/ONEBRANCH_ARTIFACT'
56+
- name: repoRoot
57+
value: $(Build.SourcesDirectory)
58+
- name: ob_sdl_tsa_configFile
59+
value: $(Build.SourcesDirectory)\.config\tsaoptions.json
60+
- name: signSrcPath
61+
value: $(repoRoot)/out
62+
- name: ob_sdl_sbom_enabled
63+
value: false
64+
- name: ob_signing_setup_enabled
65+
value: true
66+
#CodeQL tasks added manually to workaround signing failures
67+
- name: ob_sdl_codeql_compiled_enabled
68+
value: false
69+
pool:
70+
timeoutInMinutes: 30
71+
type: windows
72+
steps:
73+
- pwsh: |
74+
if (($env:RELEASECHANNEL -eq "stable") -or ($env:RELEASECHANNEL -eq "lts"))
75+
{
76+
if ($env:RELEASEVERSIONTAG -notmatch '^v\d+\.\d+\.\d+$') {
77+
throw "${$env:RELEASECHANNEL} release tag is not for a ${$env:RELEASECHANNEL} build: '${$env:RELEASEVERSIONTAG}'"
78+
}
79+
}
80+
elseif ($env:RELEASECHANNEL -eq "preview")
81+
{
82+
if ($env:RELEASEVERSIONTAG -notmatch '^v\d+\.\d+\.\d+-(preview|rc)\.\d+$') {
83+
throw "${$env:RELEASECHANNEL} release tag is not for a ${$env:RELEASECHANNEL} build: '${$env:RELEASEVERSIONTAG}'"
84+
}
85+
}
86+
else
87+
{
88+
throw "channel ${$env:RELEASECHANNEL} is not a supported channel"
89+
}
90+
displayName: stop build if release tag does not match release tag format for channel
91+
- pwsh: |
92+
$toolsFolderPath = Join-Path -Path $(repoRoot) -ChildPath 'tools'
93+
$buildHelperFolderPath = Join-Path -Path $toolsFolderPath -ChildPath 'buildHelper'
94+
$buildHelperModulePath = Join-Path -Path $buildHelperFolderPath -ChildPath 'buildHelper.psm1'
95+
Import-Module $buildHelperModulePath
96+
$channelsInfo = Get-ChannelData
97+
if ($channelsInfo)
98+
{
99+
foreach ($channelEntry in $channelsInfo)
100+
{
101+
Write-Verbose -Verbose "channel path is: $($channelEntry.Path)"
102+
if ($($channelEntry.Name) -eq "stable")
103+
{
104+
$stablePath = "$($channelEntry.Path)"
105+
Write-Host "##vso[task.setvariable variable=stableChannelPath;isOutput=true]$stablePath"
106+
}
107+
elseif ($($channelEntry.Name) -eq "preview")
108+
{
109+
$previewPath = $($channelEntry.Path)
110+
Write-Host "##vso[task.setvariable variable=previewChannelPath;isOutput=true]$previewPath"
111+
}
112+
elseif ($($channelEntry.Name) -eq "lts")
113+
{
114+
$ltsPath = $($channelEntry.Path)
115+
Write-Host "##vso[task.setvariable variable=ltsChannelPath;isOutput=true]$ltsPath"
116+
}
117+
}
118+
}
119+
displayName: 'Get Channel path info'
120+
name: setChannelPathInfo
121+
- pwsh: |
122+
Get-ChildItem env: | Out-String -Width 1000
123+
displayName: 'Capture Environment'
124+
- ${{ if eq(parameters.releaseChannel, 'stable') }}:
125+
- template: /.vsts-ci/stableReleaseStage.yml@self
126+
parameters:
127+
channel: ${{ parameters.releaseChannel }}
128+
channelPath: 'release/7-4'
129+
- ${{ if eq(parameters.releaseChannel, 'preview') }}:
130+
- template: /.vsts-ci/previewReleaseStage.yml@self
131+
parameters:
132+
channel: ${{ parameters.releaseChannel }}
133+
channelPath: 'release/7-5'
134+
- ${{ if eq(parameters.releaseChannel, 'lts') }}:
135+
- template: /.vsts-ci/ltsReleaseStage.yml@self
136+
parameters:
137+
channel: ${{ parameters.releaseChannel }}
138+
channelPath: 'release/7-2'

‎.vsts-ci/manifestSteps.yml‎

Lines changed: 26 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -1,39 +1,26 @@
1-
steps:
2-
- powershell: |
3-
$namespace = '$(releaseTag)'.ToLowerInvariant()
4-
Write-Host "##vso[task.setvariable variable=dockerNamespace;]$namespace"
5-
displayName: 'Set dockerNamespace'
6-
7-
- powershell: 'Get-ChildItem env:'
8-
displayName: 'Capture Environment'
9-
10-
- pwsh: |
11-
$PSVersionTable
12-
displayName: Capture pwsh Version
13-
14-
- powershell: |
15-
$dockerConfigFolder = "$env:userprofile/.docker"
16-
if(!(Test-Path $dockerConfigFolder)){ $null = new-item -Type Directory -Path $dockerConfigFolder}
17-
$dockerCliConfig = "$env:userprofile/.docker/config.json"
18-
$dockerCliBackup = "$env:userprofile/.docker/config-backup.json"
19-
if(Test-Path $dockerCliConfig) { copy-item $dockerCliConfig $dockerCliBackup -force}
20-
@{experimental='enabled'}|ConvertTo-Json | Out-File -Encoding ascii -FilePath $dockerCliConfig
21-
22-
displayName: 'enable docker cli experimental features'
23-
24-
- powershell: 'docker login $(dockerHost) -u $(dockerUserName) -p $(dockerKey)'
25-
displayName: 'docker login'
26-
27-
- pwsh: |
28-
./createAllManifests.ps1 -Registry '$(dockerHost)/public/$(dockerNamespace)' -Channel $(Channel)
29-
displayName: 'Create $(Channel) Manifest Lists'
30-
31-
- powershell: 'docker logout $(dockerHost)'
32-
displayName: 'docker logout'
33-
condition: always()
34-
35-
- task: ms.vss-governance-buildtask.governance-build-task-component-detection.ComponentGovernanceComponentDetection@0
36-
displayName: 'Component Detection'
37-
inputs:
38-
sourceScanPath: '$(Build.SourcesDirectory)'
39-
snapshotForceEnabled: true
1+
steps:
2+
- pwsh: |
3+
$namespace = '$(releaseTag)'.ToLowerInvariant()
4+
Write-Host "##vso[task.setvariable variable=dockerNamespace;]$namespace"
5+
displayName: 'Set dockerNamespace'
6+
- pwsh: 'Get-ChildItem env:'
7+
displayName: 'Capture Environment'
8+
- pwsh: |
9+
$PSVersionTable
10+
displayName: Capture pwsh Version
11+
- pwsh: |
12+
$dockerConfigFolder = "$env:userprofile/.docker"
13+
if(!(Test-Path $dockerConfigFolder)){ $null = new-item -Type Directory -Path $dockerConfigFolder}
14+
$dockerCliConfig = "$env:userprofile/.docker/config.json"
15+
$dockerCliBackup = "$env:userprofile/.docker/config-backup.json"
16+
if(Test-Path $dockerCliConfig) { copy-item $dockerCliConfig $dockerCliBackup -force}
17+
@{experimental='enabled'}|ConvertTo-Json | Out-File -Encoding ascii -FilePath $dockerCliConfig
18+
displayName: 'enable docker cli experimental features'
19+
- pwsh: 'docker login $(dockerHost) -u $(dockerUserName) -p $(dockerKey)'
20+
displayName: 'docker login'
21+
- pwsh: |
22+
./createAllManifests.ps1 -Registry '$(dockerHost)/public/$(dockerNamespace)' -Channel $(Channel)
23+
displayName: 'Create $(Channel) Manifest Lists'
24+
- pwsh: 'docker logout $(dockerHost)'
25+
displayName: 'docker logout'
26+
condition: always()

‎.vsts-ci/releaseJob.yml‎

Lines changed: 178 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
parameters:
2+
- name: archName
3+
default: ''
4+
- name: imageName
5+
default: 'none'
6+
- name: artifactSuffix
7+
default: ''
8+
- name: ACR
9+
default: 'No'
10+
- name: maxParallel
11+
default: 5
12+
- name: poolOS
13+
default: windows
14+
- name: poolHostArchitecture
15+
default: ''
16+
- name: poolHostVersion
17+
default: ''
18+
- name: channelPath
19+
default: ''
20+
- name: buildKitValue
21+
default: 0
22+
- name: channel
23+
default: 'preview'
24+
- name: releaseTag
25+
default: ''
26+
- name: windowsContainerImageValue
27+
default: ''
28+
jobs:
29+
- job: Job_PreBuild_${{ parameters.archName }}_${{ parameters.artifactSuffix }}
30+
variables:
31+
- name: dockerImage
32+
value: 'powershell'
33+
- name: releaseTag
34+
value: ${{ parameters.releaseTag }}
35+
- name: scanType
36+
value: Register
37+
- group: 'Azure Blob variable group'
38+
- group: Build Docker Group
39+
- group: Staging_ACR
40+
- name: repoRoot
41+
value: $(Build.SourcesDirectory)
42+
- name: ob_outputDirectory
43+
value: '$(Build.ArtifactStagingDirectory)/ONEBRANCH_ARTIFACT'
44+
- name: ob_git_checkout
45+
value: true
46+
pool:
47+
type: linux
48+
timeoutInMinutes: 135
49+
displayName: 'Prep data to be used for docker build of ${{ parameters.artifactSuffix }}'
50+
steps:
51+
- task: UseDotNet@2
52+
inputs:
53+
packageType: 'sdk'
54+
version: 7.x
55+
- pwsh: |
56+
dotnet --list-sdks
57+
displayName: dotnet info
58+
- pwsh: |
59+
Write-Host "##vso[task.setvariable variable=acrName;isOutput=true]$env:ACR_NAME_VAR"
60+
displayName: 'Set ACR name variable'
61+
name: setACRVariableStep
62+
- pwsh: |
63+
Write-Host "##vso[task.setvariable variable=channel;]${{ parameters.channel }}"
64+
displayName: 'Set channel variable'
65+
- template: /.vsts-ci/templatesGeneric/setVersionStepsTemplate.yml@self
66+
- pwsh: |
67+
$finalImageName = '${{ parameters.imageName }}'
68+
$finalContextFolder = 'docker'
69+
$testDepsString = 'test-deps'
70+
if ($finalImageName.Contains($testDepsString))
71+
{
72+
$finalImageName = $finalImageName.TrimEnd("\$testDepsString")
73+
$finalContextFolder = Join-Path -Path $testDepsString -ChildPath $finalContextFolder
74+
}
75+
$pathToContextFolder = "${{ parameters.channelPath }}/$finalImageName/$finalContextFolder"
76+
Write-Host "##vso[task.setvariable variable=dockerBuildImageName;isOutput=true]$finalImageName"
77+
Write-Host "##vso[task.setvariable variable=dockerBuildContextFolder;isOutput=true]$finalContextFolder"
78+
displayName: Get Proper ImageName and ContextFolder
79+
name: setImageNameStep
80+
- pwsh: |
81+
$modules = 'Az.Accounts', 'Az.Storage'
82+
foreach ($module in $modules) {
83+
if (!(Get-Module $module -listavailable)) {
84+
Write-Host "installing $module..." -verbose
85+
Install-Module $module -force -allowclobber
86+
} else {
87+
Write-Host "$module is already installed" -verbose
88+
}
89+
}
90+
displayName: 'Install AzAccounts'
91+
- task: AzurePowerShell@5
92+
inputs:
93+
azureSubscription: az-blob-cicd-infra
94+
scriptType: inlineScript
95+
azurePowerShellVersion: latestVersion
96+
pwsh: true
97+
inline: |
98+
$containerName = Out-String -InputObject $(DOCKERNAMESPACE) -NoNewline
99+
$containerName = $containerName.Replace('.', '-')
100+
$context = New-AzStorageContext -StorageAccountName $(StorageAccount)
101+
$querystring = New-AzStorageContainerSASToken -Context $context -Name $containerName -Permission r -ExpiryTime ((get-date).AddMinutes(180)) -Protocol HttpsOnly
102+
$uriBuilder = [System.UriBuilder]::new("https://$(StorageAccount).blob.core.windows.net")
103+
$uriBuilder.Query = $querystring.ToString()
104+
$url = $uriBuilder.ToString()
105+
Write-Host "##vso[task.setvariable variable=SasUrl;]$url"
106+
displayName: 'Set SasUrl variable'
107+
- pwsh: |
108+
Get-ChildItem env: | Out-String -Width 1000
109+
displayName: 'Capture Environment'
110+
- pwsh: |
111+
if($env:SASURL)
112+
{
113+
Write-Host 'Using SasUrl...'
114+
$buildArgsString = ./build.ps1 -SasUrl $env:SASURL -ImageName $(dockerHost) -name '$(setImageNameStep.dockerBuildImageName)' -Channel ${{ parameters.channel }} -TestLogPostfix '${{ parameters.imageName }}-${{ parameters.channel }}' -version '$(Version)' -Repository $(dockerNamespace)/$(dockerImage)
115+
Write-Host "##vso[task.setvariable variable=sasBuildArgs;isOutput=true]$buildArgsString"
116+
}
117+
displayName: 'Set build args from SAS token'
118+
name: setSASVariableStep
119+
- task: CopyFiles@2
120+
inputs:
121+
SourceFolder: $(repoRoot)/${{ parameters.channelPath }}/$(setImageNameStep.dockerBuildImageName)/$(setImageNameStep.dockerBuildContextFolder)
122+
TargetFolder: $(ob_outputDirectory)/$(setImageNameStep.dockerBuildImageName)
123+
displayName: "Copy Dockerfile to ob_outputDirectory/ONEBRANCH_ARTIFACT - '$(ob_outputDirectory)'"
124+
- job: Job_Build_${{ parameters.archName }}_${{ parameters.artifactSuffix }}
125+
dependsOn:
126+
- Job_PreBuild_${{ parameters.archName }}_${{ parameters.artifactSuffix }}
127+
variables:
128+
- name: ob_outputDirectory # this directory is uploaded to pipeline artifacts
129+
value: '$(Build.SourcesDirectory)/dst'
130+
- name: ob_git_checkout
131+
value: true
132+
- name: OB_build_container
133+
value: true # Set this variable to enable Mixed Mode
134+
- ${{ if ne(parameters.windowsContainerImageValue, '')}}:
135+
- name: WindowsContainerImage
136+
value: ${{ parameters.windowsContainerImageValue }}
137+
- name: dockerBuildImageName
138+
value: $[ dependencies.Job_PreBuild_${{ parameters.archName }}_${{ parameters.artifactSuffix }}.outputs['setImageNameStep.dockerBuildImageName'] ]
139+
- name: ACRName
140+
value: $[ dependencies.Job_PreBuild_${{ parameters.archName }}_${{ parameters.artifactSuffix }}.outputs['setACRVariableStep.acrName'] ]
141+
- name: SasBuildArgs
142+
value: $[ dependencies.Job_PreBuild_${{ parameters.archName }}_${{ parameters.artifactSuffix }}.outputs['setSASVariableStep.sasBuildArgs'] ]
143+
- name: HostVersionValue
144+
value: $[ dependencies.Job_PreBuild_${{ parameters.archName }}_${{ parameters.artifactSuffix }}.outputs['setImageNameStep.hostVersion'] ]
145+
pool:
146+
type: docker
147+
os: ${{ parameters.poolOS }}
148+
${{ if ne(parameters.poolHostArchitecture, '') }}:
149+
hostArchitecture: ${{ parameters.poolHostArchitecture }}
150+
${{ if ne(parameters.poolHostVersion, '') }}:
151+
hostVersion: ${{ parameters.poolHostVersion }}
152+
steps:
153+
- task: DownloadPipelineArtifact@2
154+
inputs:
155+
targetPath: $(Build.SourcesDirectory)/dst/$(dockerBuildImageName)
156+
artifact: drop_StageGenerateBuild_${{ parameters.channel }}_Job_PreBuild_${{ parameters.archName }}_${{ parameters.artifactSuffix }}
157+
displayName: 'Download artifact to which Dockerfile was uploaded'
158+
- task: onebranch.pipeline.containercontrol@1
159+
displayName: "Login to source ACR"
160+
inputs:
161+
command: login
162+
acr_name: $(ACRName)
163+
tenant: MSFT
164+
- task: onebranch.pipeline.imagebuildinfo@1
165+
displayName: image build
166+
inputs:
167+
dockerFileRelPath: $(dockerBuildImageName)/$(dockerBuildImageName)/Dockerfile
168+
build_tag: $(Build.BuildNumber)
169+
repositoryName: $(dockerBuildImageName)
170+
saveImageToPath: $(dockerBuildImageName).tar
171+
arguments: $(SasBuildArgs)
172+
registry: $(ACRName)
173+
buildkit: ${{ parameters.buildKitValue }}
174+
enable_network: true
175+
enable_cache: true
176+
enable_pull: true
177+
178+

0 commit comments

Comments
 (0)