Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create OneBranch Pipeline to build and release snap packages #107

Merged
merged 34 commits into from
Apr 22, 2024
Merged
Show file tree
Hide file tree
Changes from 33 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
5ab4bd4
Adding OneBranch pipeline YAML config file for PowerShell-Snap-Official
Apr 20, 2024
f1eef2a
Port build to OneBranch
TravisEz13 Apr 20, 2024
f5e4cc1
port approval as is
TravisEz13 Apr 20, 2024
2e51f93
fix pool type
TravisEz13 Apr 20, 2024
5e741b8
fix approval pool
TravisEz13 Apr 20, 2024
e0ff0c8
remove unused checkout
TravisEz13 Apr 20, 2024
6673454
setup SDL
TravisEz13 Apr 20, 2024
51519a6
setup snapd
TravisEz13 Apr 20, 2024
fc8c73b
fix steps template
TravisEz13 Apr 20, 2024
2d8c46a
use vm because snap won't run otherwise
TravisEz13 Apr 20, 2024
e9255e5
use home env to get logs
TravisEz13 Apr 20, 2024
5e34d4d
No lfs
TravisEz13 Apr 20, 2024
ded5aec
fix syntax
TravisEz13 Apr 20, 2024
2daed7d
use OBP params to disable lfs
TravisEz13 Apr 20, 2024
5ed1f65
create output dir
TravisEz13 Apr 20, 2024
f0ed457
Use lfs setting in checkout because OBP params doesn't work due to cu…
TravisEz13 Apr 20, 2024
c201c8c
Fix artifact upload for custom job
TravisEz13 Apr 20, 2024
e18ea23
move mount out of approval
TravisEz13 Apr 20, 2024
4724d77
oops
TravisEz13 Apr 20, 2024
3108a27
fix scan stage display name
TravisEz13 Apr 20, 2024
c567775
switch to mariner when using container
TravisEz13 Apr 20, 2024
302b1df
misc fixes
TravisEz13 Apr 20, 2024
5af9736
fix scan download dir
TravisEz13 Apr 20, 2024
1b63680
unsquash during build
TravisEz13 Apr 20, 2024
ada144c
publish output dir
TravisEz13 Apr 20, 2024
b965c99
make it a string
TravisEz13 Apr 20, 2024
8fdd358
complete the scan job
TravisEz13 Apr 20, 2024
3aa52be
fix tar extraction
TravisEz13 Apr 21, 2024
f910864
fix variable set statement
TravisEz13 Apr 21, 2024
8892355
remove manual approval
TravisEz13 Apr 22, 2024
b9fab1d
Update tsaoptions.json
TravisEz13 Apr 22, 2024
8c6b76d
Update .pipelines/PowerShell-Snap-Official.yml
TravisEz13 Apr 22, 2024
cbf70bb
Apply suggestions from code review
TravisEz13 Apr 22, 2024
3f1adce
Update .pipelines/templates/pushJob.yml
TravisEz13 Apr 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
11 changes: 11 additions & 0 deletions .config/tsaoptions.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"instanceUrl": "https://msazure.visualstudio.com",
"projectName": "One",
"areaPath": "One\\MGMT\\Compute\\Powershell\\Powershell\\PowerShell Core",
"notificationAliases": [
"adityap@microsoft.com",
"dongbow@microsoft.com",
"pmeinecke@microsoft.com",
"tplunk@microsoft.com"
]
}
97 changes: 97 additions & 0 deletions .pipelines/PowerShell-Snap-Official.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@

parameters:
- name: release
type: string
displayName: |
Use `stable` to release by default.

Use `private` if you want to create a branch on the store to test the package.
This will create a branch under the edge branch automatically that are difficult, but not impossible to find.

`candidate`, `beta`, and `edge` are public but we don't have any existing meaning for these channels.
values:
- private
- stable
- candidate
- beta
- edge
default: stable

trigger: none

variables:
- name: CDP_DEFINITION_BUILD_COUNT
value: $[counter('', 0)] # needed for onebranch.pipeline.version task
- name: LinuxContainerImage
value: mcr.microsoft.com/onebranch/cbl-mariner/build:2.0 # Docker image which is used to build the project
- name: DEBIAN_FRONTEND
value: noninteractive
- group: poolNames

resources:
repositories:
- repository: templates
type: git
name: OneBranch.Pipelines/GovernedTemplates
ref: refs/heads/main

extends:
template: v2/OneBranch.Official.CrossPlat.yml@templates
parameters:
git:
fetchDepth: 1
# windows only feature
#longpaths: true
retryCount: 3
# we don't use this and some of our agents doesn't have the feature installed
lfs: false
cloudvault:
enabled: false # set to true to enable cloudvault
runmode: stage # linux can run CloudVault upload as a separate stage
dependsOn: linux_build
artifacts:
- drop_linux_stage_linux_job
customTags: 'ES365AIMigrationTooling'
globalSdl:
disableLegacyManifest: true
# disabled Armorty as we dont have any ARM templates to scan. It fails on some sample ARM templates.
armory:
enabled: false
sbom:
enabled: true
compiled:
enabled: false
credscan:
enabled: true
scanFolder: $(Build.SourcesDirectory)
cg:
enabled: true
asyncSdl: # https://aka.ms/obpipelines/asyncsdl
enabled: true
forStages: [scan_lts, scan_stable, scan_preview]
credscan:
enabled: true
scanFolder: $(Build.SourcesDirectory)
#suppressionsFile: $(Build.SourcesDirectory)\PowerShell\.config\suppress.json
binskim:
enabled: false
# APIScan requires a non-Ready-To-Run build
apiscan:
enabled: false
tsaOptionsFile: .config\tsaoptions.json

stages:
- template: /.pipelines/templates/releaseBuildAndPushStage.yml@self
parameters:
channel: stable
release: ${{ parameters.release }}

- template: /.pipelines/templates/releaseBuildAndPushStage.yml@self
parameters:
channel: preview
release: ${{ parameters.release }}

- template: /.pipelines/templates/releaseBuildAndPushStage.yml@self
parameters:
channel: lts
release: ${{ parameters.release }}
31 changes: 31 additions & 0 deletions .pipelines/templates/Approval.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
parameters:
- name: displayName
type: string
- name: instructions
type: string
- name: jobName
type: string
- name: timeoutInMinutes
type: number
# 2 days
default: 2880
- name: onTimeout
type: string
default: 'reject'
values:
- resume
- reject

jobs:
- job: ${{ parameters.jobName }}
displayName: ${{ parameters.displayName }}
pool:
type: agentless
timeoutInMinutes: 4320 # job times out in 3 days
steps:
- task: ManualValidation@0
displayName: ${{ parameters.displayName }}
timeoutInMinutes: ${{ parameters.timeoutInMinutes }}
inputs:
instructions: ${{ parameters.instructions }}
onTimeout: ${{ parameters.onTimeout }}
11 changes: 11 additions & 0 deletions .pipelines/templates/InstallSnapd.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
steps:

- bash: |
sudo apt update
displayName: Apt Update
condition: succeeded()

- bash: |
sudo apt-get -y install snapd
displayName: Install snapd
condition: succeeded()
6 changes: 6 additions & 0 deletions .pipelines/templates/createOutputDirectory-linux.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
steps:

- bash: |
mkdir -p -m a=rwx $(ob_outputDirectory)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we check if the folder exists before mkdir

displayName: Create $(ob_outputDirectory)
condition: succeeded()
110 changes: 110 additions & 0 deletions .pipelines/templates/pushJob.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
parameters:
- name: channel
type: string
- name: release
default: 'private'

jobs:
- job: push
displayName: Push to ${{ parameters.release }}

pool:
type: linux
isCustom: true
name: $(ubuntuPool)
demands:
- ImageOverride -equals PSMMSUbuntu20.04-Secure

variables:
- name: ReleaseTag
value: edge/default
- group: SnapLogin
- name: channel
value: ${{ parameters.channel }}
- name: ob_outputDirectory
value: '$(Build.ArtifactStagingDirectory)/out'
- name: DOWNLOAD_DIRECTORY
value: '$(Build.ArtifactStagingDirectory)/down'

steps:
- checkout: self
lfs: false

- template: ./createOutputDirectory-linux.yml

- task: DownloadPipelineArtifact@2
displayName: 'Download build files'
inputs:
targetPath: $(DOWNLOAD_DIRECTORY)
artifact: drop_build_$(channel)_build_$(channel)

- pwsh: |
$snaps = Get-ChildItem $(DOWNLOAD_DIRECTORY)/*.snap -recurse -File
if($snaps.Count -gt 1)
{
$snaps | out-string -width 100 | Write-Verbose -verbose
Write-Error "***More than one snap found***" -errorAction stop
}
displayName: Verify that only one snap was downloaded

- pwsh: |
[string]$Branch=$env:BUILD_SOURCEBRANCH
$branchOnly = $Branch -replace '^refs/heads/';
$branchOnly = $branchOnly -replace '[_\-]'

if('${{ parameters.release }}' -eq 'private') {
if($branchOnly -eq 'master' -or $branchOnly -like '*dailytest*')
{
Write-verbose "release branch: $branchOnly" -verbose
$generatedBranch = ([guid]::NewGuid()).ToString().Replace('-','')
$releaseTag = "edge/$generatedBranch"
}
else
{
Write-verbose "non-release branch" -verbose
# Branch is named <previewname>
$releaseTag = "edge/$branchOnly"
$releaseTag += ([guid]::NewGuid()).ToString().Replace('-','')
}
} else {
TravisEz13 marked this conversation as resolved.
Show resolved Hide resolved
$releaseTag = "${{ parameters.release }}"
}

$vstsCommandString = "vso[task.setvariable variable=ReleaseTag]$releaseTag"
Write-Verbose -Message "setting ReleaseTag to $releaseTag" -Verbose
Write-Host -Object "##$vstsCommandString"
displayName: Set ReleaseTag Variable

- pwsh: |
sudo chown root:root /
displayName: 'Make sure root owns root'
condition: succeeded()

- template: ./InstallSnapd.yml

- pwsh: |
$channel = (Get-Content ./snapcraftConfig.json | ConvertFrom-Json).channel
Write-Verbose -Verbose -Message "using Channel $channel"
sudo snap install snapcraft --classic "--channel=$channel"
condition: succeeded()
displayName: 'Install snapcraft'
retryCountOnTaskFailure: 2

- pwsh: |
$track = 'latest'
if('$(channel)' -eq 'lts')
{
$track = 'lts'
}

$snaps = Get-ChildItem $(System.ArtifactsDirectory)/*.snap -recurse -File | Select-Object -ExpandProperty FullName

foreach($snap in $snaps)
{
Write-Verbose -Verbose -Message "Uploading $snap to $track/$(ReleaseTag)"
snapcraft upload --release "$track/$(ReleaseTag)" $snap
}
displayName: 'snapcraft upload'
retryCountOnTaskFailure: 1
env:
SNAPCRAFT_STORE_CREDENTIALS: $(SNAPCRAFT_STORE_CREDENTIALS)
31 changes: 31 additions & 0 deletions .pipelines/templates/releaseBuildAndPushStage.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
parameters:
- name: channel
default: ''
- name: release
default: 'private'

stages:
- stage: build_${{ parameters.channel }}
displayName: Build - ${{ parameters.channel }}
dependsOn: []
jobs:
- template: ./releaseBuildJob.yml
parameters:
channel: ${{ parameters.channel }}

- stage: scan_${{ parameters.channel }}
displayName: SDL - ${{ parameters.channel }}
dependsOn: build_${{ parameters.channel }}
jobs:
- template: ./scanJob.yml
parameters:
channel: ${{ parameters.channel }}

- stage: Push_${{ parameters.channel }}
displayName: Push - ${{ parameters.channel }} to ${{ parameters.release }}
dependsOn: scan_${{ parameters.channel }}
jobs:
- template: ./pushJob.yml
parameters:
channel: ${{ parameters.channel }}
release: ${{ parameters.release }}