Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
2bcf451
Download manifest tool
michaelpeng36 May 2, 2023
faa2059
Remove packing step at the end of the build script
michaelpeng36 May 2, 2023
18cacf6
Add artifacts to publish
michaelpeng36 May 4, 2023
e8fe839
Debug Linux
michaelpeng36 May 4, 2023
04d7138
Temporarily disable suborchestrator E2E test
michaelpeng36 May 4, 2023
aeac99a
Log SimulateReleaseBuild value
michaelpeng36 May 4, 2023
1a76b35
Pass SimulateReleaseBuild variable to Linux pipeline
michaelpeng36 May 4, 2023
7ea6706
Remove test and update pipeline conditions
michaelpeng36 May 4, 2023
a709130
Alter tar command
michaelpeng36 May 4, 2023
bb25ffa
Correct typo
michaelpeng36 May 4, 2023
64939b7
Add package folder
michaelpeng36 May 4, 2023
b563d4e
Add sources directory and environment variables for Linux
michaelpeng36 May 4, 2023
b5abd7f
Correct path typo
michaelpeng36 May 4, 2023
cb595eb
Correct another path typo
michaelpeng36 May 4, 2023
21ee9fd
Update azure-pipelines.yml for Azure Pipelines
michaelpeng36 May 4, 2023
d565bef
Trying different path specifications
michaelpeng36 May 4, 2023
7edc03e
Debugging
michaelpeng36 May 5, 2023
7cc4519
Add release builds to Module subfolder
michaelpeng36 May 5, 2023
ceee24d
Correct artifacts directory
michaelpeng36 May 5, 2023
432eb0b
Generate SBOM
michaelpeng36 May 5, 2023
e4ded3c
Parse IsReleaseBuild as a bool
michaelpeng36 May 5, 2023
240d280
Initialize IsReleaseBuild
michaelpeng36 May 5, 2023
924ac3a
Correct typo in build script
michaelpeng36 May 5, 2023
9708851
Remove telemetry file
michaelpeng36 May 5, 2023
a26d4cb
Create telemetry file
michaelpeng36 May 5, 2023
c9fefab
Add manifestToolPath
michaelpeng36 May 5, 2023
11a16ac
Remove faulty file generation
michaelpeng36 May 5, 2023
d4fe3c8
Install .NET 3.1
michaelpeng36 May 5, 2023
22ce34f
Correct spacing issue
michaelpeng36 May 5, 2023
dbc033d
Add back suborchestrator test
michaelpeng36 May 5, 2023
03dc78a
Remove redundant field
michaelpeng36 May 5, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -21,5 +21,8 @@ local.settings.json
# Temporary output folder for Durable SDK binaries
**/src/out

# Package folder for .tar file
/package/**

# VS publish settings
*.pubxml
68 changes: 67 additions & 1 deletion azure-pipelines.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,78 @@ pool:
demands:
- ImageOverride -equals $(imageName)

variables:
artifactName: 'azure-functions-durable-powershell-$(Build.SourceVersion)'
# Every build will increment
buildNumber: $[counter('build', 001) ]
modulePath: './test/E2E/durableApp/Modules/AzureFunctions.PowerShell.Durable.SDK'

steps:
- pwsh: ./test/E2E/Start-E2ETest.ps1 -UseCoreToolsBuildFromIntegrationTests
- pwsh: |
$simulateReleaseBuild = $null
Write-Host "SimulateReleaseBuild set to $env:SimulateReleaseBuild"
if (-not([bool]::TryParse($env:SimulateReleaseBuild, [ref] $simulateReleaseBuild)))
{
throw "SimulateReleaseBuild can only be set to true or false."
}

$isReleaseBuild = $false
if ($env:BuildSourceBranchName -like "release_*" -or $simulateReleaseBuild)
{
$isReleaseBuild = $true
}
Write-Host "Setting IsReleaseBuild to $isReleaseBuild because SimulateReleaseBuild is $env:SimulateReleaseBuild"
Write-Host "##vso[task.setvariable variable=IsReleaseBuild]$isReleaseBuild"
Write-Host "IsReleaseBuild: $isReleaseBuild"
displayName: Set IsReleaseBuild pipeline variable
env:
SimulateReleaseBuild: $(SimulateReleaseBuild)

- pwsh: |
Import-Module ".\pipelineUtilities.psm1" -Force
Install-Dotnet
displayName: 'Install .NET 3.1'

- pwsh: |
Write-Host "IsReleaseBuild set to $env:IsReleaseBuild"
$isReleaseBuild = $false
Comment on lines +28 to +55
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems we're assigning isReleaseBuild twice? In the first pwsh block, we do it based on the branch name and the simulateReleaseBuild variable. On the last pwsh block, we do it based on env:IsReleaseBuild.

What would happen is SimulateReleaseBuild is true but env:IsReleaseBuild were false? Wouldn't force IsReleaseBuild to false?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not quite. The intention of SimulateReleaseBuild is precisely to force the pipeline to emulate behavior as if IsReleaseBuild were set to true. There should never be a case following the first pwsh block where the SimulateReleaseBuild is true but IsReleaseBuild is false.

For the second pwsh block, the reason IsReleaseBuild pipeline variable is propagated to this step as an environment variable is because Linux agents don't always respect using the $(variableName) notation directly in the pwsh script, and variable values are not carried over between steps in ADO by default. Instead, we set the pipeline variable in the first pwsh block with the Write-Host "##vso[task.setvariable variable=IsReleaseBuild]$isReleaseBuild" statement, and set an environment variable equal to the pipeline variable in the env block of the second pwsh block.

if (-not([bool]::TryParse($env:IsReleaseBuild, [ref] $isReleaseBuild)))
{
throw "SimulateReleaseBuild can only be set to true or false."
}

# We only generate an SBOM for release or simulated release builds
Write-Host "Running ./build.ps1 -Configuration Release -AddSBOM:$isReleaseBuild..."
./build.ps1 -Configuration Release -AddSBOM:$isReleaseBuild
displayName: 'Build Durable SDK'
env:
# We include IsReleaseBuild as an environment variable since Linux agents don't seem to support including
# pipeline variables in scripts with the $(variable) syntax
IsReleaseBuild: $(IsReleaseBuild)
SBOMUtilSASUrl: $(SBOMUtilSASUrl)

- pwsh: |
./test/E2E/Start-E2ETest.ps1 -NoBuild -UseCoreToolsBuildFromIntegrationTests
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we ever call this without NoBuild?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, so the Start-E2ETest.ps1 script is intended to be used for both local E2E test runs and in CI. The CI uses the -NoBuild flag since we have a distinct step dedicated to building the SDK, but it can be omitted for a local test run.

env:
AzureWebJobsStorage: $(AzureWebJobsStorage)
displayName: 'Run E2E tests'

- task: ArchiveFiles@2
inputs:
rootFolderOrFile: '$(Build.SourcesDirectory)/test/E2E/durableApp/Modules/AzureFunctions.PowerShell.Durable.SDK'
includeRootFolder: false
archiveType: 'tar'
archiveFile: '$(Build.ArtifactStagingDirectory)/$(artifactName).tar.gz'
replaceExistingArchive: true
displayName: 'Tar build tartifacts'

- task: PublishBuildArtifacts@1
inputs:
PathtoPublish: $(Build.ArtifactStagingDirectory)
ArtifactName: $(artifactName).tar.gz
condition: and(succeeded(), eq(variables['IsReleaseBuild'], 'true'))
displayName: 'Publish build artifacts'

- task: PublishTestResults@2
inputs:
testResultsFormat: 'VSTest'
Expand Down
70 changes: 31 additions & 39 deletions build.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -3,56 +3,29 @@
param(
[ValidateSet('Debug', 'Release')]
[string]
$Configuration = 'Debug'
$Configuration = 'Debug',
[switch]
$AddSBOM
)

Import-Module "$PSScriptRoot\pipelineUtilities.psm1" -Force

$packageName = "AzureFunctions.PowerShell.Durable.SDK"
$shimPath = "$PSScriptRoot/src/DurableSDK"
$durableEnginePath = "$PSScriptRoot/src/DurableEngine"
$durableAppPath = "$PSScriptRoot/test/E2E/durableApp/Modules/$packageName"
$powerShellModulePath = "$PSScriptRoot/src/$packageName.psm1"
$manifestPath = "$PSScriptRoot/src/$packageName.psd1"

$outputPath = "$PSScriptRoot/src/out/"
if ($Configuration -eq "Debug")
{
# Publish directly to the test durable app for testing
$outputPath = $durableAppPath
}
# Publish directly to the test durable app for testing
$outputPath = $durableAppPath

$sharedDependenciesPath = "$outputPath/Dependencies/"

$netCoreTFM = 'net6.0'
$publishPathSuffix = "bin/$Configuration/$netCoreTFM/publish"

function Write-Log
{
param (
[Parameter(Mandatory=$true)]
[ValidateNotNullOrEmpty()]
[System.String]
$Message,

[Switch]
$Warning,

[Switch]
$Throw,

[System.String]
$Color
)

$Message = (Get-Date -Format G) + " -- $Message"

if ($Throw)
{
throw $Message
}

$foregroundColor = if ($Warning.IsPresent) { 'Yellow' } elseif ($Color) { $Color } else { 'Green' }
Write-Host -ForegroundColor $foregroundColor $Message
}

#region BUILD ARTIFACTS ===========================================================================
Write-Log "Build started..."
Write-Log "Configuration: '$Configuration'`nOutput folder '$outputPath'`nShared dependencies folder: '$sharedDependenciesPath'" "Gray"

Expand Down Expand Up @@ -89,18 +62,37 @@ foreach ($project in $projects.GetEnumerator()) {
$commonFiles = [System.Collections.Generic.HashSet[string]]::new()

Write-Log "Copying assemblies from the Durable Engine project into $sharedDependenciesPath" "Gray"
Get-ChildItem -Path "$durableEnginePath/$publishPathSuffix" |
Get-ChildItem -Path (Join-Path "$durableEnginePath" "$publishPathSuffix") |
Where-Object { $_.Extension -in '.dll','.pdb' } |
ForEach-Object { [void]$commonFiles.Add($_.Name); Copy-Item -LiteralPath $_.FullName -Destination $sharedDependenciesPath }

# Copy all *unique* assemblies from Durable SDK into output directory
Write-Log "Copying unique assemblies from the Durable SDK project into $outputPath" "Gray"
Get-ChildItem -Path "$shimPath/$publishPathSuffix" |
Get-ChildItem -Path (Join-Path "$shimPath" "$publishPathSuffix") |
Where-Object { $_.Extension -in '.dll','.pdb' -and -not $commonFiles.Contains($_.Name) } |
ForEach-Object { Copy-Item -LiteralPath $_.FullName -Destination $outputPath }

# Move Durable SDK manifest into the output directory
Write-Log "Copying PowerShell module and manifest from the Durable SDK source code into $outputPath" "Gray"
Copy-Item -Path $powerShellModulePath -Destination $outputPath
Copy-Item -Path $manifestPath -Destination $outputPath
Write-Log "Build succeeded!"
Write-Log "Build succeeded!"
#endregion

#region ADD SBOM ==================================================================================
if ($AddSBOM) {
# Install manifest tool
$manifestToolPath = Install-SBOMUtil
Write-Log "Manifest tool path: $manifestToolPath"

# Generate manifest
$telemetryFilePath = Join-Path $PSScriptRoot ((New-Guid).Guid + ".json")
$packageName = "AzureFunctions.PowerShell.Durable.SDK"

Write-Log "Running: dotnet $manifestToolPath generate -BuildDropPath $outputPath -BuildComponentPath $outputPath -Verbosity Information -t $telemetryFilePath -PackageName $packageName"
dotnet $manifestToolPath generate -BuildDropPath $outputPath -BuildComponentPath $outputPath -Verbosity Information -t $telemetryFilePath -PackageName $packageName

# Discard telemetry generated
Remove-Item -Path $telemetryFilePath -ErrorAction Ignore
}
#endregion
128 changes: 128 additions & 0 deletions pipelineUtilities.psm1
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
#
# Copyright (c) Microsoft. All rights reserved.
# Licensed under the MIT license. See LICENSE file in the project root for full license information.
#

using namespace System.Runtime.InteropServices

$DotnetSDKVersionRequirements = @{

# .NET SDK 3.1 is required by the Microsoft.ManifestTool.dll tool
'3.1' = @{
MinimalPatch = '415'
DefaultPatch = '415'
}
}

function Write-Log
{
param (
[Parameter(Mandatory=$true)]
[ValidateNotNullOrEmpty()]
[System.String]
$Message,

[Switch]
$Warning,

[Switch]
$Throw,

[System.String]
$Color
)

$Message = (Get-Date -Format G) + " -- $Message"

if ($Throw)
{
throw $Message
}

$foregroundColor = if ($Warning.IsPresent) { 'Yellow' } elseif ($Color) { $Color } else { 'Green' }
Write-Host -ForegroundColor $foregroundColor $Message
}

function Install-SBOMUtil
{
if ([string]::IsNullOrEmpty($env:SBOMUtilSASUrl))
{
throw "The `$SBOMUtilSASUrl environment variable cannot be null or empty when specifying the `$AddSBOM switch"
}

$MANIFESTOOLNAME = "ManifestTool"
Write-Log "Installing $MANIFESTOOLNAME..."

$MANIFESTOOL_DIRECTORY = Join-Path $PSScriptRoot $MANIFESTOOLNAME
Remove-Item -Recurse -Force $MANIFESTOOL_DIRECTORY -ErrorAction Ignore

Invoke-RestMethod -Uri $env:SBOMUtilSASUrl -OutFile "$MANIFESTOOL_DIRECTORY.zip"
Expand-Archive "$MANIFESTOOL_DIRECTORY.zip" -DestinationPath $MANIFESTOOL_DIRECTORY

$dllName = "Microsoft.ManifestTool.dll"
$manifestToolPath = Join-Path "$MANIFESTOOL_DIRECTORY" "$dllName"

if (-not (Test-Path $manifestToolPath))
{
throw "$MANIFESTOOL_DIRECTORY does not contain '$dllName'"
}

Write-Log 'Done.'

return $manifestToolPath
}


function AddLocalDotnetDirPath {
$LocalDotnetDirPath = if ($IsWindows) { "$env:ProgramFiles/dotnet" } else { "/usr/share/dotnet" }
if (($env:PATH -split [IO.Path]::PathSeparator) -notcontains $LocalDotnetDirPath) {
$env:PATH = $LocalDotnetDirPath + [IO.Path]::PathSeparator + $env:PATH
}
}

function Find-Dotnet
{
AddLocalDotnetDirPath
$listSdksOutput = dotnet --list-sdks
$installedDotnetSdks = $listSdksOutput | ForEach-Object { $_.Split(" ")[0] }
Write-Host "Detected dotnet SDKs: $($installedDotnetSdks -join ', ')"
foreach ($majorMinorVersion in $DotnetSDKVersionRequirements.Keys) {
$minimalVersion = "$majorMinorVersion.$($DotnetSDKVersionRequirements[$majorMinorVersion].MinimalPatch)"
$firstAcceptable = $installedDotnetSdks |
Where-Object { $_.StartsWith("$majorMinorVersion.") } |
Where-Object { [System.Management.Automation.SemanticVersion]::new($_) -ge [System.Management.Automation.SemanticVersion]::new($minimalVersion) } |
Select-Object -First 1
if (-not $firstAcceptable) {
throw "Cannot find the dotnet SDK for .NET Core $majorMinorVersion. Version $minimalVersion or higher is required. Please specify '-Bootstrap' to install build dependencies."
}
}
}

function Install-Dotnet {
[CmdletBinding()]
param(
[string]$Channel = 'release'
)
try {
Find-Dotnet
return # Simply return if we find dotnet SDk with the correct version
} catch { }
$obtainUrl = "https://raw.githubusercontent.com/dotnet/cli/master/scripts/obtain"
try {
$installScript = if ($IsWindows) { "dotnet-install.ps1" } else { "dotnet-install.sh" }
Invoke-WebRequest -Uri $obtainUrl/$installScript -OutFile $installScript
foreach ($majorMinorVersion in $DotnetSDKVersionRequirements.Keys) {
$version = "$majorMinorVersion.$($DotnetSDKVersionRequirements[$majorMinorVersion].DefaultPatch)"
Write-Host "Installing dotnet SDK version $version"
if ($IsWindows) {
& .\$installScript -InstallDir "$env:ProgramFiles/dotnet" -Channel $Channel -Version $Version
} else {
bash ./$installScript --install-dir "/usr/share/dotnet" -c $Channel -v $Version
}
}
AddLocalDotnetDirPath
}
finally {
Remove-Item $installScript -Force -ErrorAction SilentlyContinue
}
}
2 changes: 1 addition & 1 deletion src/DurableSDK/DurableSDK.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,6 @@

<ItemGroup>
<ProjectReference Include="..\DurableEngine\DurableEngine.csproj" />
<PackageReference Include="Microsoft.PowerShell.SDK" Version="7.2.1" PrivateAssets="all"/>
<PackageReference Include="Microsoft.PowerShell.SDK" Version="7.2.1" PrivateAssets="all" />
</ItemGroup>
</Project>
27 changes: 15 additions & 12 deletions test/E2E/Start-E2ETest.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
#
param
(
[Switch]
$NoBuild,
[Switch]
$UseCoreToolsBuildFromIntegrationTests,
[Switch]
Expand Down Expand Up @@ -98,11 +100,11 @@ Write-Host "Deleting $FUNC_CLI_DIRECTORY if it exists..."
Remove-Item -Force "$FUNC_CLI_DIRECTORY.zip" -ErrorAction Ignore
Remove-Item -Recurse -Force $FUNC_CLI_DIRECTORY -ErrorAction Ignore

if (-not $SkipCoreToolsDownload.IsPresent)
if (-not $SkipCoreToolsDownload)
{
Write-Host "Downloading Core Tools because SkipCoreToolsDownload switch parameter is not present..."
$coreToolsDownloadURL = $null
if ($UseCoreToolsBuildFromIntegrationTests.IsPresent)
if ($UseCoreToolsBuildFromIntegrationTests)
{
$coreToolsDownloadURL = "https://functionsintegclibuilds.blob.core.windows.net/builds/$FUNC_RUNTIME_VERSION/latest/Azure.Functions.Cli.$os-$arch.zip"
$env:CORE_TOOLS_URL = "https://functionsintegclibuilds.blob.core.windows.net/builds/$FUNC_RUNTIME_VERSION/latest"
Expand Down Expand Up @@ -148,16 +150,17 @@ if (-not $SkipCoreToolsDownload.IsPresent)
$env:FUNC_PATH = $funcPath
Write-Host "Set FUNC_PATH environment variable to $env:FUNC_PATH"

# For both integration build test runs and regular test runs, we copy binaries to durableApp/Modules
Write-Host "Building the DurableSDK module and copying binaries to the durableApp/Modules directory..."
$configuration = if ($env:CONFIGURATION) { $env:CONFIGURATION } else { 'Debug' }

Push-Location "$PSScriptRoot/../.."
try {
& ./build.ps1 -Configuration 'Debug'
}
finally {
Pop-Location
if (-not $NoBuild) {
# For both integration build test runs and regular test runs, we copy binaries to durableApp/Modules
Write-Host "Building the DurableSDK module and copying binaries to the durableApp/Modules directory..."

Push-Location "$PSScriptRoot/../.."
try {
& ./build.ps1 -Configuration 'Debug'
}
finally {
Pop-Location
}
}

Write-Host "Starting Core Tools..."
Expand Down
2 changes: 1 addition & 1 deletion test/E2E/durableApp/SimpleOrchestrator/run.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,4 @@ $ErrorActionPreference = 'Stop'

$output = Invoke-DurableActivity -FunctionName "Hello" -Input "Tokyo"

return $output
$output