Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create the Windows.x64 global tool with shim for signing #21559

Merged
merged 29 commits into from
May 3, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
9b833ca
Add global tool shim packing properties
adityapatwardhan Apr 23, 2024
414425f
Move property to unix project
adityapatwardhan Apr 24, 2024
468cf77
Add pack as tool only for fxdependent on linux
adityapatwardhan Apr 24, 2024
0d7f0eb
Create Windows.x64 global tool in build
adityapatwardhan Apr 27, 2024
63f7a0e
Fix conditional template call
adityapatwardhan Apr 27, 2024
8d71940
Fix typo
adityapatwardhan Apr 27, 2024
872d7f7
Fix issues
adityapatwardhan Apr 27, 2024
c264275
Fix version
adityapatwardhan Apr 27, 2024
7fdc904
Fix gci
adityapatwardhan Apr 27, 2024
57f4cfd
Add ipmo and start-psbootstrap
adityapatwardhan Apr 28, 2024
b8f0b8c
Fix typo
adityapatwardhan Apr 28, 2024
6dacc20
Fix signing path
adityapatwardhan Apr 28, 2024
be4bbac
Add artifact path
adityapatwardhan Apr 28, 2024
3c62453
Manually remove content files
adityapatwardhan Apr 30, 2024
85448da
Restore gallery modules in package
adityapatwardhan Apr 30, 2024
f2105af
Fix typo
adityapatwardhan Apr 30, 2024
f3ffdd1
Fix package version
adityapatwardhan Apr 30, 2024
5a78705
Pull global tool from build
adityapatwardhan Apr 30, 2024
71c02e3
Remove changes from unix project
adityapatwardhan Apr 30, 2024
e29b584
Add some cleanup
adityapatwardhan May 1, 2024
a3528ad
Add nuspec for properties
adityapatwardhan May 1, 2024
721e709
Fix icon file path
adityapatwardhan May 1, 2024
e77aac0
Add resolve-path
adityapatwardhan May 1, 2024
6ecc604
Fix icon file path 2
adityapatwardhan May 1, 2024
0845756
Include icon file in package
adityapatwardhan May 2, 2024
2b08bb5
Add icon file to folder
adityapatwardhan May 2, 2024
589505b
Add icon file to folder 2
adityapatwardhan May 2, 2024
f23e52c
Remove icon file copy
adityapatwardhan May 2, 2024
57c8974
Dont uses nuspec
adityapatwardhan May 2, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion .pipelines/templates/nupkg.yml
Original file line number Diff line number Diff line change
Expand Up @@ -258,7 +258,8 @@ jobs:
$linuxFxdPath = "$(Pipeline.Workspace)\CoOrdinatedBuildPipeline\drop_linux_sign_linux_fxd\Signed-fxdependent"
$alpineFxdPath = "$(Pipeline.Workspace)\CoOrdinatedBuildPipeline\drop_linux_sign_linux_fxd_x64_alpine\Signed-fxdependent-noopt-linux-musl-x64"

$packageTypes = @('Unified', 'PowerShell.Linux.Alpine', 'PowerShell.Linux.x64', 'PowerShell.Linux.arm32', 'PowerShell.Linux.arm64', 'PowerShell.Windows.x64')
# Build global tools which do not have the shims exe generated in build.
$packageTypes = @('Unified', 'PowerShell.Linux.Alpine', 'PowerShell.Linux.x64', 'PowerShell.Linux.arm32', 'PowerShell.Linux.arm64')

$packageTypes | Foreach-Object {
$PackageType = $_
Expand Down Expand Up @@ -299,6 +300,12 @@ jobs:
$nupkgOutputPath = Join-Path -Path '$(Pipeline.Workspace)' -ChildPath 'nupkg'
Get-ChildItem -Path $nupkgOutputPath -Filter *.nupkg -Recurse | Copy-Item -Destination '$(ob_outputDirectory)' -Force -Verbose

# Copy Windows.x86 global tool from build to output directory
$winX64GlobalTool = "$(Pipeline.Workspace)\CoOrdinatedBuildPipeline\drop_windows_build_windows_fxdependent_release\globaltool\powershell*.nupkg"
Write-Verbose -Verbose "Finding Windows.x64 global tool at $winX64GlobalTool"
$globalToolPath = Get-Item $winX64GlobalTool
Copy-Item -Path $globalToolPath -Destination '$(ob_outputDirectory)' -Force -Verbose

Write-Verbose -Verbose "Copying global tools to output directory"
$gblToolOutputPath = Join-Path -Path '$(Pipeline.Workspace)' -ChildPath 'globaltools'
Get-ChildItem -Path $gblToolOutputPath -Filter *.nupkg -Recurse | Copy-Item -Destination '$(ob_outputDirectory)' -Force -Verbose
Expand Down
16 changes: 12 additions & 4 deletions .pipelines/templates/obp-file-signing.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
parameters:
binPath: '$(ob_outputDirectory)'
globalTool: 'false'

steps:
- pwsh: |
Expand Down Expand Up @@ -138,10 +139,17 @@ steps:
- pwsh: |
Import-Module '$(PowerShellRoot)/build.psm1' -Force
Import-Module '$(PowerShellRoot)/tools/packaging' -Force
$pathForUpload = New-Item -ItemType Directory -Path '$(ob_outputDirectory)/Signed-$(Runtime)' -Force
Write-Verbose -Verbose -Message "pathForUpload: $pathForUpload"
Copy-Item -Path '${{ parameters.binPath }}\*' -Destination $pathForUpload -Recurse -Force -Verbose
Write-Verbose -Verbose -Message "Files copied to $pathForUpload"
$isGlobalTool = '${{ parameters.globalTool }}' -eq 'true'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not blocking. We should move any PS Script of significant size into a script or module. This does two things, makes sure they are scanned by static analysis tools and reduces total yaml size.


if (-not $isGlobalTool) {
$pathForUpload = New-Item -ItemType Directory -Path '$(ob_outputDirectory)/Signed-$(Runtime)' -Force
Write-Verbose -Verbose -Message "pathForUpload: $pathForUpload"
Copy-Item -Path '${{ parameters.binPath }}\*' -Destination $pathForUpload -Recurse -Force -Verbose
Write-Verbose -Verbose -Message "Files copied to $pathForUpload"
}
else {
$pathForUpload = '${{ parameters.binPath }}'
}

Write-Verbose "Copying third party signed files to the build folder"
$thirdPartySignedFilesPath = (Get-Item '$(Pipeline.Workspace)/thirdPartyToBeSigned').FullName
Expand Down
143 changes: 143 additions & 0 deletions .pipelines/templates/windows-hosted-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,8 @@ jobs:
$null = New-Item -ItemType Directory -Path $buildWithSymbolsPath -Force -Verbose
Start-PSBuild -Runtime $runtime -Configuration Release -Output $buildWithSymbolsPath -Clean -PSModuleRestore @params

$refFolderPath = Join-Path $buildWithSymbolsPath 'ref'
Write-Verbose -Verbose "refFolderPath: $refFolderPath"
$outputPath = Join-Path '$(ob_outputDirectory)' 'psoptions'
$null = New-Item -ItemType Directory -Path $outputPath -Force
$psOptPath = "$outputPath/psoptions.json"
Expand All @@ -106,6 +108,39 @@ jobs:
}
}

if ($runtime -eq 'fxdependent')
{
## Also build global tool
Write-Verbose -Message "Building PowerShell global tool for Windows.x64" -Verbose
$globalToolCsProjDir = Join-Path $(PowerShellRoot) 'src' 'GlobalTools' 'PowerShell.Windows.x64'
Push-Location -Path $globalToolCsProjDir -Verbose

$globalToolArtifactPath = Join-Path $(Build.SourcesDirectory) 'GlobalTool'
$vstsCommandString = "vso[task.setvariable variable=GlobalToolArtifactPath]${globalToolArtifactPath}"
Write-Host "sending " + $vstsCommandString
Write-Host "##$vstsCommandString"

dotnet publish --no-self-contained --artifacts-path $globalToolArtifactPath /property:PackageVersion=$(Version)
$globalToolBuildModulePath = Join-Path $globalToolArtifactPath 'publish' 'PowerShell.Windows.x64' 'release'
Pop-Location
# do this to ensure everything gets signed.
Restore-PSModuleToBuild -PublishPath $globalToolBuildModulePath

# Copy reference assemblies
Copy-Item -Path $refFolderPath -Destination $globalToolBuildModulePath -Recurse -Force

Write-Verbose -Verbose "clean unnecessary files in obj directory"
$objDir = Join-Path $globalToolArtifactPath 'obj' 'PowerShell.Windows.x64' 'release'

$filesToKeep = @("apphost.exe", "PowerShell.Windows.x64.pdb", "PowerShell.Windows.x64.dll", "project.assets.json")

# only four files are needed in obj folder for global tool packaging
Get-ChildItem -Path $objDir -File -Recurse |
Where-Object { -not $_.PSIsContainer } |
Where-Object { $_.name -notin $filesToKeep } |
Remove-Item -Verbose
}

Write-Verbose -Verbose "Completed building PowerShell for '$env:BuildConfiguration' configuration"
displayName: 'Build Windows Universal - $(Architecture)-$(BuildConfiguration) Symbols folder'
env:
Expand All @@ -128,4 +163,112 @@ jobs:
parameters:
binPath: '$(Pipeline.Workspace)/Symbols_$(Architecture)'

## first we sign all the files in the bin folder
- ${{ if eq(variables['Architecture'], 'fxdependent') }}:
- template: /.pipelines/templates/obp-file-signing.yml@self
parameters:
binPath: '$(GlobalToolArtifactPath)/publish/PowerShell.Windows.x64/release'
globalTool: 'true'

- pwsh: |
Get-ChildItem '$(GlobalToolArtifactPath)/obj/PowerShell.Windows.x64/release'
displayName: Capture obj files
condition: and(succeeded(), eq(variables['Architecture'], 'fxdependent'))

## Now we sign couple of file from the obj folder which are needed for the global tool packaging
- task: onebranch.pipeline.signing@1
displayName: Sign obj files
inputs:
command: 'sign'
signing_profile: external_distribution
files_to_sign: '**\*.dll;**\*.exe'
search_root: '$(GlobalToolArtifactPath)/obj/PowerShell.Windows.x64/release'
condition: and(succeeded(), eq(variables['Architecture'], 'fxdependent'))

- pwsh: |
<# The way the packaging works is a bit tricky as when it is built, we cannot add the modules that come from gallery.
We have to use dotnet pack to build the nupkg and then expand it as a zip.
After expanding we restore the signed files for the modules from the gallery.
We also delete pdbs, content and contentFiles folder which are not necessary.
After that, we repack using Compress-Archive and rename it back to a nupkg.
#>

Import-Module -Name $(PowerShellRoot)/build.psm1 -Force
Start-PSBootstrap
$packagingStrings = Import-PowerShellDataFile "$(PowerShellRoot)\tools\packaging\packaging.strings.psd1"

$outputPath = Join-Path '$(ob_outputDirectory)' 'globaltool'
$null = New-Item -ItemType Directory -Path $outputPath -Force
$globalToolCsProjDir = Join-Path $(PowerShellRoot) 'src' 'GlobalTools' 'PowerShell.Windows.x64'
Push-Location -Path $globalToolCsProjDir -Verbose

<#
$nuspecFilePath = "$globalToolCsProjDir\PowerShell.Windows.x64.nuspec"
$nuSpec = $packagingStrings.WindowsX64GlobalToolNuspec -f '$(Version)'
$nuSpec | Out-File -FilePath $nuspecFilePath -Encoding ascii
#>

dotnet pack --output $outputPath --no-build --artifacts-path '$(GlobalToolArtifactPath)' /property:PackageVersion=$(Version) /property:PackageIcon=Powershell_64.png

Write-Verbose -Verbose "Deleting content and contentFiles folders from the nupkg"

$nupkgs = Get-ChildItem -Path $outputPath -Filter powershell*.nupkg

$nupkgName = $nupkgs.Name
$newName = $nupkgName -replace '(\.nupkg)$', '.zip'
Rename-Item -Path $nupkgs.FullName -NewName $newName

$zipPath = Get-ChildItem -Path $outputPath -Filter powershell*.zip

# Expand zip and remove content and contentFiles folders
Expand-Archive -Path $zipPath -DestinationPath "$outputPath\temp" -Force

$modulesToCopy = @(
'PowerShellGet'
'PackageManagement'
'Microsoft.PowerShell.PSResourceGet'
'Microsoft.PowerShell.Archive'
'PSReadLine'
'ThreadJob'
)

$sourceModulePath = Join-Path '$(GlobalToolArtifactPath)' 'publish' 'PowerShell.Windows.x64' 'release' 'Modules'
$destModulesPath = Join-Path "$outputPath" 'temp' 'tools' 'net9.0' 'any' 'modules'

$modulesToCopy | ForEach-Object {
$modulePath = Join-Path $sourceModulePath $_
Copy-Item -Path $modulePath -Destination $destModulesPath -Recurse -Force
}

# Copy ref assemblies
Copy-Item '$(Pipeline.Workspace)/Symbols_$(Architecture)/ref' "$outputPath\temp\tools\net9.0\any\ref" -Recurse -Force

$contentPath = Join-Path "$outputPath\temp" 'content'
$contentFilesPath = Join-Path "$outputPath\temp" 'contentFiles'

Remove-Item -Path $contentPath,$contentFilesPath -Recurse -Force

# remove PDBs to reduce the size of the nupkg
Remove-Item -Path "$outputPath\temp\tools\net9.0\any\*.pdb" -Recurse -Force

Compress-Archive -Path "$outputPath\temp\*" -DestinationPath "$outputPath\$nupkgName" -Force

Remove-Item -Path "$outputPath\temp" -Recurse -Force
Remove-Item -Path $zipPath -Force

if (-not (Test-Path "$outputPath\powershell.windows.x64.*.nupkg")) {
throw "Global tool package not found at $outputPath"
}
displayName: 'Pack Windows.x64 global tool'
condition: and(succeeded(), eq(variables['Architecture'], 'fxdependent'))

- task: onebranch.pipeline.signing@1
displayName: Sign nupkg files
inputs:
command: 'sign'
cp_code: 'CP-401405'
files_to_sign: '**\*.nupkg'
search_root: '$(ob_outputDirectory)\globaltool'
condition: and(succeeded(), eq(variables['Architecture'], 'fxdependent'))

- template: /.pipelines/templates/step/finalize.yml@self
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net9.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<PackAsTool>true</PackAsTool>
<PackAsToolShimRuntimeIdentifiers>win-x64</PackAsToolShimRuntimeIdentifiers>
<ToolCommandName>pwsh</ToolCommandName>
<Version>$(PackageVersion)</Version>
</PropertyGroup>

<ItemGroup>
<Compile Include="..\..\powershell\Program.cs" Exclude="bin\**;obj\**;**\*.xproj;packages\**" />
<Content Include="..\..\Modules\Windows\**\*;..\..\Modules\Shared\**\*">
<Link>Modules\%(RecursiveDir)\%(FileName)%(Extension)</Link>
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<CopyToPublishDirectory>PreserveNewest</CopyToPublishDirectory>
</Content>
<None Include="Powershell_64.png" Pack="true" PackagePath="" />
</ItemGroup>

<ItemGroup>
<ProjectReference Include="..\..\Microsoft.PowerShell.SDK\Microsoft.PowerShell.SDK.csproj" />
<ProjectReference Include="..\..\Microsoft.PowerShell.Commands.Diagnostics\Microsoft.PowerShell.Commands.Diagnostics.csproj" />
<ProjectReference Include="..\..\Microsoft.Management.Infrastructure.CimCmdlets\Microsoft.Management.Infrastructure.CimCmdlets.csproj" />
<ProjectReference Include="..\..\Microsoft.WSMan.Management\Microsoft.WSMan.Management.csproj" />
</ItemGroup>

</Project>
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
30 changes: 16 additions & 14 deletions tools/packaging/packaging.psm1
Original file line number Diff line number Diff line change
Expand Up @@ -4341,20 +4341,22 @@ function New-GlobalToolNupkgSource
$toolSettings = $packagingStrings.GlobalToolSettingsFile -f "pwsh.dll"
}

"PowerShell.Windows.x64"
{
$PackageName = "PowerShell.Windows.x64"
$RootFolder = New-TempFolder

Copy-Item -Path $iconPath -Destination "$RootFolder/$iconFileName" -Verbose

$ridFolder = New-Item -Path (Join-Path $RootFolder "tools/$script:netCoreRuntime/any") -ItemType Directory

Write-Log "New-GlobalToolNupkgSource: Copying runtime assemblies from $WindowsDesktopBinPath for $PackageType"
Copy-Item "$WindowsDesktopBinPath/*" -Destination $ridFolder -Recurse
Remove-Item -Path $ridFolder/runtimes/win-arm -Recurse -Force
$toolSettings = $packagingStrings.GlobalToolSettingsFile -f "pwsh.dll"
}
# Due to needing a signed shim for the global tool, we build the global tool in build instead of packaging.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove dead code?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to keep the code and document why we don't build Windows.x64 here.

# keeping the code for reference.
# "PowerShell.Windows.x64"
# {
# $PackageName = "PowerShell.Windows.x64"
# $RootFolder = New-TempFolder

# Copy-Item -Path $iconPath -Destination "$RootFolder/$iconFileName" -Verbose

# $ridFolder = New-Item -Path (Join-Path $RootFolder "tools/$script:netCoreRuntime/any") -ItemType Directory

# Write-Log "New-GlobalToolNupkgSource: Copying runtime assemblies from $WindowsDesktopBinPath for $PackageType"
# Copy-Item "$WindowsDesktopBinPath/*" -Destination $ridFolder -Recurse
# Remove-Item -Path $ridFolder/runtimes/win-arm -Recurse -Force
# $toolSettings = $packagingStrings.GlobalToolSettingsFile -f "pwsh.dll"
# }

"PowerShell.Windows.arm32"
{
Expand Down
23 changes: 23 additions & 0 deletions tools/packaging/packaging.strings.psd1
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,29 @@ open {0}
</packageTypes>
</metadata>
</package>
'@

WindowsX64GlobalToolNuspec = @'
<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2013/05/nuspec.xsd">
<metadata>
<id>PowerShelll.Windows.x64</id>
<version>{0}</version>
<authors>Microsoft</authors>
<owners>Microsoft,PowerShell</owners>
<projectUrl>https://github.com/PowerShell/PowerShell</projectUrl>
<icon>Powershell_64.png</icon>
<requireLicenseAcceptance>false</requireLicenseAcceptance>
<description>PowerShell global tool</description>
<license type="expression">MIT</license>
<tags>PowerShell</tags>
<language>en-US</language>
<copyright>&#169; Microsoft Corporation. All rights reserved.</copyright>
<packageTypes>
<packageType name="DotnetTool" />
</packageTypes>
</metadata>
</package>
'@

GlobalToolSettingsFile = @'
Expand Down