Skip to content
This repository has been archived by the owner on Jan 12, 2024. It is now read-only.

Build conda packages for IQ#, qsharp.py #37

Merged
merged 85 commits into from
Sep 25, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
a104b49
Started iqsharp recipe.
cgranade Aug 9, 2019
fb7339a
Update Jupyter.Core dependency to get --sys-prefix.
cgranade Aug 9, 2019
820fe05
Started work on packing new recipe.
cgranade Aug 9, 2019
22426b6
Modify steps to use conda env instead.
cgranade Aug 9, 2019
4b63796
Remove redundant step.
cgranade Aug 9, 2019
1c81089
Updated description.
cgranade Aug 9, 2019
51c3c0b
Add conda package to drop.
cgranade Aug 9, 2019
ab170a4
Some very painful shell scripting.
cgranade Aug 9, 2019
78587b6
Move iqsharp conda install to opt.
cgranade Aug 9, 2019
614bcda
Fix order in which we add to path.
cgranade Aug 9, 2019
72146b2
Fix some bugs building dotnetcore-sdk from Linux.
cgranade Aug 9, 2019
5629692
Use powershell instead of pwsh when building from Windows.
cgranade Aug 9, 2019
0c2c32b
Suppress profile when building iqsharp on POSIX.
cgranade Aug 9, 2019
fa9e256
Made pwsh dependency from iqsharp POSIX-only.
cgranade Aug 9, 2019
2fe1e67
Started working on pwsh package.
cgranade Aug 9, 2019
21f4b6e
Only pack pwsh on non-Windows hosts.
cgranade Aug 9, 2019
5cf641a
Prevent pwsh from capturing NuGet cache.
cgranade Aug 9, 2019
a3eeb5f
Fix PowerShell 5.1 vs 6.2 issues.
cgranade Aug 9, 2019
36030bb
Fixed name of DOTNET_ROOT env var.
cgranade Aug 9, 2019
36dbe87
Fix Runtime ID check on PS 5.1.
cgranade Aug 9, 2019
53a5169
Fix name of activation/deactivation scripts.
cgranade Aug 9, 2019
e9858db
Force line endings to enable patching 10607.
cgranade Aug 9, 2019
f6fae69
Add SHA256 sums.
cgranade Aug 9, 2019
4a4f478
Build fixes for # [non win].
cgranade Aug 9, 2019
505cc3e
cgranade Aug 9, 2019
ac71c2a
Fixes to test.
cgranade Aug 10, 2019
55f35ad
Fixed meta.yaml for windows.
cgranade Aug 10, 2019
539ea23
Version bump
cgranade Aug 10, 2019
0b463f7
Suppress posh profile on Windows.
cgranade Aug 10, 2019
e30a049
Fixed line endings in dotnet/cli#10607 workaround.
cgranade Aug 10, 2019
e6bf609
Pin versions in pwsh.
cgranade Aug 10, 2019
2d80ecf
Added slightly more tests to iqsharp package.
cgranade Aug 10, 2019
0fa785d
Use Jinja2 to handle IQ# version passed by set-env.ps1.
cgranade Aug 12, 2019
d69051f
Added qsharp package as well.
cgranade Aug 12, 2019
ed87d4d
Added qsharp package to pack as well.
cgranade Aug 12, 2019
1ce876d
macOS fixes
Aug 16, 2019
e029ba3
Merge branch 'master' into cgranade/conda-build
cgranade Sep 10, 2019
23145de
Fix some metadata.
cgranade Sep 11, 2019
45596a4
Extended build definition for xplat conda build.
cgranade Sep 11, 2019
5000926
Renamed xplat conda build jobs.
cgranade Sep 11, 2019
8c8039f
Propagate step template parameter into env.
cgranade Sep 11, 2019
1c72a99
Adding more detail to pack diagnostics.
cgranade Sep 11, 2019
f132e77
Handle stderr from conda-build better.
cgranade Sep 11, 2019
0fffb49
Detect $IsWindows even on Windows.
cgranade Sep 11, 2019
bc5ab10
Revert and downgrade to warning.
cgranade Sep 11, 2019
9febe94
Separate out conda-build call into separate job.
cgranade Sep 11, 2019
3cfcd26
Find .NET Core SDK more reliabily.
cgranade Sep 12, 2019
3283808
Write conda build logs to temporary file as well.
cgranade Sep 12, 2019
74b61d2
trying again to log.
cgranade Sep 12, 2019
31a4390
Explicitly write all conda-build output to host.
cgranade Sep 12, 2019
0abc0eb
Fixing property name in log upload.
cgranade Sep 12, 2019
a97dc5c
Fix uploadfile task command.
cgranade Sep 12, 2019
c5644d7
Trying again to get any log information out of ADO.
cgranade Sep 12, 2019
a9ccc9b
Don't trust conda-build's exit code.
cgranade Sep 12, 2019
9b17aed
Switch to using pwsh everywhere.
cgranade Sep 12, 2019
0764e1b
Fix macOS-specific bug.
cgranade Sep 12, 2019
ddc0200
Fix pack-conda diagnostics.
cgranade Sep 12, 2019
6119125
Forgot to remove log upload.
cgranade Sep 12, 2019
540df3b
First stab at removing dotnetcore-sdk and pwsh recipes.
cgranade Sep 12, 2019
f073363
Updated pack script for reduced package set.
cgranade Sep 12, 2019
86c7fbc
Fixed test-time requirements for iqsharp.
cgranade Sep 12, 2019
4041e18
Force pwsh vs powershell.
cgranade Sep 18, 2019
c6a1783
Try using pwsh on Windows.
cgranade Sep 18, 2019
527d240
Lock conda-build to 3.18.8.
cgranade Sep 18, 2019
eeaf8a9
Added call to dotnet --info.
cgranade Sep 18, 2019
fdf0bd8
Revert "Added call to dotnet --info."
cgranade Sep 18, 2019
2943c2a
Lock conda build to dotnet 2.2.401
cgranade Sep 18, 2019
18c370d
Adding more diagnostics.
cgranade Sep 18, 2019
dcb6457
More diagnostics.
cgranade Sep 19, 2019
4f16132
even more diagnostics
cgranade Sep 19, 2019
7f99746
Disable binary prefix searching.
cgranade Sep 19, 2019
22b9f6d
yet more debugging
cgranade Sep 19, 2019
343a0fe
fix diagnostics.
cgranade Sep 19, 2019
edde3fa
One more fix to diagnostics.
cgranade Sep 19, 2019
33a4eeb
one moar fix
cgranade Sep 19, 2019
9e66ee4
undo diagnostics.
cgranade Sep 19, 2019
311fd0b
Workaround for conda/conda-build#3736.
cgranade Sep 19, 2019
35b4a6f
Workaround for conda/conda-build#3170.
cgranade Sep 19, 2019
aa93293
Trying again to redirect stderr.
cgranade Sep 19, 2019
646299f
trying one more time to suppress stderr.
cgranade Sep 19, 2019
86a5a87
Putting try catch back.
cgranade Sep 19, 2019
d6c2e0a
Redirect just stderr.
cgranade Sep 19, 2019
9f9ebcc
Downgrade error action as well.
cgranade Sep 19, 2019
5b26947
Build 3.6 and 3.7 versions.
cgranade Sep 19, 2019
739b88f
Adding python version to outputs as well.
cgranade Sep 19, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added .ionide/symbolCache.db
Binary file not shown.
27 changes: 27 additions & 0 deletions .vscode/launch.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
{
// Use IntelliSense to find out which attributes exist for C# debugging
// Use hover for the description of the existing attributes
// For further information visit https://github.com/OmniSharp/omnisharp-vscode/blob/master/debugger-launchjson.md
"version": "0.2.0",
"configurations": [
{
"name": ".NET Core Launch (console)",
"type": "coreclr",
"request": "launch",
"preLaunchTask": "build",
// If you have changed target frameworks, make sure to update the program path.
"program": "${workspaceFolder}/src/Tool/bin/Debug/netcoreapp2.2/Microsoft.Quantum.IQSharp.dll",
"args": [],
"cwd": "${workspaceFolder}/src/Tool",
// For more information about the 'console' field, see https://aka.ms/VSCode-CS-LaunchJson-Console
"console": "internalConsole",
"stopAtEntry": false
},
{
"name": ".NET Core Attach",
"type": "coreclr",
"request": "attach",
"processId": "${command:pickProcess}"
}
]
}
42 changes: 42 additions & 0 deletions .vscode/tasks.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
{
"version": "2.0.0",
"tasks": [
{
"label": "build",
"command": "dotnet",
"type": "process",
"args": [
"build",
"${workspaceFolder}/src/Tool/Tool.csproj",
"/property:GenerateFullPaths=true",
"/consoleloggerparameters:NoSummary"
],
"problemMatcher": "$msCompile"
},
{
"label": "publish",
"command": "dotnet",
"type": "process",
"args": [
"publish",
"${workspaceFolder}/src/Tool/Tool.csproj",
"/property:GenerateFullPaths=true",
"/consoleloggerparameters:NoSummary"
],
"problemMatcher": "$msCompile"
},
{
"label": "watch",
"command": "dotnet",
"type": "process",
"args": [
"watch",
"run",
"${workspaceFolder}/src/Tool/Tool.csproj",
"/property:GenerateFullPaths=true",
"/consoleloggerparameters:NoSummary"
],
"problemMatcher": "$msCompile"
}
]
}
2 changes: 1 addition & 1 deletion NOTICE.txt
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

-------------------------------------------------------------------

Microsoft.Jupyter.Core 1.1.14623 - MIT
Microsoft.Jupyter.Core 1.1.17470 - MIT
(c) 2008 VeriSign, Inc.
(c) Microsoft Corporation.
Copyright (c) Microsoft Corporation.
Expand Down
18 changes: 18 additions & 0 deletions build/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,21 @@ jobs:
displayName: 'Component Detection'
inputs:
failOnAlert: true

- job: "pack_conda_linux"
pool:
vmImage: ubuntu-16.04
steps:
- template: steps-conda.yml

- job: "pack_conda_macos"
pool:
vmImage: macOS-10.14
steps:
- template: steps-conda.yml

- job: "pack_conda_win"
pool:
vmImage: windows-2019
steps:
- template: steps-conda.yml
69 changes: 69 additions & 0 deletions build/pack-conda.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License.
$ErrorActionPreference = 'Stop'

& "$PSScriptRoot/set-env.ps1"
$all_ok = $True
$CondaPlatform = (conda info --json) `
| ConvertFrom-Json `
| Select-Object -ExpandProperty platform;

# Write out diagnostics about what version of PowerShell we're on.
$PSVersionTable | Format-Table | Out-String | Write-Host;

function Pack-CondaRecipe() {
param(
[string] $Path,
[string[]] $PythonVersions = @("3.6", "3.7")
);

if (-not (Get-Command conda-build -ErrorAction SilentlyContinue)) {
Write-Host "##vso[task.logissue type=warning;] conda-build not installed. " + `
"Will skip creation of conda package for $Path.";
return;
}

# conda-build prints some warnings to stderr, which can lead to false positives.
# We wrap in a try-finally to make sure we can condition on the built file being there, and not on
# writing to stderr.
try {
$OldPreference = $ErrorActionPreference;
$ErrorActionPreference = "Continue";
$PythonVersions | `
ForEach-Object {
Write-Host "##[info]Running: conda build $(Resolve-Path $Path) --python=$_"
# See https://stackoverflow.com/a/20950421/267841 for why this works to force conda
# to output all log messages to stdout instead of stderr.
conda build (Resolve-Path $Path) --python=$_ 2>&1 | ForEach-Object { "$_" };
}
} catch {
Write-Host "##vso[task.logissue type=warning;]conda build error: $_";
} finally {
$ErrorActionPreference = $OldPreference;
$TargetDir = (Join-Path $Env:CONDA_OUTDIR $CondaPlatform);
New-Item -ItemType Directory -Path $TargetDir -Force -ErrorAction SilentlyContinue;
$PythonVersions | `
ForEach-Object {
$PackagePath = (conda build (Resolve-Path $Path) --python=$_ --output);
if (Test-Path $PackagePath) {
Copy-Item `
$PackagePath `
$TargetDir `
-ErrorAction Continue `
-Verbose;
Write-Host "##[info]Copied $PackagePath to $TargetDir.";
} else {
Write-Host "##vso[task.logissue type=error;]Failed to create conda package for $Path (Python $_)."
$script:all_ok = $False
}
}
}
}

Write-Host "##[info]Packing conda recipes..."
Pack-CondaRecipe -Path "../conda-recipes/iqsharp"
Pack-CondaRecipe -Path "../conda-recipes/qsharp"

if (-not $all_ok) {
throw "At least one package failed to build. Check the logs."
}
5 changes: 0 additions & 5 deletions build/pack.ps1
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License.

$ErrorActionPreference = 'Stop'

& "$PSScriptRoot/set-env.ps1"
Expand Down Expand Up @@ -92,7 +91,3 @@ Pack-Wheel '../src/Python/'

Write-Host "##[info]Packing Docker image..."
Pack-Image -RepoName "iqsharp-base" -Dockerfile '../images/iqsharp-base/Dockerfile'

if (-not $all_ok) {
throw "At least one package failed to build. Check the logs."
}
3 changes: 3 additions & 0 deletions build/set-env.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ If (-not (Test-Path -Path $Env:NUGET_OUTDIR)) { [IO.Directory]::CreateDirectory(
If ($Env:PYTHON_OUTDIR -eq $null) { $Env:PYTHON_OUTDIR = (Join-Path $Env:DROPS_DIR "wheels") }
If (-not (Test-Path -Path $Env:PYTHON_OUTDIR)) { [IO.Directory]::CreateDirectory($Env:PYTHON_OUTDIR) }

If ($Env:CONDA_OUTDIR -eq $null) { $Env:CONDA_OUTDIR = (Join-Path $Env:DROPS_DIR "conda") }
If (-not (Test-Path -Path $Env:CONDA_OUTDIR)) { [IO.Directory]::CreateDirectory($Env:CONDA_OUTDIR) }

If ($Env:DOCS_OUTDIR -eq $null) { $Env:DOCS_OUTDIR = (Join-Path $Env:DROPS_DIR "docs") }
If (-not (Test-Path -Path $Env:DOCS_OUTDIR)) { [IO.Directory]::CreateDirectory($Env:DOCS_OUTDIR) }

36 changes: 36 additions & 0 deletions build/steps-conda.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
##
# Build and test IQ#.
##

steps:

##
# Pre-reqs
##
- task: CondaEnvironment@1
inputs:
packageSpecs: "python=3.6 pip setuptools pytest jupyter conda-build=3.18.8 conda-package-handling=1.3.11"
displayName: 'Use conda environment w/ Python 3.6'

- task: UseDotNet@2
displayName: 'Use .NET Core sdk'
inputs:
packageType: sdk
version: 2.2.401

##
# Pack conda packages
##
- pwsh: .\pack-conda.ps1
displayName: "Packing IQ# packages"
workingDirectory: '$(System.DefaultWorkingDirectory)/build'

##
# Publish build artifacts.
##
- task: PublishBuildArtifacts@1
displayName: 'Publish Artifact: drop'
condition: succeededOrFailed()
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)'
artifactName: iqsharp
19 changes: 8 additions & 11 deletions build/steps.yml
Original file line number Diff line number Diff line change
@@ -1,33 +1,30 @@
##
# Build and test IQ#.
##

steps:

##
# Pre-reqs
##
- task: UsePythonVersion@0
- task: CondaEnvironment@1
inputs:
versionSpec: '3.6'
architecture: 'x64'
displayName: 'Use Python 3.6'

- script: pip install setuptools wheel pytest jupyter
displayName: 'Install Python tools'
packageSpecs: "python=3.6 pip setuptools pytest jupyter"
displayName: 'Use conda environment w/ Python 3.6'

##
# Build, test & pack
##
- powershell: .\build.ps1
- pwsh: .\build.ps1
displayName: "Building IQ#"
workingDirectory: '$(System.DefaultWorkingDirectory)/build'

- powershell: .\test.ps1
- pwsh: .\test.ps1
displayName: "Testing IQ#"
workingDirectory: '$(System.DefaultWorkingDirectory)/build'
condition: and(succeeded(), ne(variables['Skip.Tests'], 'true'))

- powershell: .\pack.ps1
- pwsh: .\pack.ps1
displayName: "Packing IQ#"
workingDirectory: '$(System.DefaultWorkingDirectory)/build'

Expand All @@ -47,4 +44,4 @@ steps:
condition: succeededOrFailed()
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)'
artifactName: iqsharp
artifactName: iqsharp
45 changes: 45 additions & 0 deletions conda-recipes/iqsharp/ToolStandalone.csproj
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
<Project Sdk="Microsoft.NET.Sdk">
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need a different project? can you just target the existing one?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dotnet publish has a lot of bugs when running on a project that has <PackAsTool>true</PackAsTool>. This was the most reliable workaround that I could find for those issues.


<PropertyGroup>
<OutputType>Exe</OutputType>
<PlatformTarget>x64</PlatformTarget>
<TargetFramework>netcoreapp2.2</TargetFramework>
<RootNamespace>Microsoft.Quantum.IQSharp</RootNamespace>
<AssemblyName>Microsoft.Quantum.IQSharp</AssemblyName>
</PropertyGroup>

<PropertyGroup>
<NoWarn>0162</NoWarn>
<Authors>Microsoft</Authors>
<Description>Microsoft's IQ# Server.</Description>
<Copyright>© Microsoft Corporation. All rights reserved.</Copyright>
<ContentTargetFolders>\</ContentTargetFolders>
</PropertyGroup>

<ItemGroup>
<Compile Include="..\..\build\DelaySign.cs" Link="Properties\DelaySign.cs" />
</ItemGroup>

<ItemGroup>
<None Update="appsettings.json">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
</ItemGroup>

<ItemGroup>
<ProjectReference Include="..\Jupyter\Jupyter.csproj" />
<ProjectReference Include="..\Core\Core.csproj" />
<ProjectReference Include="..\Web\Web.csproj" />
</ItemGroup>

<ItemGroup>
<PackageReference Include="Microsoft.AspNetCore.App" />
<PackageReference Include="Microsoft.AspNetCore.Mvc.Core" Version="2.2.5" />
<PackageReference Include="Microsoft.AspNetCore.Razor.Design" Version="2.2.0" PrivateAssets="All" />
<PackageReference Include="Microsoft.Extensions.Logging" Version="2.2.0" />
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="2.2.0" />
<PackageReference Include="Microsoft.NETCore.Portable.Compatibility" Version="1.0.1" />
<PackageReference Include="System.Net.Http" Version="4.3.4" />
</ItemGroup>

</Project>
1 change: 1 addition & 0 deletions conda-recipes/iqsharp/bld.bat
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
pwsh -NoProfile iqsharp/conda-recipes/iqsharp/build.ps1
Loading