diff --git a/DECISIONS.md b/DECISIONS.md deleted file mode 100644 index 3c8fca1001..0000000000 --- a/DECISIONS.md +++ /dev/null @@ -1,115 +0,0 @@ -# Design Decisions - -This document records significant design decisions made during the implementation of the F# Compiler Regression Testing pipeline. - -## Decision 1: Template-Based Architecture - -**Context**: Need to implement regression testing that can be reused across pipelines. - -**Options Considered**: -1. Inline job definitions in azure-pipelines-PR.yml -2. Reusable Azure DevOps template in eng/templates/ - -**Decision**: Use reusable template approach. - -**Rationale**: -- Follows Azure DevOps best practices -- Reduces code duplication -- Makes it easy to extend with new libraries -- Consistent with existing patterns in the repository (eng/common/templates/) - ---- - -## Decision 2: Optimized Artifact Publishing - -**Context**: Need to share compiler artifacts between jobs. - -**Options Considered**: -1. Publish entire artifacts folder (~1.8GB) -2. Publish only essential directories (fsc and FSharp.Core) (~79MB) - -**Decision**: Publish only essential directories. - -**Rationale**: -- Reduces artifact size from 1.8GB to ~79MB -- Faster artifact upload/download -- Contains all necessary components for regression testing -- Matches approach in previous PR #18803 - ---- - -## Decision 3: Using F# Script for Directory.Build.props Setup - -**Context**: Need to inject UseLocalCompiler.Directory.Build.props import into third-party repos. - -**Options Considered**: -1. Pure PowerShell XML manipulation -2. F# script with proper XML handling -3. Simple file replacement - -**Decision**: Use F# script with XML handling. - -**Rationale**: -- Properly handles existing Directory.Build.props files -- Correctly inserts import at beginning of Project element -- Native F# tooling in an F# project -- Matches approach in previous PR #18803 - ---- - -## Decision 4: Specific Commit SHAs for Third-Party Libraries - -**Context**: Need reproducible regression tests. - -**Options Considered**: -1. Use main/master branch -2. Use specific commit SHAs - -**Decision**: Use specific commit SHAs. - -**Rationale**: -- Ensures reproducible test results -- Protects against breaking changes in third-party libraries -- Allows controlled updates when ready -- Standard practice for regression testing - ---- - -## Decision 5: Removal of Strategy Matrix from EndToEndBuildTests - -**Context**: The original EndToEndBuildTests job had a matrix for regular vs experimental features. - -**Options Considered**: -1. Keep the matrix and publish artifacts only from one configuration -2. Remove the matrix entirely -3. Publish artifacts from both configurations - -**Decision**: Remove the matrix entirely (per previous PR approach). - -**Rationale**: -- Simplifies artifact publishing -- Regression tests need consistent baseline -- Both configurations were building with empty experimental flag anyway -- Matches approach in previous PR #18803 - ---- - -## Decision 6: Use net10.0 Target Framework in Template - -**Context**: The pipeline needs to reference the correct .NET target framework for the compiler artifacts. - -**Options Considered**: -1. Hardcode net9.0 (as in PR #18803) -2. Use net10.0 (per current UseLocalCompiler.Directory.Build.props) -3. Make it configurable - -**Decision**: Use net10.0 to match the current project state. - -**Rationale**: -- The current repository uses .NET 10 SDK -- UseLocalCompiler.Directory.Build.props references net10.0 paths -- The artifacts/bin/fsc/Release/net10.0 directory exists -- PR #18803 used net9.0 because it was based on an older version of the repository -- Using net10.0 ensures compatibility with the current build output - ---- diff --git a/OBSTACLES.md b/OBSTACLES.md deleted file mode 100644 index 502da94973..0000000000 --- a/OBSTACLES.md +++ /dev/null @@ -1,79 +0,0 @@ -# Obstacles Encountered - -This document tracks any obstacles encountered during the implementation of the Azure DevOps regression testing template. - -## Current Obstacles - -None at this time. - -## Resolved Obstacles - -### 1. FSharpPlus Build Failure: `_GetRestoreSettingsPerFramework` Target Missing - -**Date:** 2025-11-28 - -**Symptom:** -``` -D:\a\_work\1\TestRepo\src\FSharpPlus.TypeLevel\Providers\FSharpPlus.Providers.fsproj : error MSB4057: The target "_GetRestoreSettingsPerFramework" does not exist in the project. [TargetFramework=net8.0] -``` - -**Root Cause:** -The artifacts were being downloaded to an incorrect path structure. The UseLocalCompiler.Directory.Build.props file expects: -```xml -$(LocalFSharpCompilerPath)/artifacts/bin/fsc/$(LocalFSharpCompilerConfiguration)/net10.0 -``` - -But the artifacts were being downloaded to: -``` -$(Pipeline.Workspace)/FSharpCompiler/bin/fsc (missing /artifacts/) -``` - -This caused the FSharpTargetsShim property to point to a non-existent path, which prevented Microsoft.FSharp.Targets from being imported. Without Microsoft.FSharp.Targets, Microsoft.Common.targets was not imported, which in turn prevented NuGet.targets from being imported - and NuGet.targets is where `_GetRestoreSettingsPerFramework` is defined. - -**Investigation Steps:** -1. Used `dotnet msbuild -pp` to compare preprocessed output with and without local compiler -2. Found NuGet.targets was referenced 9 times without local compiler, but only 2 times with local compiler -3. Traced the import chain: FSharpTargetsShim -> Microsoft.FSharp.NetSdk.targets -> Microsoft.FSharp.Targets -> Microsoft.Common.targets -> NuGet.targets -4. Discovered FSharpTargetsShim was pointing to path with doubled `/artifacts/artifacts/` -5. Realized artifact download path didn't match UseLocalCompiler.Directory.Build.props expectations - -**Solution:** -Changed the artifact download paths to include `/artifacts/bin/`: -```yaml -downloadPath: '$(Pipeline.Workspace)/FSharpCompiler/artifacts/bin/fsc' -downloadPath: '$(Pipeline.Workspace)/FSharpCompiler/artifacts/bin/FSharp.Core' -``` - -This ensures the directory structure matches what UseLocalCompiler.Directory.Build.props expects. - -**Lessons Learned:** -- UseLocalCompiler.Directory.Build.props has specific path expectations that must be matched exactly -- Use `dotnet msbuild -pp` to debug MSBuild import issues -- The `_GetRestoreSettingsPerFramework` error often indicates broken MSBuild import chain - -### 2. Git Submodule Initialization - -**Date:** 2025-11-28 - -**Symptom:** -Type provider source files were missing. - -**Root Cause:** -FSharpPlus uses git submodules for `FSharp.TypeProviders.SDK`. - -**Solution:** -1. Changed `git clone` to `git clone --recursive` to clone with submodules -2. Added explicit `git submodule update --init --recursive` after `git checkout` - -### 3. No Binary Log Files Generated - -**Date:** 2025-11-28 - -**Symptom:** -No `.binlog` files were being collected or published. - -**Root Cause:** -The build was failing early before any MSBuild commands could generate binlog files. - -**Solution:** -Fixing the path issues above allows the build to proceed, enabling binlog generation. diff --git a/TODO.md b/TODO.md deleted file mode 100644 index adc652c918..0000000000 --- a/TODO.md +++ /dev/null @@ -1,68 +0,0 @@ -# TODO: Azure DevOps Template for F# Compiler Regression Testing - -## Overview -Implement a reusable Azure DevOps template for F# compiler regression testing, integrated with the existing PR pipeline infrastructure. - -## Current Issues (from review feedback) - -### Build Failure & Binlog Issues -- [x] Fix binlog collection (was failing due to missing submodules) -- [x] Fix the `_GetRestoreSettingsPerFramework` issue (caused by missing git submodules) -- [x] Add `--recursive` flag to git clone command -- [x] Add explicit `git submodule update --init --recursive` after checkout -- [x] Document findings in OBSTACLES.md - -## Tasks - -### Infrastructure Setup -- [x] Analyze existing PR pipeline structure (`azure-pipelines-PR.yml`) -- [x] Review previous PR #18803 implementation details -- [x] Understand `UseLocalCompiler.Directory.Build.props` configuration - -### Implementation -- [x] Create `eng/templates/` directory -- [x] Create `eng/templates/regression-test-jobs.yml` template - - [x] Define parameters for testMatrix - - [x] Implement job that depends on EndToEndBuildTests - - [x] Add artifact download steps (FSharpCompilerFscArtifacts, FSharpCoreArtifacts, UseLocalCompilerProps) - - [x] Add third-party repo checkout step - - [x] Add .NET SDK installation step - - [x] Add Directory.Build.props setup step referencing standalone F# script - - [x] Add environment reporting step - - [x] Add build execution step - - [x] Add artifact publishing step - - [x] Add result reporting step - - [x] Add optional imageOverride per tested repo - -### F# Script for Repository Setup -- [x] Create standalone `eng/scripts/PrepareRepoForRegressionTesting.fsx` -- [x] Test script locally with FSharpPlus repository -- [x] Handle both existing and missing Directory.Build.props cases - -### Integration -- [x] Update `azure-pipelines-PR.yml`: - - [x] Modify EndToEndBuildTests to publish focused artifacts - - [x] Remove strategy/matrix section from EndToEndBuildTests - - [x] Add artifact publishing tasks for fsc, FSharp.Core, and UseLocalCompiler props - - [x] Move template invocation to stage level (outside common template) - - [x] Add template invocation with FSharpPlus test matrix - -### Documentation -- [x] Create `docs/regression-testing-pipeline.md` - - [x] Purpose and overview - - [x] How it works - - [x] Current test matrix - - [x] Adding new libraries - - [x] Pipeline configuration - - [x] Troubleshooting - - [x] Technical details - -### Validation -- [x] Verify YAML syntax is valid -- [x] Verify template structure matches Azure DevOps best practices -- [x] Ensure Release configuration is used throughout -- [x] Test F# script locally with FSharpPlus - -## References -- Previous PR: https://github.com/dotnet/fsharp/pull/18803 -- Files: `eng/templates/regression-test-jobs.yml`, `azure-pipelines-PR.yml`, `docs/regression-testing-pipeline.md`, `eng/scripts/PrepareRepoForRegressionTesting.fsx`