Provide errors for missing and unexpected parameters in azure pipelines #238

bryanbcook · 2023-10-14T18:35:19Z

This PR includes fixes and supporting tests for:

This also includes a dedicated xUnit test fixture for the SDK and a basic framework for validating azure pipeline YAML parsing. A test-fixture that automatically detects azure pipelines in testworkflows/azpipelines is included:

These regression tests are different than the current tests in the PR build validation in that they include assertions for error conditions.

Each regression test can optionally provide meta-data in the form of comments at the top of the test:

Name: Display name for the test (Optional)
ExpectedException. Class name for the exception if this scenario is designed to throw an exception. (Optional)
- Assumes that the class name is either from the GitHub.DistributedTask.ObjectTemplating or System namespace.
ExpectedErrorMessage: The error message for the expected exception. Only valid if the ExpectedException is specifed. (Optional)
LocalRepository: Mapping between repositoryAndRef and folder location relative to testworkflows/azpipelines (Optional).
- Multiple instances of this meta-data are allowed.
- Format must be project/name@ref=folder where folder is relative to testworkflows/azpipelines
- For YAML validation purposes, this meta-data is only required when the pipeline includes a template reference to this repository.

bryanbcook · 2023-10-14T19:07:01Z

src/Sdk.Tests/AzurePipelines/Util/PipelineExtensions.cs

+                    if (svalue.Contains("${{"))
+                    {
+                        eventInfo = new YamlDotNet.Serialization.ScalarEventInfo(new ReplaceDescriptor { Value = svalue.Replace("${{", "${{ '${{' }}"), Type = eventInfo.Source.Type, StaticType = eventInfo.Source.StaticType, ScalarStyle = eventInfo.Source.ScalarStyle });
+                    }


I borrowed this from the Blazor app you provided, but I haven't seen the scenario where this is used.

If you have something like that

steps: - script: echo '${{ '${{ something }}' }}' # We Escape the expression literal, comes from GitHub Actions forums

Then it is required to reescape, so the output keeps valid yaml for the template engine to feed the result back and process the result again.

I think the official preview api, doesn't consider this scenario at all.

bryanbcook · 2023-10-14T19:07:31Z

src/Sdk.Tests/AzurePipelines/Util/PipelineExtensions.cs

@@ -0,0 +1,61 @@
+namespace Runner.Server.Azure.Devops
+{
+    // TODO: Move to Sdk.AzurePipelines folder


Can we move this into the SDK project?

Moving it to the sdk is ok for me.

I think this Pipeline pipeline should be done via a new interface type with method ToContextData()

So any pipeline component can be converted to yaml, not just pipeline.

IContextDataProvider?

Sounds good to me, can be part of a later change

bryanbcook · 2023-10-14T19:11:03Z

src/Sdk/AzurePipelines/AzureDevops.cs

@@ -540,7 +540,6 @@ public class AzureDevops {
                throw new Exception("Provided undeclared parameters");
            }


can you provide an example where parameters is referred to as a mapping and not a sequence? I didn't touch this because I couldn't reproduce this scenario.

parameters: val: - a - b steps: - script: echo ${{ converttojson(parameters.val) }}

old syntax, I can run this online via azure pipelines. Only Provided undeclared parameters is never reported.

src/Sdk.Tests/AzurePipelines/Util/LocalFileProvider.cs

src/Sdk.Tests/AzurePipelines/Util/PipelineExtensions.cs

ChristopherHX · 2023-10-14T19:58:00Z

src/Test/Test.csproj

You broke the old Test project (see ci failure), could be due to breaking xunit changes.

I saw that you reverted the change. I changed it originally because I'm using Visual Studio 2022 and none of the tests were loading. I'm not seeing that issue anymore, but I still have tests in the Test project that are failing.

These tests are expected to fail if you never ran the ./dev test script, it download external tools like nodejs.

To be honest some of the original actions/runner tests fail for me locally due to my non English system language on Windows and that are more failures than you see.

I changed it originally because I'm using Visual Studio 2022 and none of the tests were loading. I'm not seeing that issue anymore

I fixed the Tests, by updating one specfic dependency of the test host and removing a lot of broken tests of partial merges / non relevant tests for trimmed package updates

@ChristopherHX fyi - i wasn't aware of the ./dev[.cmd] script.

I managed to get all the tests to pass. I ran the dotnet msbuild src/dir.proj -t:GenerateConstant which resolved one of the tests with BuildConstants.cs and running ./dev.cmd build and ./dev.cmd layout resolved the others.

ChristopherHX · 2023-10-14T19:59:32Z

testworkflows/azpipelines/extends-variables-error/pipeline.yml

@@ -0,0 +1,8 @@
+# ExpectedException: NotSupportedException
+# ExpectedErrorMessage: __built-in-schema.yml (Line: 40, Col:11): 'variables' is already defined


This error is not really expected, it needs to change.

(Line: 40, Col:11) is just wrong, __built-in-schema.yml is an implementation details of the ms engine.

It is expected. I introduced this exception in #227 to mimic the exception produced by Azure Pipelines when the calling pipeline and the pipeline being extended both declare variables.

I aggree an error is expected.

I disagree about the hardcoded file location

Fair. If there is a specific error then there is an opportunity to represent that feedback somewhere. For example, a vscode extension could highlight the offending line. I'll take a closer look.

ChristopherHX · 2023-10-14T20:56:20Z

Please add a calling entry in

runner.server/.github/workflows/build.yml

Lines 84 to 88 in c7ee3e7

    
               - name: Unit Tests 
        
                 if: matrix.runtime != 'linux-arm64' && matrix.runtime != 'linux-arm' && matrix.runtime != 'osx-arm64' 
        
                 run: | 
        
                   ${{ matrix.devScript }} test 
        
                 working-directory: src

so the GitHub Actions are actually running your tests.

Should be run by some dotnet sdk command.

ChristopherHX

Thank you for the unit test framework, I have changed it to work on linux and macOS (by not hardcoding the \ in path management)

bryanbcook added 2 commits October 14, 2023 14:22

azp: detect unrecognized parameters and missing required parameters

f7e20bb

add test framework for azpipelines

3202b00

bryanbcook commented Oct 14, 2023

View reviewed changes

Merge branch 'main' into feature/test-suite-parameter-fixes

80bd1cf

bryanbcook commented Oct 14, 2023

View reviewed changes

src/Sdk.Tests/AzurePipelines/Util/LocalFileProvider.cs Show resolved Hide resolved

ChristopherHX reviewed Oct 14, 2023

View reviewed changes

src/Sdk.Tests/AzurePipelines/Util/PipelineExtensions.cs Outdated Show resolved Hide resolved

ChristopherHX reviewed Oct 14, 2023

View reviewed changes

Revert Test.csproj (Due to build failure)

aca81a5

bryanbcook and others added 2 commits October 14, 2023 19:19

removed try/catch, fix for null reference types

ff4d268

fix tests on linux and run them in ci

7387dfe

ChristopherHX approved these changes Oct 15, 2023

View reviewed changes

ChristopherHX merged commit efdebf7 into ChristopherHX:main Oct 15, 2023
38 checks passed

bryanbcook deleted the feature/test-suite-parameter-fixes branch October 17, 2023 16:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Provide errors for missing and unexpected parameters in azure pipelines #238

Provide errors for missing and unexpected parameters in azure pipelines #238

bryanbcook commented Oct 14, 2023 •

edited

Loading

bryanbcook Oct 14, 2023

ChristopherHX Oct 14, 2023 •

edited

Loading

bryanbcook Oct 14, 2023

ChristopherHX Oct 14, 2023

bryanbcook Oct 15, 2023

ChristopherHX Oct 15, 2023

bryanbcook Oct 14, 2023

ChristopherHX Oct 14, 2023

ChristopherHX Oct 14, 2023

bryanbcook Oct 14, 2023

ChristopherHX Oct 15, 2023

ChristopherHX Oct 15, 2023

bryanbcook Oct 16, 2023

ChristopherHX Oct 14, 2023

bryanbcook Oct 14, 2023

ChristopherHX Oct 14, 2023 •

edited

Loading

bryanbcook Oct 16, 2023

ChristopherHX commented Oct 14, 2023

ChristopherHX left a comment

		@@ -540,7 +540,6 @@ public class AzureDevops {
		throw new Exception("Provided undeclared parameters");
		}

		@@ -0,0 +1,8 @@
		# ExpectedException: NotSupportedException
		# ExpectedErrorMessage: __built-in-schema.yml (Line: 40, Col:11): 'variables' is already defined

Provide errors for missing and unexpected parameters in azure pipelines #238

Provide errors for missing and unexpected parameters in azure pipelines #238

Conversation

bryanbcook commented Oct 14, 2023 • edited Loading

Choose a reason for hiding this comment

ChristopherHX Oct 14, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChristopherHX Oct 14, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChristopherHX commented Oct 14, 2023

ChristopherHX left a comment

Choose a reason for hiding this comment

bryanbcook commented Oct 14, 2023 •

edited

Loading

ChristopherHX Oct 14, 2023 •

edited

Loading

ChristopherHX Oct 14, 2023 •

edited

Loading