The build is successful even if Windows image tests fail

[lemeurherve]

Jenkins and plugins versions report

N/A

What Operating System are you using (both controller, and any agents involved in the problem)?

ci.jenkins.io

Reproduction steps

1. Open a pull request (ex: dummy PR for testing purpose #290)
2. Check the logs and notice errors in tests logs
3. Notice the build succeeded anyway 🤔

Expected Results

The Windows tests fail the build if they failed.

Actual Results

The build is successful even if the Windows tests failed.

Anything else?

Current error logs:

 Testing jdk11-nanoserver-1809...
 Pester v4.10.1
 Executing all tests in 'tests'
 
 Executing script C:\Jenkins\agent\workspace\ackaging_docker-ssh-agent_PR-290\tests\sshAgent.Tests.ps1
 
   Describing [jdk11-nanoserver-1809] image is present
     [-] builds image 448ms
       Expected 0, but got 1.
       57:         $exitCode | Should -Be 0
       at <ScriptBlock>, C:\Jenkins\agent\workspace\ackaging_docker-ssh-agent_PR-290\tests\sshAgent.Tests.ps1: line 57
 
   Describing [jdk11-nanoserver-1809] image has setup-sshd.ps1 in the correct location
 powershell.exe : docker: Invalid containerPort: jdk11-nanoserver-1809.
 At C:\Jenkins\agent\workspace\ackaging_docker-ssh-agent_PR-290@tmp\durable-d392bd33\powershellWrapper.ps1:3 char:1
 + & powershell -NoProfile -NonInteractive -ExecutionPolicy Bypass -Comm ...
 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     + CategoryInfo          : NotSpecified: (docker: Invalid...anoserver-1809.:String) [], RemoteException
     + FullyQualifiedErrorId : NativeCommandError
  
 See 'docker run --help'.
     [-] Error occurred in Describe block 0ms
       Expected $true, but got $null.
       64:         Is-ContainerRunning $global:CONTAINERNAME | Should -BeTrue
       at <ScriptBlock>, C:\Jenkins\agent\workspace\ackaging_docker-ssh-agent_PR-290\tests\sshAgent.Tests.ps1: line 64
       at Invoke-Blocks, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\SetupTeardown.ps1: line 135
       at Invoke-TestGroupSetupBlocks, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\SetupTeardown.ps1: line 121
       at DescribeImpl, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\Describe.ps1: line 209
 
   Describing [jdk11-nanoserver-1809] checking image metadata
     [-] has correct volumes 64ms
       Expected 0, but got 1.
       80:         $exitCode | Should -Be 0
       at <ScriptBlock>, C:\Jenkins\agent\workspace\ackaging_docker-ssh-agent_PR-290\tests\sshAgent.Tests.ps1: line 80
     [-] has the source GitHub URL in docker metadata 69ms
       Expected 0, but got 1.
       88:         $exitCode | Should -Be 0
       at <ScriptBlock>, C:\Jenkins\agent\workspace\ackaging_docker-ssh-agent_PR-290\tests\sshAgent.Tests.ps1: line 88
 
   Describing [jdk11-nanoserver-1809] image has correct version of java installed and in the PATH
 docker: Invalid containerPort: jdk11-nanoserver-1809.
 See 'docker run --help'.
     [-] has java installed and in the path 66ms
       Expected 0, but got 1.
       101:         $exitCode | Should -Be 0
       at <ScriptBlock>, C:\Jenkins\agent\workspace\ackaging_docker-ssh-agent_PR-290\tests\sshAgent.Tests.ps1: line 101
 
   Describing [jdk11-nanoserver-1809] create agent container with pubkey as argument
 "docker run" requires at least 1 argument.
 See 'docker run --help'.
 
 Usage:  docker run [OPTIONS] IMAGE [COMMAND] [ARG...]
 
 Create and run a new container from an image
     [-] Error occurred in Describe block 0ms
       Expected $true, but got $null.
       118:         Is-ContainerRunning $global:CONTAINERNAME | Should -BeTrue
       at <ScriptBlock>, C:\Jenkins\agent\workspace\ackaging_docker-ssh-agent_PR-290\tests\sshAgent.Tests.ps1: line 118
       at Invoke-Blocks, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\SetupTeardown.ps1: line 135
       at Invoke-TestGroupSetupBlocks, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\SetupTeardown.ps1: line 121
       at DescribeImpl, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\Describe.ps1: line 209
 
   Describing [jdk11-nanoserver-1809] create agent container with pubkey as envvar
 docker: No port specified: :/tcp<empty>.
 See 'docker run --help'.
     [-] Error occurred in Describe block 0ms
       Expected $true, but got $null.
       135:         Is-ContainerRunning $global:CONTAINERNAME | Should -BeTrue
       at <ScriptBlock>, C:\Jenkins\agent\workspace\ackaging_docker-ssh-agent_PR-290\tests\sshAgent.Tests.ps1: line 135
       at Invoke-Blocks, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\SetupTeardown.ps1: line 135
       at Invoke-TestGroupSetupBlocks, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\SetupTeardown.ps1: line 121
       at DescribeImpl, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\Describe.ps1: line 209
 
   Describing [jdk11-nanoserver-1809] create agent container like docker-plugin with '/usr/sbin/sshd -D -p 22' as argument
 docker: No port specified: :/tcp<empty>.
 See 'docker run --help'.
     [-] Error occurred in Describe block 0ms
       Expected $true, but got $null.
       154:         Is-ContainerRunning $global:CONTAINERNAME | Should -BeTrue
       at <ScriptBlock>, C:\Jenkins\agent\workspace\ackaging_docker-ssh-agent_PR-290\tests\sshAgent.Tests.ps1: line 154
       at Invoke-Blocks, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\SetupTeardown.ps1: line 135
       at Invoke-TestGroupSetupBlocks, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\SetupTeardown.ps1: line 121
       at DescribeImpl, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\Describe.ps1: line 209
 
   Describing [jdk11-nanoserver-1809] build args
     [-] uses build args correctly 11.77s
       Expected 0, but got 1.
       179:         $exitCode | Should -Be 0
       at <ScriptBlock>, C:\Jenkins\agent\workspace\ackaging_docker-ssh-agent_PR-290\tests\sshAgent.Tests.ps1: line 179
 Tests completed in 142.81s
 Tests Passed: 0, Failed: 9, Skipped: 0, Pending: 0, Inconclusive: 0 
 Testing jdk11-windowsservercore-ltsc2019...
 Pester v4.10.1
 Executing all tests in 'tests'
 
 Executing script C:\Jenkins\agent\workspace\ackaging_docker-ssh-agent_PR-290\tests\sshAgent.Tests.ps1
 
   Describing [jdk11-windowsservercore-ltsc2019] image is present
     [-] builds image 70ms
       Expected 0, but got 1.
       57:         $exitCode | Should -Be 0
       at <ScriptBlock>, C:\Jenkins\agent\workspace\ackaging_docker-ssh-agent_PR-290\tests\sshAgent.Tests.ps1: line 57
 
   Describing [jdk11-windowsservercore-ltsc2019] image has setup-sshd.ps1 in the correct location
 docker: Invalid containerPort: jdk11-windowsservercore-ltsc2019.
 See 'docker run --help'.
     [-] Error occurred in Describe block 0ms
       Expected $true, but got $null.
       64:         Is-ContainerRunning $global:CONTAINERNAME | Should -BeTrue
       at <ScriptBlock>, C:\Jenkins\agent\workspace\ackaging_docker-ssh-agent_PR-290\tests\sshAgent.Tests.ps1: line 64
       at Invoke-Blocks, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\SetupTeardown.ps1: line 135
       at Invoke-TestGroupSetupBlocks, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\SetupTeardown.ps1: line 121
       at DescribeImpl, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\Describe.ps1: line 209
 
   Describing [jdk11-windowsservercore-ltsc2019] checking image metadata
     [-] has correct volumes 64ms
       Expected 0, but got 1.
       80:         $exitCode | Should -Be 0
       at <ScriptBlock>, C:\Jenkins\agent\workspace\ackaging_docker-ssh-agent_PR-290\tests\sshAgent.Tests.ps1: line 80
     [-] has the source GitHub URL in docker metadata 58ms
       Expected 0, but got 1.
       88:         $exitCode | Should -Be 0
       at <ScriptBlock>, C:\Jenkins\agent\workspace\ackaging_docker-ssh-agent_PR-290\tests\sshAgent.Tests.ps1: line 88
 
   Describing [jdk11-windowsservercore-ltsc2019] image has correct version of java installed and in the PATH
 docker: Invalid containerPort: jdk11-windowsservercore-ltsc2019.
 See 'docker run --help'.
     [-] has java installed and in the path 64ms
       Expected 0, but got 1.
       101:         $exitCode | Should -Be 0
       at <ScriptBlock>, C:\Jenkins\agent\workspace\ackaging_docker-ssh-agent_PR-290\tests\sshAgent.Tests.ps1: line 101
 
   Describing [jdk11-windowsservercore-ltsc2019] create agent container with pubkey as argument
 "docker run" requires at least 1 argument.
 See 'docker run --help'.
 
 Usage:  docker run [OPTIONS] IMAGE [COMMAND] [ARG...]
 
 Create and run a new container from an image
     [-] Error occurred in Describe block 0ms
       Expected $true, but got $null.
       118:         Is-ContainerRunning $global:CONTAINERNAME | Should -BeTrue
       at <ScriptBlock>, C:\Jenkins\agent\workspace\ackaging_docker-ssh-agent_PR-290\tests\sshAgent.Tests.ps1: line 118
       at Invoke-Blocks, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\SetupTeardown.ps1: line 135
       at Invoke-TestGroupSetupBlocks, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\SetupTeardown.ps1: line 121
       at DescribeImpl, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\Describe.ps1: line 209
 
   Describing [jdk11-windowsservercore-ltsc2019] create agent container with pubkey as envvar
 docker: No port specified: :/tcp<empty>.
 See 'docker run --help'.
     [-] Error occurred in Describe block 0ms
       Expected $true, but got $null.
       135:         Is-ContainerRunning $global:CONTAINERNAME | Should -BeTrue
       at <ScriptBlock>, C:\Jenkins\agent\workspace\ackaging_docker-ssh-agent_PR-290\tests\sshAgent.Tests.ps1: line 135
       at Invoke-Blocks, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\SetupTeardown.ps1: line 135
       at Invoke-TestGroupSetupBlocks, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\SetupTeardown.ps1: line 121
       at DescribeImpl, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\Describe.ps1: line 209
 
   Describing [jdk11-windowsservercore-ltsc2019] create agent container like docker-plugin with '/usr/sbin/sshd -D -p 22' as argument
 docker: No port specified: :/tcp<empty>.
 See 'docker run --help'.
     [-] Error occurred in Describe block 0ms
       Expected $true, but got $null.
       154:         Is-ContainerRunning $global:CONTAINERNAME | Should -BeTrue
       at <ScriptBlock>, C:\Jenkins\agent\workspace\ackaging_docker-ssh-agent_PR-290\tests\sshAgent.Tests.ps1: line 154
       at Invoke-Blocks, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\SetupTeardown.ps1: line 135
       at Invoke-TestGroupSetupBlocks, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\SetupTeardown.ps1: line 121
       at DescribeImpl, C:\Program Files\WindowsPowerShell\Modules\Pester\4.10.1\Functions\Describe.ps1: line 209
 
   Describing [jdk11-windowsservercore-ltsc2019] build args
     [-] uses build args correctly 45.65s
       Expected 0, but got 125.
       182:         $exitCode | Should -Be 0
       at <ScriptBlock>, C:\Jenkins\agent\workspace\ackaging_docker-ssh-agent_PR-290\tests\sshAgent.Tests.ps1: line 182
 Tests completed in 175.38s
 Tests Passed: 0, Failed: 9, Skipped: 0, Pending: 0, Inconclusive: 0 
 Build finished successfully

See https://ci.jenkins.io/job/Packaging/job/docker-ssh-agent/job/PR-290/1/execution/node/40/log/

[lemeurherve]

Addressed in #289