Parameter testfilter on ALOpsAppTest not working?

[lvanvugt]

As per #310 I have applied the parameter testfilter (testfilter: '<>0') on ALOpsAppTest to make sure all tests residing in the database (per step ALOpsAppPublish) will be run. I am, however, getting the following error:

Apply filter for Test-Codeunits: [<>0]
##[error]Run-Test-Exception: Exception calling ".ctor" with "2" argument(s): "Value cannot be null.

the used yaml

name: $(Build.BuildId)

trigger:
 branches:
   include:
     - master/TestFixtureInitializer

variables:
- group: 'PipelineVariables'

pool: TestAutomationExamples

jobs:
- job: default
  timeoutInMinutes: 600
  pool: TestAutomationExamples

  steps:
  - checkout: self
    clean: true 

  - task: ALOpsDockerCreate@1
    displayName: 'ALOPS - Create Docker Image' # It will reuse the image if already exists, or build one if it doesn't.
    inputs:
      forcecreateimage: $(forceCreateImage)
      licensefile: '$(LuxBeheerBcDevLicense)'

  - task: ALOpsDockerStart@1
    displayName: 'ALOPS - Start Docker Container' # No need to provide any details - it will get the details from previous step

  - task: ALOpsDockerWait@1
    displayName: 'ALOPS - Wait for Docker Container to start'
    inputs:
      search_string: 'Ready for connections!'

  - task: ALOpsDockerExec@1
    displayName: 'Uninstall MS apps (but leave Application apps)'
    inputs:
      inline_script: |
        Get-NAVAppInfo -ServerInstance BC -tenant default -tenantspecificproperties | 
          where isinstalled -eq $true | 
            Where Publisher -eq Microsoft | 
              where {$_.Name -ne 'Base Application' -and $_.Name -ne 'System Application' -and $_.Name -ne 'Application'}  | 
                uninstall-navapp -tenant default -force

  - task: ALOpsAppPublish@1
    displayName: 'ALOPS - Install AL TestTool' # install necessary dependent apps for testability
    inputs:
      usedocker: true
      installaltesttool: true
      skip_verification: true
      install_al_app_names: |
        Any
        Library Assert
        Library Variable Storage
        System Application Test Library
        Tests-TestLibraries
        Tests-Cash Flow
        Tests-Cost Accounting
        Tests-CRM integration
        Test Runner

  - task: ALOpsAppCompiler@1
    displayName: 'ALOPS - Compile Extension: App'
    inputs:
      usedocker: true
      nav_app_version: '1.0.[yyyyWW].*'
      targetproject: 'TestFixtureInitializer/app/app.json'
      app_file_suffix: '_APP'                                 # A trick to uniquely identify the APP (from the TEST-app)

  - task: ALOpsAppPublish@1
    displayName: 'ALOPS - Publish Extension'
    inputs:
      usedocker: true
      nav_artifact_app_filter: '*.app'
      skip_verification: true

  - task: ALOpsAppCompiler@1
    displayName: 'ALOPS - Compile Extension: Test-App'
    inputs:
      usedocker: true
      nav_app_version: '1.0.[yyyyWW].*'
      targetproject: 'TestFixtureInitializer/test/app.json'
      failed_on_warnings: true    
      app_file_suffix: '_TEST'                                 # A trick to uniquely identify the Test-App (from the App)

  - task: ALOpsAppPublish@1
    displayName: 'ALOPS - Publish Extension'
    inputs:
      usedocker: true
      nav_artifact_app_filter: '*.app'
      skip_verification: true   

  - task: ALOpsAppTest@1
    displayName: 'ALOPS - Run Tests'
    inputs:
      usedocker: true
      testfilter: '<>0'
      testsuite: 'DEFAULT'
      disabledtests: '$(Build.SourcesDirectory)\TestFixtureInitializer\Test\DisabledTests.json'
      import_testtoolkit: false
      import_action: 'Skip'
      show_available_tests: false
      testpage: '130455'
      failed_test_action: 'Ignore'
    continueOnError: true
    timeoutInMinutes: 90

  - task: PublishPipelineArtifact@1
    displayName: 'Publish Artifact Test Results **/TestResults.xml'
    inputs:
      targetPath: '$(System.DefaultWorkingDirectory)/TestResults.xml'
      artifact: 'TestResults.xml'
      publishLocation: 'pipeline'

  - task: PublishTestResults@2
    displayName: 'Publish Test Results **/TestResults.xml'
    inputs:
      testResultsFormat: XUnit
      testResultsFiles: '**/TestResults.xml'
      failTaskOnFailedTests: true
      testRunTitle: 'BC Test Results: $(Build.BuildId)'

  - task: ALOpsDockerRemove@1
    displayName: 'ALOPS - Remove Docker Container'
    enabled: true
    condition: always()
    inputs:
      createsqlbackup: '$(createSqlBackup)'

the output

ALOPS - Run Tests
*** Import Modules in Docker Session
*** Import UI Client [C:\Run\DevOps\BCAppTest\bin\Microsoft.Dynamics.Framework.UI.Client.dll]
VERBOSE: Loading module from path 'C:\Run\DevOps\BCAppTest\bin\Microsoft.Dynamics.Framework.UI.Client.dll'.
*** Import NewtonSoft JSON [C:\Program Files\Microsoft Dynamics NAV\170\Service\Newtonsoft.Json.dll]
*** Import Modules in Docker Session: class_ClientContext
*** Test Context Url: https://c25c1ebbc269/BC/cs?tenant=default
*** Setup BC Credentials in Docker
*** Resolve Docker Credentials
*** Install Test-Tool extension for creating Test-Suites
*** Import NAV/BC Management DLL's
*** Loading assemblies for ServiceTier [BC].
*** Loading assemblies from: [C:\program files\microsoft dynamics nav\170\service]
*** Selected module: [C:\program files\microsoft dynamics nav\170\service\Microsoft.Dynamics.Nav.Apps.Management.dll]
*** Loading module: [C:\program files\microsoft dynamics nav\170\service\Microsoft.Dynamics.Nav.Apps.Management.dll]
*** Selected module: [C:\program files\microsoft dynamics nav\170\service\Microsoft.Dynamics.Nav.Management.dll]
*** Loading module: [C:\program files\microsoft dynamics nav\170\service\Microsoft.Dynamics.Nav.Management.dll]
*** Add Test-Runner Extension for ALOps
*** Installing Applications:

Publisher Name Path ProcessOrder
--------- ---- ---- ------------
          Path                  


*** Get App Info: C:\Run\DevOps\BCAppTest\HodorNV_ALOpsTestRunner_1.0.0.0.app

*** Get App Tenant Info
*** Publish App: [Hodor NV / ALOps Test Runner]
*** Sync App: [Hodor NV / ALOps Test Runner]
*** Install App: [Hodor NV / ALOps Test Runner]
***

*** Install Test-Tool extension for creating Test-Suites Finished
*** Set TCP Keep-Alive
*** Start TestSuite
*** Running TestSuite with authentication [NavUserPassword] with User [admin]
*** Run warm-up with page [130455]
*** Warm-up Client Context
*** Create Client Context
*** Add Tests to Default-TestSuite with filter [<>0]
Apply filter for Test-Codeunits: [<>0]
##[error]Run-Test-Exception: Exception calling ".ctor" with "2" argument(s): "Value cannot be null.
Parameter name: logicalControl"
*** Get XUnit Test file from [] to [C:\vsts-agent-win-x64-2.177.1\_work\32\s\]
##[error]Cannot bind argument to parameter 'Path' because it is null.
Finishing: ALOPS - Run Tests

Expected behavior
All tests are run.

[waldo1001]

But Luc,

I told you about this (#310 (comment)).
And we agreed to take the other route: run tests by ExtensionId..

[lvanvugt]

I guess I read it differently: it should work right now (as #310 has the released tag), but for future sake we take the ExtensionID route.
So, it does not work at al at this moment?

[waldo1001]

Exactly - apparently, in some versions of BC, the app we use to hack this possibility, doesn't work (comes with an unexplained "dialog" - making it impossible to install the app we need to import the tests).. .

So .. we'll abandon that idea, and fully go for the ExtensionId approach.

[lvanvugt]

@waldo1001 any update on this?

[AdminHodor]

Dear @lvanvugt ,

Please check out our latest release v1.441 which included this feature.

- task: ALOpsAppTest@1
  displayName: 'ALOPS - Run TestSuite'
  inputs:
    usedocker: true
    import_action: "Skip"
    import_testtoolkit: false
    testpage: '130455'
    extensionid: '9bf23d52-8194-4857-8896-5e48b24493f6'
    resultfilename: 'TestResults_Job.xml'
    failed_test_action: 'Ignore'
  continueOnError: true   
 
- task: PublishTestResults@2
  displayName: 'Publish Test Results **/TestResults_Job.xml'
  inputs:
    testResultsFormat: XUnit
    testResultsFiles: '**/TestResults_Job.xml'
    testRunTitle: 'BC Test Results: $(Build.BuildId)'

Kind regards,

[lvanvugt]

Will do. Might not be soon, but in the course of updating my book this will be picked up in the coming weeks.

[lvanvugt]

Tested and found extensionid parameter working fine.