Skip to content
This plugin integrates Jenkins with ECU-TEST and generates reports on automated test execution.
Java HTML Groovy Other
Branch: master
Clone or download
Pull request Compare This branch is 546 commits ahead, 1 commit behind tracetronic:master.

README.md

TraceTronic ECU-TEST Plugin

Build Status FOSSA Status Dependabot Status Jenkins Plugin Jenkins Release Plugin Installs License

This plugin integrates Jenkins with ECU-TEST and generates reports on automated test execution.

ECU-TEST Logo

ECU-TEST is a test automation software for the validation of embedded systems in automotive environments developed by TraceTronic GmbH.
This software executes regression tests which are essential for validating complex technical products such as electronic control units (ECUs).
It supports standardized access to a broad range of test tools and provides automation of distributed test environments (SiL – MiL – HiL – vehicle).

Table of Contents

Features

  • Provides an easy integration and control of ECU-TEST and the Tool-Server with Jenkins
  • Enables the execution of ECU-TEST packages and projects with their respective configurations

Moreover publishing test results is included in form of:

  • Linking the TRF reports
  • Generating various report formats
  • Displaying test result trend graphs
  • Uploading ATX test reports to TEST-GUIDE
  • Parsing the ECU-TEST logs for warnings and errors

Advanced workflows integrated for:

Configuration

ECU-TEST configuration

ECU-TEST installations are administrated in the Jenkins system configuration or starting with Jenkins 2.0 in the global tool configuration at section "ECU-TEST". An installation entry is specified by an arbitrary name and the path to the installation directory. The execution on a Jenkins slave requires the adaption of the ECU-TEST installation directory on the slave configuration page.

ECU-TEST

TEST-GUIDE configuration

TEST-GUIDE is also configured in the Jenkins system configuration at section "TEST-GUIDE". A new entry requires both an arbitrary name and the selection of a previously configured ECU-TEST installation. Further settings like the server or ATX specific configuration can be found in the advanced block.

TEST-GUIDE

Job configuration

The plugin provides several new build steps:

Build Steps

Note:

  • All build steps have an "Advanced" button through which more options can be accessed
  • All text fields can be parameterized and are resolved at build execution
    • Syntax: $parameter or ${parameter}

The post-build actions are extended by these test report related post-build steps:

Post-Build Steps

Job execution

  • After job configuration is done a new build can be triggerd by clicking on the "Build now" button
  • The console log contains detailed information per build step execution
  • The test results are published both on the build page and the job page based on the latest build

Demo Job Demo Build Demo Log

Environment variables

  • This plugin also exposes the tool and test related information of the current build as environment variables
  • These variables are composed of $TT<TOOL | TEST>_n where n is the 0-based index of the related build steps in the job configuration
  • The summary of all generated environment variables is also shown on the build page
Tool related variables
Environment variable Description
TT_TOOL_NAME Name of the selected ECU-TEST installation
TT_TOOL_INSTALL Full path to related ECU-TEST executable
TT_TOOL_VERSION Program version of selected ECU-TEST installation
TT_TOOL_WORKSPACE ECU-TEST workspace directory
TT_TOOL_SETTINGS ECU-TEST settings directory
TT_TOOL_DEBUG ECU-TEST debug mode
TT_TOOL_TIMEOUT Maximum execution time
TT_TOOL_LAST_TBC Last loaded TBC file path
TT_TOOL_LAST_TCF Last loaded TCF file path
Test related variables
Environment variable Description
TT_TEST_NAME Name of the executed test
TT_TEST_FILE Full path to related test file
TT_TEST_TYPE Type of related test (PACKAGE, PROJECT)
TT_TEST_DESCRIPTION Test description
TT_TEST_TBC Full path to test bench configuration file
TT_TEST_TCF Full path to test configuration file
TT_TEST_TIMEOUT Maximum execution time
TT_TEST_RESULT Test result (SUCCESS, INCONCLUSIVE, FAILED, ERROR, NONE)
TT_TEST_REPORT Full path to test report directory

Downstream report generation

The concept of downstream report generation allows to execute the report generators like ATX, UNIT, HTML etc. in a downstream-based project. Therefore the report artifacts like TRF reports etc. must be archived after the test execution stage and will then be copied into a separate workspace of the triggered downstream project.

This downstream project acts like a central reporting service which waits for inputs from other test executions. It will be an ease to scale this service with help of Jenkins parallelization and distribution features. The big advantage of this approach is to save time for the report generation and uploading process on expensive testing resources like HiL test benches. These resources can be directly used for the next test.

Downstream Report Generation

Example configurations

Preconditions

To prevent duplicate reports the build workspace should be cleaned before running the tests and before copying artifacts inside of the downstream project. This can be achieved by using clean checkouts from repository or using the Workspace Cleanup Plugin when preparing the build environment.

To transfer the report artifacts from one node to another the Copy Artifact Plugin is required for freestyle projects. When using pipelines also stash/unstash commands can be used.

Freestyle configuration

Executing tests and archiving the artifacts (upstream)

Upstream Build Steps Upstream Post-Build Steps

Copying artifacts and generating ATX report (downstream)

Downstream Build Steps Downstream Post-Build Steps

Pipeline configuration

Upstream pipeline definition
node('windows') {
    deleteDir()

    startET 'ECU-TEST'
    testPackage testFile: 'sample.pkg', testConfig: [tbcFile: 'sample.tbc', tcfFile: 'sample.tcf']
    stopET 'ECU-TEST'

    archiveArtifacts 'TestReports/**/*.trf'
    build job: 'downstream', wait: false
}
Downstream pipeline definition
node('windows') {
    deleteDir()

    copyArtifacts filter: 'TestReports/**/*.trf', projectName: 'upstream', selector: lastSuccessful()

    // Class notation is required for publisher steps inside downstream wrapper!
    // Available: ATXPublisher, ETLogPublisher, JUnitPublisher, ReportGeneratorPublisher, TMSPublisher, TRFPublisher
    downstreamPublisher workspace: '', publishers: [
        [$class: 'ATXPublisher', atxName: 'TEST-GUIDE'],
        [$class: 'JUnitPublisher', toolName: 'ECU-TEST']]
}

Downstream trace analysis

In addition to downstream report generation the plugin allows to run trace analyses in a downstream-based project. Therefore the test cases are executed without trace analysis at first. Afterwards all trace analysis artifacts like analysis job files, traces, trace step templates and TRF reports etc. must be archived and will then be copied into a separate workspace of the triggered downstream project.

This downstream project acts like a central trace analysis service which waits for inputs from other test executions and is designed to be highly scalable. The huge advantage of better ressource efficiency is the same as for downstream report generation and can also be used in combination.

Downstream Trace Analysis

Example configurations

Preconditions

Same as in downstream report generation.

Freestyle configuration

Executing test cases without trace analysis and archiving the artifacts (upstream)

Upstream Build Steps Upstream Post-Build Steps

Copying artifacts and running the trace analysis (downstream)

Downstream Build Steps Downstream Post-Build Steps

Pipeline configuration

Upstream pipeline definition
node('windows') {
    deleteDir()

    startET 'ECU-TEST'
    testPackage testFile: 'sample.pkg', testConfig: [tbcFile: 'sample.tbc', tcfFile: 'sample.tcf'], packageConfig: [runTraceAnalysis: false]
    stopET 'ECU-TEST'

    archiveArtifacts 'TestReports/**/*.trf, TestReports/**/*.ajob, Packages/**, Traces/**, TraceStepTemplates/**'
    build job: 'downstream', wait: false
}
Downstream pipeline definition
node('windows') {
    deleteDir()

    copyArtifacts filter: 'TestReports/**/*.trf, TestReports/**/*.ajob, Packages/**, Traces/**, TraceStepTemplates/**', projectName: 'upstream', selector: lastSuccessful()

    // Class notation is required for publisher steps inside downstream wrapper!
    downstreamPublisher workspace: '', publishers: [
        [$class: 'TraceAnalysisPublisher', toolName: 'ECU-TEST', mergeReports: true]]
}

Extensions

Job DSL

The Job DSL Plugin allows the programmatic creation of projects using a DSL. This plugin provides some specific DSL extensions.

Furthermore, if there is a method which is currently not supported then Job DSL also provides an auto-generator.

Controlling tools
steps {
  // Start ECU-TEST
  startET(String installation) {
    workspaceDir(String directory)
    settingsDir(String directory)
    timeout(int | String timeout = 120)
    debugMode(boolean debugMode = false)
    keepInstance(boolean keepInstance = false)
 }
  // Stop ECU-TEST
  stopET(String installation) {
    timeout(int | String timeout = 30)
  }
  // Start Tool-Server
  startTS(String installation) {
    toolLibsIni(String toolLibsPath)
    tcpPort(int | String port = 5017)
    timeout(int | String timeout = 120)
    keepInstance(boolean keepInstance = false)
 }
  // Stop Tool-Server
  stopTS(String installation)
    timeout(int | String timeout = 30)
  }
}
Test execution
steps {
  // Run package
  testPackage(String packageFile) {
    testConfig {
      tbcFile(String tbcFile)
      tcfFile(String tcfFile)
      forceReload(boolean forceReload = false)
      loadOnly(boolean loadOnly = false)
      constants {
        constant(String name, String value)
        constant {
          name(String name)
          value(String value)
        }
      }
    }
    packageConfig {
      runTest(boolean runTest = true)
      runTraceAnalysis(boolean runTraceAnalysis = true)
      parameters {
        parameter(String name, String value)
        parameter {
          name(String name)
          value(String value)
        }
      }
    }
    executionConfig {
      timeout(int | String timeout = 3600)
      checkTestFile(boolean checkTestFile = true)
      stopOnError(boolean stopOnError = true)
    }
  }
  // Run project
  testProject(String projectFile) {
    testConfig { (...) }
    projectConfig {
      execInCurrentPkgDir(boolean currentPkgDir = false)
      filterExpression(String expression)
      // modes: NO_EXECUTION(0), SEQUENTIAL_EXECUTION(1),
      // PARALLEL_EXECUTION(2), SEPARATE_SEQUENTIAL_EXECUTION(5),
      // SEPARATE_PARALLEL_EXECUTION(6), NO_TESTCASE_EXECUTION(9)
      jobExecutionMode(int | String execMode = 1)
    }
    executionConfig { (...) }
  }
  // Run test folder
  testFolder(String testFolder) {
    // modes: PACKAGES_ONLY, PROJECTS_ONLY, PACKAGES_AND_PROJECTS
    scanMode(String mode = 'PACKAGES_AND_PROJECTS')
    recursiveScan(boolean recursive = false)
    testConfig { (...) }
    packageConfig { (...) }
    projectConfig { (...) }
    executionConfig { (...) }
  }
}
Test management
steps {
  // Import projects
  importProjects {
    importFromArchive(String archivePath, String importPath, String importConfigPath, boolean replaceFiles)
    importFromArchive(String archivePath) {
      importPath(String importPath)
      importConfigPath(String importConfigPath)
      replaceFiles(boolean replaceFiles)
    }
    importFromTMS(String credentialsId, String projectPath, String importPath, int | String timeout = 60)
    importFromTMS(String credentialsId, String projectPath) {
      importPath(String importPath)
      timeout(int | String timeout = 60)
    }
    importFromTMSDir(String credentialsId, String projectDirPath, String importPath, int | String timeout = 60)
    importFromTMSDir(String credentialsId, String projectDirPath) {
      importPath(String importPath)
      timeout(int | String timeout = 60)
    }
    importAttributesFromTMS(String credentialsId, String projectPath, int | String timeout = 60)
    importAttributesFromTMS(String credentialsId, String projectPath) {
      timeout(int | String timeout = 60)
    }
  }
  // Import packages
  importPackages {
    importFromTMS { (...) }
    importFromTMSDir { (...) }
    importAttributesFromTMS { (...) }
  }
  // Export projects
  exportProjects {
    exportToTMS(String credentialsId, String projectPath, String exportPath, boolean createNewPath, int | String timeout = 60)
    exportToTMS(String credentialsId, String projectPath) {
      exportPath(String exportPath)
      createNewPath(boolean createNewPath)
      timeout(int | String timeout = 60)
    }
    exportAttributesToTMS(String credentialsId, String projectPath, int | String timeout = 60)
    exportAttributesToTMS(String credentialsId, String projectPath) {
      timeout(int | String timeout = 60)
    }
  }
  // Export packages
  exportPackages {
    exportToTMS { (...) }
    exportAttributesToTMS { (...) }
  }
}
Publishing reports
publishers {
  // Publish ATX reports
  publishATX(String atxInstallation) {
    // Archive options
    allowMissing(boolean allowMissing = false)
    runOnFailed(boolean runOnFailed = false)
    archiving(boolean archiving = true)
    keepAll(boolean keepAll = true)
  }
  // Publish TRF reports
  publishTRF() {
    // Archive options...
  }
  // Publish UNIT reports
  publishUNIT(String installation) {
    unstableThreshold(double threshold = 0)
    failedThreshold(double threshold = 0)
    // Archive options...
  }
  // Publish ECU-TEST logs
  publishETLogs() {
    unstableOnWarning(boolean markUnstable = false)
    failedOnError(boolean markFailed = false)
    testSpecific(boolean testSpecific = false)
    // Archive options...
  }
  // Publish Report Generators
  publishGenerators(String installation) {
    generators {
      generator(String generatorName) {
        settings {
          setting(String name, String value)
          setting {
            name(String name)
            value(String value)
          }
        }
      }
    }
    customGenerators {
       customGenerator(String generatorName) { (...) }
    }
    // Archive options...
  }
  // Publish to Test Management System
  publishTMS(String installation, String credentialsId) {
    timeout(int | String timeout = 60)
    // Archive options...
  }
}

Pipeline

The Pipeline Plugin allows to orchestrate automation, simple or complex, using a DSL. This plugin supports the use of all provided build steps and post-build actions from within a Jenkins Pipeline build. The appropriate DSL syntax for these steps and actions can be easily generated with help of the Pipeline Snippet Generator using the related custom pipeline step or the 'step' command, when using plugin version prior to 1.13.

Basic pipeline example
node('windows') {
    stage('Start Tools') {
        startET 'ECU-TEST'
    }
    stage('Test Execution') {
        testPackage testFile: 'sample.pkg', testConfig: [tbcFile: 'sample.tbc', tcfFile: 'sample.tcf']
        testProject testFile: 'sample.prj', testConfig: [tbcFile: 'sample.tbc', tcfFile: 'sample.tcf']
    }
    stage('Publish Reports') {
        publishATX 'TEST-GUIDE'
    }
    stage('Stop Tools') {
        stopET 'ECU-TEST'
    }
}
Dynamic ECU-TEST pipeline example
node('windows') {
    // ECU-TEST installation using global variable ET
    def et = ET.newInstallation toolName: 'ECU-TEST', installPath: 'C:\\Program Files\\ECU-TEST 8.0'

    // or getting existing installation from global tool configuration
    def et = ET.installation('ECU-TEST')

    // Start ECU-TEST
    et.start('C:\\Data\\workspace', 'C:\\Data\\settings')

    // Stop ECU-TEST
    et.stop()
}
Dynamic TEST-GUIDE pipeline example
node('windows') {
    // Start tools, execute tests
    ...

    // TEST-GUIDE server instantiation using global variable ATX
    def atx = ATX.newServer atxName: 'TEST-GUIDE', toolName: 'ECU-TEST',
                            fullServerURL: 'http://localhost:8085', uploadToServer: false,
                            authKey: 'xxx', projectId: '1'

    // or getting existing instance from global configuration
    def atx = ATX.server('TEST-GUIDE')

    // Override one or multiple settings explicitly
    atx.overrideSetting('useHttpsConnection', true)
    atx.overrideSettings([serverURL: '127.0.0.1', useHttpsConnection: true])

    // Publish ATX reports directly
    atx.publish()
}

Issues

To report a bug or request an enhancement to this plugin please raise a new GitHub issue or create a ticket in JIRA.

Known limitations

When using the plugin in a slave-based setup (especially running the slave at command line via JNLP) you also need to restart the slave when restarting the master. This is due to already loaded libraries, for more information see this blog post and related issue JENKINS-31961.

Release 1.8 implemented an extension point that will workaround this issue. In order to work a new task has to be created in the Windows Task Scheduler named RESTART_JENKINS_SLAVE and configured with actions how to restart the slave.

FAQ

When encountering problems or error messages, please check the installed plugin version at first and update to the most recent version, if any. If the problem still exists search the following list of issues for possible solutions, otherwise you are asked to create an issue.

🔵 Common issues

The configured tests are aborted after 60 minutes.

There is a default maximum execution timeout of 3600 seconds that can be changed in the advanced build step section under execution settings. Setting the value to 0 disables the timeout.

The ECU-TEST GUI is not visible or there are problems with external tool connections.

When running the Jenkins slave (or master) as a Windows service that executes ECU-TEST then by default this service runs in scope of the SYSTEM user in a background Windows session. Therefore all processes started in this session are not visible to the normal user session and there could be problems due to insufficient user rights.

In summary it is NOT recommended to install the slave as a service but to run the slave from command line or to launch via Java Web Start.

In order to be able to use this prefered approach follow these instructions:

  • Open "Global Security Configuration" in Jenkins and set the "TCP port for JNLP agents" either to "random" or to "fixed" unused port number greater than 1024.
  • Create a new slave or configure an existing one and set the launch method to "Launch agent via Java Web Start".
  • Start the slave by either using the "Launch" button or execute the displayed Java command in a command prompt.

Now the ECU-TEST GUI should appear in the same session as the Jenkins slave is running when starting a new build. Additionally, other problems with external tools connected to ECU-TEST or the Tool-Server should be limited.

🔴 Error messages

[TT] ERROR: Caught ComException: Can't co-create object / Can't get object clsid from progid
  1. Re-register the ECU-TEST COM server with the user that runs your Jenkins slave or master.
    Since ECU-TEST 6.6 it is possible to register the COM server for each user separately.
    This is useful if administrative rights are not available.
  • With administrator privileges
    • Open a new command prompt (cmd.exe) with administrator privileges
    • Run following command: "C:\Program Files [(x86)]\ECU-TEST [Version]\ECU-TEST_COM.exe" /register
    • Close command prompt after successful execution
  • Without administrator privileges
    • Open a new command prompt (cmd.exe) without administrator privileges
    • Run following command: "C:\Program Files [(x86)]\ECU-TEST [Version]\ECU-TEST_COM.exe" /register /peruser
    • Close command prompt after successful execution
  1. Check if the port number 8000 is already in use by another running program. This port is needed in ECU-TEST for making internal XML-RPC requests. To change the default port please read the instructions in the ECU-TEST manual at section "9.5. Remote control of ECU-TEST".
  1. Check if the COM server was registered properly in the Windows registry (regedit.exe) by searching for these entries: HKEY_CLASSES_ROOT\ECU-TEST.Application and HKEY_LOCAL_MACHINE\SOFTWARE\Classes\ECU-TEST.Application
  1. Try to use the version specific COM programmatic identifier for the configured ECU-TEST installation. This id can be set in the advanced settings of the ECU-TEST installation list.
[TT] ERROR: Caught ComException: Invoke of: Item Source: Description: <Fault 1: 'Exception: method "CurrentTestConfiguration.GlobalConstants.Tell.GetName" is not supported'>

This exception is already fixed in release 2.0, please update to at least this version.

[TT] ERROR: Caught ComException: Invoke of: Abort Source: Description: <Fault 1: 'tt.Error: Fehler bei Abbruch per API! Status ABORTED nach 20.0s nicht erreicht.'>

This is a known defect in ECU-TEST and occurs when the test execution could not be aborted within a fixed timeout of 20 seconds.

[TT] ERROR: Error while uploading ATX report: 401: report - ErrorCode-1118: Upload nicht möglich, da die Berechtigung fehlt. API-Schlüssel und Benutzerrechte überprüfen.

An upload authentication key is required to upload reports to TEST-GUIDE, which can be set in the TEST-GUIDE upload configuration.

[TT] ERROR: ECU-TEST executable could not be found!

Please check the installation paths for the configured ECU-TEST installation, both on master in the global tool configuration and in the slave configuration.

[TT] ERROR: No running ECU-TEST instance found, please configure one at first!

At first, check the installation paths for the configured ECU-TEST installation in the global tool configuration.

Otherwise there could be a limitation when using a 64-bit ECU-TEST installation running on a machine with 32-bit Java installed only. To solve this architecture incompatibility install a 64-bit Java Runtime Environment and ensure to run your master or slave on this version.

java.io.IOException: Remote call on xxx failed at hudson.remoting.Channel.call

When using the Swarm Plugin there exists an incompatibility with the plugin dependencies regarding the Apache Commons Lang Library. Please download the modified Swarm client from here and replace it with the existing swarm-client-jar-with-dependencies.jar.

java.lang.UnsatisfiedLinkError: Native Library jacob-1.18-x64/86.dll already loaded in another classloader

This is a known limitation when running the ECU-TEST slave in a user session while the master restarts and reconnects to the slave. Ensure that the slave agent is always restarted when the master restarts.

This can be achieved by creating an appropriate Windows task named RESTART_JENKINS_SLAVE.

Starting with release 2.6 the default Windows task name can be overridden by system property ecutest.taskName on each slave individually.

An example configuration export is attached here and can be easily imported and modified in the Windows Task Scheduler.

Compatibility

  • Jenkins LTS 2.60.3 or higher
  • Java SE Runtime Environment 8 or higher
  • ECU-TEST 7.0.0 or higher
  • optional: TEST-GUIDE 1.55.0 or higher

License

This plugin is licensed under 3-clause BSD license. More information can be found inside the LICENSE file.

Changelog

You can’t perform that action at this time.