Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide JUnit output for PIO Unit Testing #2891

Closed
tschissler opened this issue Jul 29, 2019 · 13 comments
Closed

Provide JUnit output for PIO Unit Testing #2891

tschissler opened this issue Jul 29, 2019 · 13 comments

Comments

@tschissler
Copy link

tschissler commented Jul 29, 2019

I was able to create a set of Unit Tests with PlatformIO and run them as native within a docker container. I also managed to run those tests as port of my build pipeline on Azure DevOps. All looks pretty good.

Thank you so much for building this amazing tool!!!

Only thing which nags me is that failing unit tests make my build fail with no additional information available before I dig into the log files.
It would be amazing, if I could extract the build results into a test result file in a JUnit format or something alike which I then could use to provide my build system with additional information.
I'm wondering if something like this is build in right now or if I have to build something on my own.

Useful links

@ivankravets ivankravets transferred this issue from platformio/platformio-docs Aug 11, 2019
@ivankravets
Copy link
Member

Do you mean pio test -v?

@tschissler
Copy link
Author

Not really. That would only add more details, if I understand this option correctly.
I have solved the issue by writing the console output to a file and then parsing this to create the XML format used by NUnit and others.
I'll publish the python script I created and link it to this thread within the next couple of days and then close it.

@ivankravets
Copy link
Member

Maybe pio test —json-output will help here?

@ivankravets
Copy link
Member

Where is the specification for this format in XML?

@tschissler
Copy link
Author

@ivankravets I do not know the pio test --json-output option and on my version it seems not to be supported nor can I find documentation about it. That would certainly be helpful as JSON would be easier to parse than the text output.

The XML format I am using is this here: https://github.com/nunit/docs/wiki/Test-Result-XML-Format. But there would be other options possible as my goal is to use this task in a Azure DevOps build pipeline: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml to publish my test results and make them available to be seen on the build report. The result then looks like this:
image

I have shared the Python script I have build here: https://github.com/tschissler/sharing/tree/master/Scripts
Bare with me as I am a Python newbie :-) But maybe someone else might find this helpful.

I'm now working on an article describing how to integrate Platformio with Azure DevOps CI/CD. Maybe you want to consider adding this option to your documentation of CI/CD systems.

@ivankravets ivankravets changed the title Unit Test Results File Provide JSON output for PIO Unit Testing Aug 20, 2019
@ivankravets
Copy link
Member

I renamed this issue to "Provide JSON output for PIO Unit Testing". So we will implement it soon.

@ivankravets ivankravets added this to the Backlog milestone Aug 30, 2019
@jpsfs
Copy link

jpsfs commented Mar 21, 2020

Hi!
I'm also looking for the same kind of functionality. Is there any idea when will this be available?

@tschissler could you share your yml pipeline?

@ivankravets
Copy link
Member

@tschissler
Copy link
Author

I'll share my solution later. Working on a Hackton against the virus this weekend.

@tschissler
Copy link
Author

@jpsfs This is my YAML file for the build. I use a docker container where the build runs in.
Let me know if you have any questions.

# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

resources:
  containers:
  - container: platformio
    image: infinitecoding/platformio-for-ci:latest   
    options: -u root    

trigger:
- master

jobs:
- job: esp32_platformio
  displayName: "PlatformIO build"
  container: platformio

  pool:
    vmImage: 'ubuntu-latest'

  steps: 
    - script: dir -a
      displayName: "Show current directory content"
    - script: cd battleship-embedded/BattleShip
      displayName: "Open Battleship project directory"
    - script: platformio test -e native -d battleship-embedded/BattleShip >> testresults.txt
      displayName: "Run Unit Tests"
      continueOnError: true
    - script: python $(Build.SourcesDirectory)/battleship-embedded/BattleShip/_deploy/TestResultsParser.py testresults.txt testresults.xml
      displayName: "Converting Test Results"
    - task: PublishTestResults@2
      inputs:
        testResultsFormat: 'NUnit'
        testResultsFiles: 'testresults.xml'
        failTaskOnFailedTests: true
    - script: platformio run -e esp32 -d battleship-embedded/BattleShip
      displayName: "Build for ESP32 platform"
    - task: CopyFiles@2
      inputs:
        SourceFolder: $(Build.SourcesDirectory)/battleship-embedded/BattleShip/.pio/build/esp32/
        Contents: '*.bin'
        TargetFolder: $(Build.ArtifactStagingDirectory)
      displayName: "Copy build output files to ArtifactsStagingDirectory"
    - task: CopyFiles@2
      inputs:
        SourceFolder: $(Build.SourcesDirectory)/battleship-embedded/BattleShip/_deploy
        Contents: '*.ps1'
        TargetFolder: $(Build.ArtifactStagingDirectory)
      displayName: "Copy deployment scripts to ArtifactsStagingDirectory"
    - task: PublishBuildArtifacts@1
      inputs:
        ArtifactName: 'Firmware_$(Build.BuildNumber)'
        PathtoPublish: $(Build.ArtifactStagingDirectory)
        publishLocation: Container
        TargetPath: .

@tux19
Copy link

tux19 commented Apr 7, 2021

I would like to have the test results integrated into gitlab-ci. Is there already a solution to export test results without manually parsing the output?

@mickeprag
Copy link

If the output could follow some common format this would be easier to integrate into different solutions. GitLab supports JUnit files so my vote goes for this format.
https://docs.gitlab.com/ee/ci/unit_test_reports.html#how-to-set-it-up

@ivankravets ivankravets modified the milestones: Backlog, 5.3.0 Apr 11, 2022
@ivankravets ivankravets changed the title Provide JSON output for PIO Unit Testing Provide JUint output for PIO Unit Testing Apr 21, 2022
@ivankravets ivankravets changed the title Provide JUint output for PIO Unit Testing Provide JUnit output for PIO Unit Testing Apr 23, 2022
@ivankravets
Copy link
Member

ivankravets commented Apr 23, 2022

Hi all,

The new 2 formats are implemented: JSON and JUnit. Please upgrade PlatformIO Core via pio upgrade --dev and re-test with pio test --output-junit.

Does it work for you?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants