Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI post functions for hardware testing #216

Open
hakehuang opened this issue Mar 9, 2020 · 12 comments
Open

CI post functions for hardware testing #216

hakehuang opened this issue Mar 9, 2020 · 12 comments
Assignees

Comments

@hakehuang
Copy link

@hakehuang hakehuang commented Mar 9, 2020

from Testing CI requirements, we expect the CI to upload two things to a result repo:

  1. the SHA number to a text file.
  2. the qemu board testing result.

then result repo can used by all community member to create a real board testing.

see below red parts of proposed board testing process.

image

@hakehuang hakehuang changed the title cloudy CI post fucntions CI post fucntions Mar 9, 2020
@nashif nashif self-assigned this Mar 16, 2020
@nashif
Copy link
Member

@nashif nashif commented Mar 17, 2020

image

An updated workflow proposal that utilizes AWS S3 for data collection instead of a git repo. This is easier to implement and provides one single place for all test results and reports.

We will still use the test_results repo to collect data from members and community, but the central place for all raw results will be in AWS S3. Once all data is pushed to testrail, we can also put HTML test reports and other collateral in AWS S3.

Pushing data to git by CI is much more complex and can come with many conflicts.

@xuhao8210
Copy link

@xuhao8210 xuhao8210 commented Mar 19, 2020

@nashif -- Could you please help review my few questions below ? Thanks in advance.

  1. Could you please introduce what is the result bucket used for?
  2. After the test results is pushed to testrail (e.g. once one day) , do we need the delete the test results in the Result Repo ?
  3. About the description: “we will put HTML test reports and other collateral in AWS S3.” , will it also be implemented in github’s action to use another workflow ? And it seems that there is no the related API by Test Reail provided to export the HTML test reports in qm repo, could you please help double confirm it ? Thanks again.

@nashif
Copy link
Member

@nashif nashif commented Mar 19, 2020

1. Could you please introduce what is the result bucket used for?

This is where all test results will end up, raw results and reports generated from testrail for example. This is the same as the "archive on AWS S3" box.

image

After the test results is pushed to testrail (e.g. once one day) , do we need the delete the test results in the Result Repo ?

No.

3\. About the description: “we will put HTML test reports and other collateral in AWS S3.” , will it also be implemented in github’s action to use another workflow ? And it seems that there is no the related API by Test Reail provided to export the HTML test reports in qm repo, could you please help double confirm it ?   Thanks again.

Did you try http://docs.gurock.com/testrail-api2/reference-reports
We need to find a way to download reports and archive them, make them available outside of testrail.

@hakehuang
Copy link
Author

@hakehuang hakehuang commented Apr 15, 2020

image

update user case diagram

@hakehuang
Copy link
Author

@hakehuang hakehuang commented Apr 21, 2020

@xuhao8210 I create two issues hakehuang/zephyr_ci_repo#2, maybe we can use this as a trial test repo, and I will try to create GH hooks for below steps:

  1. log latest hash number in a text file
  2. upload qemu board test result by date

@nashif
Copy link
Member

@nashif nashif commented Apr 22, 2020

next steps:

  • synchronize matrix to build one commit
  • adapt sanitycheck to support only running a group or simulators/emulators in CI

considerations:

  • All results will be in one file for all platforms running in CI, not 1 per platform.

@hakehuang
Copy link
Author

@hakehuang hakehuang commented Apr 22, 2020

https://github.com/hakehuang/zephyr_ci_repo/actions/runs/84702849

@nashif I meet an issue when I try to create a 3 qemu builds. can we apply for more space?

@hakehuang
Copy link
Author

@hakehuang hakehuang commented Apr 22, 2020

next steps:

  • synchronize matrix to build one commit
  • adapt sanitycheck to support only running a group or simulators/emulators in CI

considerations:

  • All results will be in one file for all platforms running in CI, not 1 per platform.

I check the json file

[
  "zephyr-v2.2.0-1791-g8d984b336e",
  "zephyr-v2.2.0-1794-g670358c039",
  "zephyr-v2.2.0-1797-ga7e243bea4",
  "zephyr-v2.2.0-1803-g6e3f86876e"
]

do we want the list keep growing? Besides, what is the API to upload build report and this version json file?

@nashif
Copy link
Member

@nashif nashif commented Apr 22, 2020

@nashif I meet an issue when I try to create a 3 qemu builds. can we apply for more space?

not sure what are you trying to do there, we do not need to run sanitycheck in actions, this is currently not tested and not supported and requires alot of setup and resources. We will run sanitycheck as before in shippable, this part is step 1 and I am taking caring of.
We will only use GH actions for all other interactions and processing of results and uploading them to testrail.

@nashif
Copy link
Member

@nashif nashif commented Apr 22, 2020

do we want the list keep growing? Besides, what is the API to upload build report and this version json file?

I do not see any reason why not, this is the manifest of all versions. This is preliminary and we could tag the latest version in the json file.

Uploading results and changing this file and other files will all happen using AWS S3 through the GH action. So it is normal file operations.

@nashif nashif changed the title CI post fucntions CI post fucntions for hardware testing Apr 22, 2020
@nashif
Copy link
Member

@nashif nashif commented Apr 22, 2020

@hakehuang see this https://github.com/zephyrproject-rtos/test_results, this is the job creating the version file and uploading results from CI.

@hakehuang
Copy link
Author

@hakehuang hakehuang commented Apr 24, 2020

@nashif I meet an issue when I try to create a 3 qemu builds. can we apply for more space?

not sure what are you trying to do there, we do not need to run sanitycheck in actions, this is currently not tested and not supported and requires alot of setup and resources. We will run sanitycheck as before in shippable, this part is step 1 and I am taking caring of.
We will only use GH actions for all other interactions and processing of results and uploading them to testrail.

I understand now, Thanks

@carlescufi carlescufi changed the title CI post fucntions for hardware testing CI post functions for hardware testing Apr 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants