Dev jobs publish test reports to doc server #685
Conversation
|
The current upload seems to not remove old existing test results? So I would assume they will accumulate over time and the wiki would show a mixture of current and outdated information. |
|
If there is no good reason to choose a different term I would suggest to |
|
Also can you give an estimate how much additional data we are roughly talking about to store on docs.ros.org? |
|
@dirk-thomas Thank you for your review!
Good point! I added in removing the remote data via the
Got it. Changed it to "test results".
Sorry, I don't really have access to that data. |
ros_buildfarm/templates/snippet/publisher_publish-over-ssh.xml.em
Outdated
Show resolved
Hide resolved
|
The patch itself looks fine to me. Thanks for iterating. The question now seems to be if we still want these test result files to be copied from the Jenkins server (see related discussion on droter/roswiki#1 (comment))? It seems that publishing the html files from HAROS might already satisfy the current use case. |
|
@dirk-thomas Thank you for the review and sorry for the late reply. Personally, I don't understand enough about the ROS infrastructure to argue for one option over another, but it seems like both options (copying test results to the doc server or showing html-based test results on Jenkins) are both being somewhat opposed: #693 Could we merge the two discussions and find some consensus? |
|
This pull request has been mentioned on ROS Discourse. There might be relevant details there: https://discourse.ros.org/t/next-ros-quality-assurance-working-group-february-2020/12571/1 |
|
The QAWG asked if I could help get this merged. I don't know enough about the build farm to safely merge this, but I would like to learn. Anyone want to walk me through the review and testing? |
|
@kscottz Please read through the above conversation. The pending question is:
|
|
This pull request has been mentioned on ROS Discourse. There might be relevant details there: https://discourse.ros.org/t/next-ros-quality-assurance-working-group-march-2020/12914/1 |
|
Overall there's two competing resources here. The first element is storage space. We have lots of space on docs.ros.org but in general content that gets added there stays there. On the other side jenkins can also host the data and make it available. But it generates so much data that all jobs have retention policies for cleaning up older builds after some period or number of cycles. To that end we cannot make a decision on this without knowing the actual data volumes we're talking about. If it's low enough that we can afford to persist the integrated data for all builds ad-infanitum uploading to docs.ros.org with full history is reasonable. However that seems unlikely. For the documentation we only keep the latest version of content for each package (per rosdistro) so the data volume doesn't grow over time. If we want to keep some level of history it's probably better to keep it on jenkins which has mechanisms for setting retention policies. The main drawback to that is that if this becomes something that's moderately high volume of consumption it could relatively easily overwhelm the jenkins instance with load serving the content. It's already by far our biggest server for hosting and is the one that we have to pay the most attention to to keep it running. This is why we boiled it down to needing to know what the data looks like and how much volume of access we're expecting to experience. We can agree that hosting these files is a good idea. But until we can understand what the costs will be (storage space, and server load) it's impossible for us to evaluate the tradeoffs for this request and the other side of it here: #693 |
|
And at the moment publishing the results to another server yields no obvious benefit (beside bypassing the Jenkins retention limits). |
|
In my mind having the build results for all builds for all of time is a "nice to have" but not really necessary for testing to provide utility. A simpler and incremental improvement is to have just last N build test results (most of the time you're looking at the delta, and the results should nominally be 100% passing). Are Jenkins retention policies per-artifact or per-build? If the latter something between the previous 3-7 previous builds seems sufficient to me. I take it there is no "test" build farm where you could test this and see how it performs. It sounds like the action item is to pick a dozen or so packages and run the test suite over them and then characterize the output so we can upper bound the amount of retained data. |
|
You can set a max number of builds or a max time. If either are surpassed it will get pruned. I believe there's typically an exception for one build results in the time based threshold. |
|
@max-krichenbauer @kscottz What is the status on this? I am inclined to close the ticket since there is no clear rational why the artifacts should be uploaded / how the uploaded artifacts would be used. |
|
Sorry, I'm no longer participating in the QA working group. |
This PR adds a feature to dev jobs to upload output files of automated test to the doc server so they can be accessed easily at the same place with all the information regarding the package.
Currently, the wiki will show a "CI badge" informing people of passed or failed recent CI tests, but the details of each test require visiting the build farm Jenkins master.
In the future, we suggest that the ROS Wiki or index.ros.org hosts the most recent test results for easy access.
The update to the Wiki to display links to the test reports is under review here:
droter/roswiki#1
This PR is based on previous similar PR to both the Wiki and the build farm that introduced the CI badge. ( ros-infrastructure/roswiki#246 and #541 )