Test results aggregator #136
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…local/version.json
…utcome can be confirmed.
|
There's a conflict, and also that bug with logs. Ping me when you want it merged |
|
... perhaps rebase on master aswell... |
holiman
reviewed
Nov 7, 2018
| <<<<<<< HEAD | ||
| t.l.Log("Establishing criteria: Will succeed if a valid pong packet is received.") | ||
| ======= | ||
| >>>>>>> 32f8a2836b9a2cd8ae2cf36b969fb21d3312c4a4 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There are several main areas of change here
The results are output in a slightly different folder structure than prior, and are now (almost) the same for each type of test. From the output root, the first subfolder is a datetime, followed by the test type and name (eg: simulator_ethereum_consensus), followed by the client (eg:go-ethereum_master). This same structure applies for the benchmarks and validators. Simulators now have their log file in the test folder, rather than in the test/client subfolder, as a simulation can now involve multiple clients.
Hive now supports 'multi-client' simulations. The "Ethereum/consensus" tests were modified to preserve running the tests across all specified client. To see how this works take a look at line 157 of simulators/Ethereum/consensus/hivemodel.py
The same output artifacts are preserved, with the addition of a running, appended results summary report in the output root, by default named as 'listing.json' . (There are also some additional .json test metadata files, to facilitate building other result viewing clients) The create-listings.py is now probably not necessary. The listing.json file is not rebuilt, it is appended. Deleting subfolders will not affect listing.json.
The flags now include a results-root flag, so you can run Hive and have the output go directly to the target folder - the start-hive.sh script steps that move folder around are probably no longer necessary.
The existing testreport_template was modified to work with all test types and display the results. In order to keep Hive and the viewer versions together, I added the index.html and dependent libraries into the Hive repo. They are in the folder resultview. This default viewer should be placed in the same folder as results-root.