Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
There isn't really a document about the whole methodology. There is some brief programmer instructions on how to add tests to the test framework here:
There's also some documentation on one particular test, the "ssl_test":
And some documentation on the fuzz testing that we do here:
We have some continuous integration in the form of Travis (annoyingly broken for quite a long time, although still very useful when creating PRs because the "broken" tests aren't run on individual PRs):
We integrate some tests from external third parties which you can read about here:
Thank you! That's very interesting. Especially pulling in third party test suites.
I noticed that there are some PRs where Coverity is mentioned. Are other static analyzers used?
Also, I see that ubsan, asan, and mbsan are used in the travis builds, and that there is a NOTES.VALGRIND file for using valgrind. Any other tools like this used?
I see there's at least three people with more than 10 years experience with the code base that do code reviews. Are there any review guidelines/checklists used? I see there is a PR template with a checklist for docs and tests here: https://github.com/openssl/openssl/blob/master/.github/PULL_REQUEST_TEMPLATE.md
Oh, and Appveyor is used for CI on windows: https://github.com/openssl/openssl/blob/master/appveyor.yml
What tools are used for benchmarking and performance testing?
Not generally. Sometimes members of the community will report bugs/fix issues on the basis of other static analyzers they have individually used. But as a project we just use Coverity.
Yes all of these are integrated into our build system and tested via travis and/or run-checker. run-checker is a cron job we run regularly which runs through all of our build time config options to check that they all still build and test ok.
We don't have review guidelines as such. There is a coding style document:
And some guidelines for committers:
And some guidelines for contributors:
Nothing formal from a project perspective.