How is OpenSSL tested?

[illume]

Hello!

I was wondering if there is a document on how OpenSSL is tested?

What tools, infrastructure and methods are used?

cheers,

[mattcaswell]

There isn't really a document about the whole methodology. There is some brief programmer instructions on how to add tests to the test framework here:

https://github.com/openssl/openssl/blob/master/test/README

There's also some documentation on one particular test, the "ssl_test":

https://github.com/openssl/openssl/blob/master/test/README.ssltest.md

And some documentation on the fuzz testing that we do here:

https://github.com/openssl/openssl/blob/master/fuzz/README.md

We have some continuous integration in the form of Travis (annoyingly broken for quite a long time, although still very useful when creating PRs because the "broken" tests aren't run on individual PRs):

https://travis-ci.org/openssl/openssl/builds

And Appveyor:

https://ci.appveyor.com/project/openssl/openssl/history

We integrate some tests from external third parties which you can read about here:

https://github.com/openssl/openssl/blob/master/test/README.external

[illume]

Thank you! That's very interesting. Especially pulling in third party test suites.

I noticed that there are some PRs where Coverity is mentioned. Are other static analyzers used?

Also, I see that ubsan, asan, and mbsan are used in the travis builds, and that there is a NOTES.VALGRIND file for using valgrind. Any other tools like this used?

I see there's at least three people with more than 10 years experience with the code base that do code reviews. Are there any review guidelines/checklists used? I see there is a PR template with a checklist for docs and tests here: https://github.com/openssl/openssl/blob/master/.github/PULL_REQUEST_TEMPLATE.md

Oh, and Appveyor is used for CI on windows: https://github.com/openssl/openssl/blob/master/appveyor.yml

What tools are used for benchmarking and performance testing?

[mattcaswell]

I noticed that there are some PRs where Coverity is mentioned. Are other static analyzers used?

Not generally. Sometimes members of the community will report bugs/fix issues on the basis of other static analyzers they have individually used. But as a project we just use Coverity.

Also, I see that ubsan, asan, and mbsan are used in the travis builds, and that there is a NOTES.VALGRIND file for using valgrind. Any other tools like this used?

Yes all of these are integrated into our build system and tested via travis and/or run-checker. run-checker is a cron job we run regularly which runs through all of our build time config options to check that they all still build and test ok.

Are there any review guidelines/checklists used?

We don't have review guidelines as such. There is a coding style document:

https://www.openssl.org/policies/codingstyle.html

And some guidelines for committers:

https://www.openssl.org/policies/committers.html

And some guidelines for contributors:

https://github.com/openssl/openssl/blob/master/CONTRIBUTING

What tools are used for benchmarking and performance testing?

Nothing formal from a project perspective.

[mattcaswell]

This question seems to have been answered. Closing.