Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How is OpenSSL tested? #9831

Open
illume opened this issue Sep 9, 2019 · 3 comments

Comments

@illume
Copy link

@illume illume commented Sep 9, 2019

Hello!

I was wondering if there is a document on how OpenSSL is tested?

What tools, infrastructure and methods are used?

cheers,

@mattcaswell

This comment has been minimized.

Copy link
Member

@mattcaswell mattcaswell commented Sep 9, 2019

There isn't really a document about the whole methodology. There is some brief programmer instructions on how to add tests to the test framework here:

https://github.com/openssl/openssl/blob/master/test/README

There's also some documentation on one particular test, the "ssl_test":

https://github.com/openssl/openssl/blob/master/test/README.ssltest.md

And some documentation on the fuzz testing that we do here:

https://github.com/openssl/openssl/blob/master/fuzz/README.md

We have some continuous integration in the form of Travis (annoyingly broken for quite a long time, although still very useful when creating PRs because the "broken" tests aren't run on individual PRs):

https://travis-ci.org/openssl/openssl/builds

And Appveyor:

https://ci.appveyor.com/project/openssl/openssl/history

We integrate some tests from external third parties which you can read about here:

https://github.com/openssl/openssl/blob/master/test/README.external

@illume

This comment has been minimized.

Copy link
Author

@illume illume commented Sep 16, 2019

Thank you! That's very interesting. Especially pulling in third party test suites.

I noticed that there are some PRs where Coverity is mentioned. Are other static analyzers used?

Also, I see that ubsan, asan, and mbsan are used in the travis builds, and that there is a NOTES.VALGRIND file for using valgrind. Any other tools like this used?

I see there's at least three people with more than 10 years experience with the code base that do code reviews. Are there any review guidelines/checklists used? I see there is a PR template with a checklist for docs and tests here: https://github.com/openssl/openssl/blob/master/.github/PULL_REQUEST_TEMPLATE.md

Oh, and Appveyor is used for CI on windows: https://github.com/openssl/openssl/blob/master/appveyor.yml

What tools are used for benchmarking and performance testing?

@mattcaswell

This comment has been minimized.

Copy link
Member

@mattcaswell mattcaswell commented Sep 17, 2019

I noticed that there are some PRs where Coverity is mentioned. Are other static analyzers used?

Not generally. Sometimes members of the community will report bugs/fix issues on the basis of other static analyzers they have individually used. But as a project we just use Coverity.

Also, I see that ubsan, asan, and mbsan are used in the travis builds, and that there is a NOTES.VALGRIND file for using valgrind. Any other tools like this used?

Yes all of these are integrated into our build system and tested via travis and/or run-checker. run-checker is a cron job we run regularly which runs through all of our build time config options to check that they all still build and test ok.

Are there any review guidelines/checklists used?

We don't have review guidelines as such. There is a coding style document:

https://www.openssl.org/policies/codingstyle.html

And some guidelines for committers:

https://www.openssl.org/policies/committers.html

And some guidelines for contributors:

https://github.com/openssl/openssl/blob/master/CONTRIBUTING

What tools are used for benchmarking and performance testing?

Nothing formal from a project perspective.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.