forked from curl/curl
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
tests/CI.md: add more information on CI environments
Fixes curl#8012.
- Loading branch information
1 parent
4d97fe5
commit da4c113
Showing
3 changed files
with
111 additions
and
35 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| # Continuous Integration for curl | ||
|
|
||
| Curl runs in many different environments, so every change is run against a large | ||
| number of test suites. | ||
|
|
||
| Every pull request is verified for each of the following: | ||
|
|
||
| - ... it still builds, warning-free, on Linux and macOS, with both | ||
| clang and gcc | ||
| - ... it still builds fine on Windows with several MSVC versions | ||
| - ... it still builds with cmake on Linux, with gcc and clang | ||
| - ... it follows rudimentary code style rules | ||
| - ... the test suite still runs 100% fine | ||
| - ... the release tarball (the "dist") still works | ||
| - ... it builds fine in-tree as well as out-of-tree | ||
| - ... code coverage does not shrink drastically | ||
| - ... different TLS backends still compile and pass tests | ||
|
|
||
| If the pull-request fails one of these tests, it will show up as a red X and | ||
| you are expected to fix the problem. If you do not understand when the issue is | ||
| or have other problems to fix the complaint, just ask and other project | ||
| members will likely be able to help out. | ||
|
|
||
| Consider the following table while looking at pull request failures: | ||
|
|
||
| | CI platform as shown in PR | State | What to look at next | | ||
| | ----------------------------------- | ------ | -------------------------- | | ||
| | CI / codeql | stable | quality check results | | ||
| | CI / fuzzing | stable | fuzzing results | | ||
| | CI / macos ... | stable | all errors and failures | | ||
| | Code scanning results / CodeQL | stable | quality check results | | ||
| | FreeBSD FreeBSD: ... | stable | all errors and failures | | ||
| | LGTM analysis: Python | stable | new findings | | ||
| | LGTM analysis: C/C++ | stable | new findings | | ||
| | buildbot/curl_winssl_ ... | stable | all errors and failures | | ||
| | continuous-integration/appveyor/pr | stable | all errors and failures | | ||
| | curl.curl (linux ...) | stable | all errors and failures | | ||
| | curl.curl (windows ...) | flaky | repetitive errors/failures | | ||
| | deepcode-ci-bot | stable | new findings | | ||
| | musedev | stable | new findings | | ||
|
|
||
| Sometimes the tests fail due to a dependency service temporarily being offline | ||
| or otherwise unavailable, eg. package downloads. In this case you can just | ||
| try to update your pull requests to rerun the tests later as described below. | ||
|
|
||
| ## CI servers | ||
|
|
||
| Here are the different CI environments that are currently in use, and how they | ||
| are configured: | ||
|
|
||
| ### Github Actions | ||
|
|
||
| Github Actions runs the following tests: | ||
|
|
||
| - Mac OS tests with a variety of different compilation options | ||
| - Fuzz tests ([see tests/fuzz/README for | ||
| more info](https://github.com/curl/curl/blob/master/tests/fuzz/README)). | ||
| - Curl compiled using the Rust TLS backend with Hyper | ||
| - CodeQL static analysis | ||
|
|
||
| These are each configured in different files in `.github/workflows`. | ||
|
|
||
| ### Azure | ||
|
|
||
| The following tests are run in Microsoft Azure CI environment: | ||
|
|
||
| - Ubuntu tests with a variety of different compilation options. | ||
| - Windows tests with a variety of different compilation options. | ||
|
|
||
| These are all configured in `.azure-pipelines.yml`. | ||
|
|
||
| As of November 2021 Daniel Stenberg is the only person with administrator access | ||
| to the Azure CI environment. | ||
|
|
||
| ### Appveyor | ||
|
|
||
| Appveyor runs a variety of different Windows builds, with different compilation | ||
| options. | ||
|
|
||
| As of November 2021 Daniel Stenberg is the only person with administrator access | ||
| to the Appveyor CI environment. | ||
|
|
||
| The tests are configured in appveyor.yml. | ||
|
|
||
| ### Zuul | ||
|
|
||
| [Zuul](https://zuul-ci.org/) is an open source CI tool. A number of Curl tests | ||
| are run at [curl.zuul.vexxhost.dev](https://curl.zuul.vexxhost.dev/builds): | ||
|
|
||
| - Source code is formatted according to expectations (`make checksrc`) | ||
| - Curl compiled with a number of different TLS configurations (WolfSSL, rustls, | ||
| BoringSSL, etc). | ||
| - Curl compiled with different C compilers. | ||
|
|
||
| As of November 2021, the tests run (sometimes) but do not run consistently and | ||
| do not report results to the Github checks runner - you need to manually check | ||
| for failures. See [#7522](https://github.com/curl/curl/issues/7522) for more | ||
| information. | ||
|
|
||
| As of November 2021 Daniel Stenberg is the only person with administrator access | ||
| to the Zuul CI environment. | ||
|
|
||
| These are configured in `zuul.d` and have test runners in `scripts/zuul`. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters