-
-
Notifications
You must be signed in to change notification settings - Fork 658
Continuous Integration
DRAFT This page hopes to describe the plan for the Continuous Integration process to use in Racket.
Goal: to ensure Racket is and remains to be a reliable system, without unexplained regressions or performance changes.
If you are interested in contributing machines, volunteering with code, or anything else, get in touch. We gather in channel #ci on Slack.
Due to available resources for testing and benchmarking, there are X support tiers. TODO: decide how many support tiers are available and which arch/vm/os/options are in each tier.
This section describes the CI environments (hardware/software combinations) on which we can guarantee that Racket works.
All CI workflows live in racket/.github/workflows:
-
ci-pr.yml
: This workflow is ran for each Pull Request. It is a fast workflow that attempts to test most common configurations and OSes. Its goal is to give confidence to the reviewers and the PR author that the code does not break Racket. -
ci-push.yml
: This workflow is slightly slower than the PR workflow but more methodical in its testing. It does take longer to run but it also provides a higher confidence that the racket repo is still in good shape. -
ci-nightly.yml
(TODO): This workflow is slow but ran once a day (nightly in Flatt's timezone). It is thorough and gives the highest confidence that the repo remains green in all tested configurations.
Note: There is a possibility that a commit, passes the PR but breaks Push or even passes both PR and Push workflows but breaks nightly. Since we do not want a red state in the repo - the policy is to revert the commit and open a bug report assigned to the commiter of the broken build so that it can be fixed and re-committed.
Whenever running tests, we sometimes refer to the standard test set or Test (std.)
. These are not all the available tests but those deemed to be most relevant. At the moment these are (which have a dependency on racket-test
and db-test
packages):
raco test -l tests/racket/test
racket -l tests/racket/contract/all
raco test -l tests/json/json
raco test -l tests/file/main
raco test -l tests/net/head
raco test -l tests/net/uri-codec
raco test -l tests/net/url
raco test -l tests/net/url-port
raco test -l tests/net/encoders
raco test -l tests/openssl/basic
raco test -l tests/openssl/https
raco test -l tests/match/main
raco test -l tests/zo-path
raco test -c tests/xml
raco test -l tests/db/all-tests
raco test -c tests/stxparse
Build:
linux |
darwin |
windows |
|
---|---|---|---|
amd64 |
cgc,3m | cgc,3m | cgc,3m,cs |
Test (std.):
linux |
darwin |
windows |
|
---|---|---|---|
amd64 |
3m | 3m | 3m |
Architecture names, operating system names, and their valid combinations are defined by the $GOOS and $GOARCH variables used by the Go compiler.
- @jackfirth Some racket features, such as memory accounting, require specifying certain options when compiling Racket. How do we test that these features work on supported platforms? Does it count as a separate variant of racket? The compiled racket binary is different and could behave differently.
linux |
darwin |
windows |
|
---|---|---|---|
386 |
|||
amd64 |
|||
arm |
|||
arm64 |
|||
mips |
|||
mipsle |
|||
mips64 |
|||
mips64le |
|||
ppc64 |
|||
ppc64le |
|||
s390x |
TODO TODO Where does QNX and FreeBSD fit in?
- @jackfirth For reference, Go just flat out doesn't support QNX at all. We should see how other languages handle this.
linux |
darwin |
windows |
|
---|---|---|---|
386 |
|||
amd64 |
|||
arm |
|||
arm64 |
|||
mips |
|||
mipsle |
|||
mips64 |
|||
mips64le |
|||
ppc64 |
|||
ppc64le |
|||
s390x |
TODO
We are committed to using GitHub Actions for CI as it provides an unparalleled level of integration with our current workflow.
The long-term plan is to develop a single-page dashboard that gather information from test runs and benchmarks and displays it in a succing and understandable way so that Racket devs know at-a-glance what the status of the project is.