This repository was archived by the owner on Oct 7, 2020. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 205
This repository was archived by the owner on Oct 7, 2020. It is now read-only.
Overhaul test-suites #1585
Copy link
Copy link
Open
Labels
testingtype: enhancementtype: refactorRefactor and tidy up internals.Refactor and tidy up internals.
Description
A single test-run takes a huge amount of time and, by now, we have a huge array of VM tests.
Currently, we have the resulting matrix of test-runs:
| Tool\GHC | 8.4.2 | 8.4.3 | 8.4.4 | 8.6.1 | 8.6.2 | 8.6.3 | 8.6.4 | 8.6.5 | 8.8.1 |
|---|---|---|---|---|---|---|---|---|---|
| Cabal | ❌ | ❌ | ✔️ | ❌ | ❌ | ❌ | ❌ | ✔️ | ✔️ |
| Stack | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
Resulting in 12 VMs to just build in every configuration, making it 36 VMs for every supported operating system: Linux | MacOs | Windows.
Tests take roughly 20 minutes or even more (on my machine 26 min) to run.
To make the matter worse, we should have tests where only either cabal or stack is installed and both.
What I want to discuss in this post is an overhaul of the test system with the following points:
Functional testsimport suggestionsare incredibly slow- Demote them to
unit-tests# Fixed by Demote HsImport func-tests to unit-test #1591
- Demote them to
- Prefer
directcradle over stack cradle- On my system a call to
stack repltakes roughly1.5s,- I estimate we have 140 calls to
stack repl, replacing this with adirectcradle will reduce run-time by two minutes at least
- I estimate we have 140 calls to
- On my system a call to
- Add tests for loading project context with
stackandcabal, similar to cabal-helper.- Since
hie-biosabstracts the build-tools, we dont have to execute all tests for each build-tool as long as we test that the build-tools initialize the context correctly and provide compilation arguments.
- Since
Avi-D-coder, jneira, alanz and michaelpj
Metadata
Metadata
Assignees
Labels
testingtype: enhancementtype: refactorRefactor and tidy up internals.Refactor and tidy up internals.