Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add automated End to End tests #402

Closed
wants to merge 22 commits into from

Conversation

Projects
None yet
3 participants
@MatthewMerrill
Copy link
Contributor

commented Jun 4, 2019

Tests are defined by a subdirectory of e2e/ containing a test.sh and expected.txt. After tests are run the output can be analyzed in e2e_out/.

Example of failed test (changed a number in the test file):

### Running Test: fpu              ###
!!! Failed: fpu
1,5d0
< 16.12
< -15.88
< 1.92
< 0.01
< 133.33
6a2,6
> -15.87
> 2.08
> 0.01
> 123.08
> 16.14
### Running Test: hello            ###
### Passed: 1/2 (50.00%)           ###

Example of all passes:

### Running Test: fpu              ###
### Running Test: hello            ###
### Passed: 2/2 (100.00%)          ###

Exit code is 1 if any test failed, so this can be integrated into travis testing.

@tbodt
Copy link
Owner

left a comment

🔥 🔥 🔥

Just a couple of things to adjust:

  • I'd like to group all the tests together in a tests directory. Probably the sanest way to do that is to rename the existing tests to tests/manual and put the e2e tests in tests/e2e.
  • Would be cool if the tests could be run with a simple meson test, and have that reproducibly set up the test environment and everything.
e2e.bash Outdated
EXPECTED_LOG="e2e/$1/expected.txt"
ACTUAL_LOG="e2e_out/$1/actual.txt"
DIFF_LOG="e2e_out/$1/diff.txt"
diff -w "$EXPECTED_LOG" "$ACTUAL_LOG" > $DIFF_LOG

This comment has been minimized.

Copy link
@tbodt

tbodt Jun 4, 2019

Owner

diff -u is nicer looking than diff -w IMHO.

@MatthewMerrill MatthewMerrill force-pushed the MatthewMerrill:e2e branch from 06d3c2b to 3667663 Jun 6, 2019

@MatthewMerrill

This comment has been minimized.

Copy link
Contributor Author

commented Jun 6, 2019

moving to a new PR with squashed commits as per request

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.