Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add smoke tests #106

Merged
merged 4 commits into from
Aug 11, 2023
Merged

Add smoke tests #106

merged 4 commits into from
Aug 11, 2023

Conversation

ErikSchierboom
Copy link
Member

This PR adds smoke tests for some common scenarios.

The goal of the smoke tests is to increase confidence that the Docker image that is produced works as
intended.
The repo contains other tests, which are great, but what this PR adds is that it runs an integration
test using the following setup:

  1. Build the Docker image
  2. Run the Docker image with the tests directory mapped into the container
  3. Run the Docker image's normal entrypoint (bin/run.sh) on each of the test cases defined in the
    tests directory
  4. Verify that the output of those test runs matches the expected output (which is stored inside the
    tests directories

If successful, we then know that:

  1. The Docker image can be built successfully
  2. The bin/run.sh entrypoint inside the Docker image is setup correctly
  3. The test runner generates the correct output

@ErikSchierboom ErikSchierboom requested a review from a team as a code owner August 9, 2023 18:37
@github-actions
Copy link

github-actions bot commented Aug 9, 2023

Pull Request Test Coverage Report for Build 5813213734

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 16 unchanged lines in 1 file lost coverage.
  • Overall coverage decreased (-0.9%) to 64.952%

Files with Coverage Reduction New Missed Lines %
testrunner/ast.go 16 74.11%
Totals Coverage Status
Change from base Build 5467785147: -0.9%
Covered Lines: 404
Relevant Lines: 622

💛 - Coveralls

@iHiD
Copy link
Member

iHiD commented Aug 9, 2023

For context, nearly all test runners have this setup. A few older ones (the OGs!) don't have the exact same setup, so this is an aim to standardise.

@ErikSchierboom
Copy link
Member Author

Addendum: I know that the Go code also has golden tests, but AFAICT it doesn't build and use the Docker image to verify it.

@ErikSchierboom
Copy link
Member Author

This doesn't yet work due to some permissions issue. If anyone has an idea how to solve that, I'd be interested to hear :)

@ErikSchierboom
Copy link
Member Author

Fixed the issue!

@ErikSchierboom
Copy link
Member Author

As nobody seems around to review this, I'm gonna go ahead and merge this.

@ErikSchierboom ErikSchierboom merged commit d10bdb5 into main Aug 11, 2023
10 checks passed
@ErikSchierboom ErikSchierboom deleted the smoke-tests branch August 11, 2023 06:35
@junedev
Copy link
Member

junedev commented Sep 3, 2023

@ErikSchierboom I was on vacation but I'm sure this is fine, thanks!
Just as a heads up, when we update the Go version, there are sometimes minor changes in the test result output, mostly something like whitespace changes. I think the test still helps to increase confidence regarding the docker the file etc as you wrote, just limiting the expectations a bit that we won't need to touch them at all.

@ErikSchierboom
Copy link
Member Author

Just as a heads up, when we update the Go version, there are sometimes minor changes in the test result output, mostly something like whitespace changes.

That's perfectly fine, it happens to many test runners. One easy thing to do is:

bin/run-tests-in-docker.sh
find tests -name results.json -execdir cp results.json expected_results.json \;

This will create a results.json and then copy it to the expected_results.json, which allows you to quickly update the expected_results.json files with the latest version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants