Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update makefile to use examples harness #30

Merged
Merged
Changes from all commits
Commits
File filter...
Filter file types
Jump to…
Jump to file
Failed to load files.

Always

Just for now

Update the Makefile to invoke the examples; add a levels concept to t…

…ests.

The Makefile's default target has been updated so that it now requires
that the 'system' level tests build. This is one of 3 new targets that
have been added:

  - test_level_unittests, which should cause the unit tests to run.
    We don't have any yet, but some will be added shortly.

  - test_level_integration, which causes the integration tests to run.
    These are the tests in the 'tests' directory, which exercise the
    external interface in a test-like environment to see that it
    functions as desired.
    This level is dependant on the unittests passing before it will
    be run.

  - test_level_system, which causes the examples to be run.
    These examples are the sorts of programs that a user might write,
    and which we expect to be invoked from the command and to do
    their job.
    This level is dependant on the integration tests passing before
    it will be run.

In CI, the individual test targets (rather than the levels) would
probably be invoked, with each one still dependant on the earlier ones
in stages (or maybe just run them all in parallel, as they're all very
fast).
  • Loading branch information
gerph committed Jan 11, 2020
commit da4bb4f3e1ba4e8214b4e41c6e415f8d559ff6d6
@@ -1,7 +1,7 @@
# Common User Targets:
#
# make tests
# - Sxecute the tests
# - Execute the tests
#
# make shell
# - Set up the environment and drop to a shell (re-uses existing shells if they
@@ -40,12 +40,34 @@ NOTICE = @notice() { printf "\n${COL_NOTICE}+++ %s${COL_RESET}\n" "$$@"; } && no
GOOD = @notice() { printf "\n${COL_GOOD}+++ %s${COL_RESET}\n" "$$@"; } && notice


tests: test_testable
${NOTICE} "Running tests"
tests: test_level_system
${GOOD} "All tests passed"

# 'test_level_*' targets run all the tests up to that level
test_level_unittests: unittests
test_level_integration: test_level_unittests integrationtests
test_level_system: test_level_integration systemtests

# Unit tests test individual parse of a small unit.
unittests: test_testable
${NOTICE} "Running unit tests"
${GOOD} "Unit tests passed (we don't have any yet)"

# Integration tests check the integration of those units.
integrationtests: test_testable
${NOTICE} "Running integration tests"
@# Note: We cd into the tests directory, so that we are testing the installed version, not
@# the version in the repository.
${ACTIVATE} && cd tests && python -munittest -v ${TEST_MODULES}
${GOOD} "Tests passed"
${GOOD} "Integration tests passed"

# System tests check that the way that a user might use it works.
systemtests: test_testable
${NOTICE} "Running system tests"
@# We only run 1000 runs; just enough that we get to see that it's running the tests.
${ACTIVATE} && examples/run_all_examples.py --runs 1000
${GOOD} "System tests passed"


venv: venv/successful-${PYTHON_TOOL}

ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.