Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Chutney ci 035 #736

Open
wants to merge 5 commits into
base: maint-0.3.5
Choose a base branch
from
Open

Chutney ci 035 #736

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f7590f6
Add a new testing target, test-network-forgiving
nmathewson Feb 27, 2019
21c3d43
Travis: control "make check" with its own flag
nmathewson Feb 27, 2019
1579747
Add a chutney mode to travis.yml.
nmathewson Feb 27, 2019
911371f
Merge branch 'chutney_ci_029' into chutney_ci_034
nmathewson Feb 27, 2019
68ef52d
Merge branch 'chutney_ci_034' into chutney_ci_035
nmathewson Feb 27, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 10 additions & 3 deletions .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,14 +48,18 @@ matrix:
## We check asciidoc with distcheck, to make sure we remove doc products
- env: DISTCHECK="yes" ASCIIDOC_OPTIONS=""
# We clone our stem repo and run `make test-stem`
- env: TEST_STEM="yes"
- env: TEST_STEM="yes" SKIP_MAKE_CHECK="yes"
## Check rust online with distcheck, to make sure we remove rust products
- env: DISTCHECK="yes" RUST_OPTIONS="--enable-rust --enable-cargo-online-mode"
## Check disable module dirauth with and without rust
- env: MODULES_OPTIONS="--disable-module-dirauth" RUST_OPTIONS="--enable-rust" TOR_RUST_DEPENDENCIES=true
- env: MODULES_OPTIONS="--disable-module-dirauth"
## Check NSS
- env: NSS_OPTIONS="--enable-nss"
## We also try running a hardened clang build with chutney on Linux.
- env: CHUTNEY="yes" SKIP_MAKE_CHECK="yes"
compiler: clang
os: linux

## Uncomment to allow the build to report success (with non-required
## sub-builds continuing to run) if all required sub-builds have
Expand Down Expand Up @@ -183,6 +187,8 @@ install:
- if [[ "$TOR_RUST_DEPENDENCIES" == "true" ]]; then export TOR_RUST_DEPENDENCIES=$PWD/src/ext/rust/crates; fi
- if [[ "$TEST_STEM" != "" ]]; then git clone --depth 1 https://github.com/torproject/stem.git ; export STEM_SOURCE_DIR=`pwd`/stem; fi
##
## If we're running chutney, install it.
- if [[ "$CHUTNEY" != "" ]]; then git clone --depth 1 https://github.com/torproject/chutney.git ; export CHUTNEY_PATH="$(pwd)/chutney"; fi
## Finally, list installed package versions
- if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then dpkg-query --show; fi
- if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then brew list --versions; fi
Expand All @@ -201,16 +207,17 @@ script:
- echo "Configure flags are $CONFIGURE_FLAGS"
- ./configure $CONFIGURE_FLAGS
## We run `make check` because that's what https://jenkins.torproject.org does.
- if [[ "$DISTCHECK" == "" && "$TEST_STEM" == "" ]]; then make check; fi
- if [[ "$SKIP_MAKE_CHECK" == "" ]]; then make check; fi
- if [[ "$TEST_STEM" != "" ]]; then make src/app/tor test-stem; fi
- if [[ "$DISTCHECK" != "" && "$TEST_STEM" == "" ]]; then make distcheck DISTCHECK_CONFIGURE_FLAGS="$CONFIGURE_FLAGS"; fi
- if [[ "$CHUTNEY" != "" ]]; then make test-network-forgiving; fi

after_failure:
## configure will leave a log file with more details of config failures.
## But the log is too long for travis' rendered view, so tail it.
- tail -1000 config.log || echo "tail failed"
## `make check` will leave a log file with more details of test failures.
- if [[ "$DISTCHECK" == "" ]]; then cat test-suite.log || echo "cat failed"; fi
- if [[ "$SKIP_MAKE_CHECK" == "" ]]; then cat test-suite.log || echo "cat failed"; fi
## `make distcheck` puts it somewhere different.
- if [[ "$DISTCHECK" != "" ]]; then make show-distdir-testlog || echo "make failed"; fi
- if [[ "$DISTCHECK" != "" ]]; then make show-distdir-core || echo "make failed"; fi
Expand Down
5 changes: 5 additions & 0 deletions Makefile.am
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,11 @@ need-chutney-path:
test-network: need-chutney-path $(TESTING_TOR_BINARY) src/tools/tor-gencert
$(top_srcdir)/src/test/test-network.sh $(TEST_NETWORK_FLAGS)

test-network-forgiving: need-chutney-path $(TESTING_TOR_BINARY) src/tools/tor-gencert
@ export TEST_NETWORK_FLAGS="$(TEST_NETWORK_FLAGS)"; \
export tor_srcdir=$(top_srcdir); \
$(top_srcdir)/src/test/test-network-forgiving.sh $(TEST_NETWORK_FLAGS)

# Run all available tests using automake's test-driver
# only run IPv6 tests if we can ping6 ::1 (localhost)
# only run IPv6 tests if we can ping ::1 (localhost)
Expand Down
3 changes: 3 additions & 0 deletions changes/chutney_ci
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
o Minor features (continuous integration):
- Our Travis configuration now uses Chutney to run some network
integration tests automatically. Closes ticket 29280.
24 changes: 24 additions & 0 deletions src/test/test-network-forgiving.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
#!/bin/sh
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we put test-network-forgiving.sh in chutney/tools rather than tor/src/test?
It would be a useful script to have in chutney, and chutney's CI may need it.
(We moved almost all of our chutney-specific scripts to chutney many releases ago. tor/src/test/test-network.sh is a stub that finds chutney/tools/test-network.sh. I want to avoid any more chutney-specific scripts in Tor.)

But if it's in chutney, how can we reliably find it from the Makefile, without duplicating the logic in tor/src/test/test-network.sh?

Maybe we could make the code in test-network-forgiving.sh part of chutney/tools/test-network.sh?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to have this become part of test-network.sh eventually, and I tried to do that at first, but I think for now this is a good compromise. The issue I had was that the maint-0.2.9 version of test-network.sh and the master version of test-network.sh are pretty different, and I want to actually run the same version of test-network.sh that the associated Tor has.

Once maint-0.2.9 is EOL, I would be much less confused about folding this into the test-network.sh script that chutney has.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to have this become part of test-network.sh eventually, and I tried to do that at first, but I think for now this is a good compromise. The issue I had was that the maint-0.2.9 version of test-network.sh and the master version of test-network.sh are pretty different, and I want to actually run the same version of test-network.sh that the associated Tor has.

But tor's test-network.sh calls chutney's test-network.sh?

(It certainly does on 0.3.4. And it does on my macOS machine in 0.2.9. But it seems to be broken in our 0.2.9 CI on Linux, which probably means we need to backport some fixes to 0.2.9's test-network.sh. I'll track them down.)

So I think we should modify chutney's test-network.sh in chutney master, and leave tor's test-network.sh alone.

Here's how it's supposed to work:

Since 0.2.9.1-alpha, tor's test-network.sh automatically calls chutney's test-network.sh, using the same arguments it was passed. So any new features in chutney's test-network.sh can be used by calling tor's test-network.sh with the correct arguments.

And since 0.3.1.1-alpha, there's no useful code in tor's test-network.sh: all it does is search for chutney's test-network.sh and exec it.

Once maint-0.2.9 is EOL, I would be much less confused about folding this into the test-network.sh script that chutney has.

In 0.2.9, we decided not to add any more chutney-related features to tor. I'd like to keep that policy, because moving these kinds of scripts is a pain.


TEST_NET_RETRIES=${TEST_NET_RETRIES:-3}

n_tries=0
test_net_okay=0

while [ "${n_tries}" -lt "${TEST_NET_RETRIES}" ]; do
n_tries=$((n_tries + 1))
# shellcheck disable=SC2086
if "${top_srcdir:-.}/src/test/test-network.sh" ${TEST_NETWORK_FLAGS}; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we use the script directory instead of the current working directory?

Suggested change
if "${top_srcdir:-.}/src/test/test-network.sh" ${TEST_NETWORK_FLAGS}; then
if "${top_srcdir:-`dirname -- "$0"`}/src/test/test-network.sh" ${TEST_NETWORK_FLAGS}; then

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm. The script directory would mean that the default was something like scripts/test/src/test/test-network.sh, which won't work. Maybe the default should be the script directory, followed by /../.. ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you're right, we need to go back those two steps, then forward.

echo "Chutney ran successfully after ${n_tries} attempt(s)."
test_net_okay=1
break
fi
echo "Chutney failed on attempt ${n_tries}/${TEST_NET_RETRIES}."
done;

if [ "$test_net_okay" != 1 ]; then
echo "Too many chutney failures; failing."
exit 1;
fi

exit 0