New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
travis: upgrade to bionic, clang-9, improve readability #5370
Conversation
7457d79
to
04a4467
Compare
This change was prompted by flaky tests due to failure of apt to download from the PPA. Removal of external dependencies should make CI more reliable. The new Ubuntu version also uncovered a build failure with libssh (not libssh2), a workaround is prepended as well as the CMake config I used to test it:
|
289d98e
to
f31ca1b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In general this is looking good to me. I would just recommend in the future to do separate commits for whitespace and logic changes, so that it is easier to follow them separately.
apt: | ||
sources: | ||
- *common_sources | ||
- llvm-toolchain-xenial-7 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am just wondering why all the llvm-toolchain-*
packages are no longer needed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ubuntu 14.04 (Trusty) shipped with clang 3.4 by default,
Ubuntu 16.04 (Xenial, Travis CI default) included clang 3.8 by default (with clang-6 as option), and
Ubuntu 18.04 (Bionic) included clang 6 by default with clang-9 as option.
In order to use a more recent version of Clang with better diagnostics, the old config therefore used the LLVM apt repo.
Clang 10 is the latest stable version upstream and an option in Ubuntu 20.04, but is not an option in Travis CI.
Regarding the whitespace changes, sorry for making it harder to review. Since I have to rebase this anyway, do you prefer me to make a new whitespace-only change before the other functional changes? |
Yes please, that would be nice! I actually wanted to review this too, but haven't yet because it was hard because of the whitespace changes. |
f31ca1b
to
26187bb
Compare
Rebased and separated the whitespace-only change, it should hopefully be easier to follow now. Some review guidance:
Once again, apologies for making it harder to review before. Please let me know if I can make it even easier for review. |
.travis.yml
Outdated
config: | ||
retries: true | ||
sources: &common_sources | ||
- ubuntu-toolchain-r-test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we want to update to GCC 9, we need to add this again. So I'd maybe rather update to GCC 9 right now instead of removing this and adding it again later?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Clang 10 is the latest stable version upstream and an option in Ubuntu 20.04,
but is not an option in Travis CI.
Clang 10 is available in Travis, using:
dist: focal
compiler: clang
addons:
apt:
sources:
sourceline:
- deb http://apt.llvm.org/focal/ llvm-toolchain-focal-10 main
key_url:
- https://apt.llvm.org/llvm-snapshot.gpg.key
packages:
- clang-10
|
gcc-8 seems recent enough, removing the external PPA should reduce the number CI failures due to network issues. If a newer GCC version is desired (the current stable is 10), it could be added at the top-level.
Ah ok, what I meant it that it is not available in the standard repos. The same reason as for GCC applies, I want to reduce build failures due to external effects. If you prefer to have the latest and greatest compiler for both, I could add another change on top of it. Hopefully with the |
26187bb
to
734c4cb
Compare
Dropped the libssh patches and try Ubuntu 20.04 focal instead:
The commit message has also been updated, I am not using focal everywhere yet because Travis CI does not document this as option (it is work in progress and probably in very alpha state). I have no idea whether this build will even succeed. If focal becomes available, then compilers can be upgraded to gcc-10 and clang-10. Additionally, I pushed a patch on top of it that simplifies the quiche build instructions (from #5359). |
734c4cb
to
30c6192
Compare
The focal build failed due to python-impacket being removed, leaving python3-impacket (since 20.04) as only option. For simplicity I've just skipped installation of this now. When moving to focal for all future builds, we have to make sure that I also had to drop the quiche improvement, it uncovered a bug in configure.ac related to pthreads detection... argh. Will work on it later. |
Or, switch from "python" to "python3" when invoking it.
|
That is the long-term plan, but it will require installing impacket through pip or upgrading to focal since bionic does not have python3-impacket for SMB tests. python2 is EOL, so it would be reasonable to force python3 in a future change. A few CI jobs failed due to test flakiness (or a setup failure in case of FreeBSD). @dfandrich are you also happy with the current patch or would you like me to make further changes before merging? |
Automatically apply a consistent indentation with: python3 -c 'from ruamel.yaml import YAML;y=YAML();d=y.load(open(".travis.yml"));y.width=500;y.dump(d,open(".travis.yml.new","w"))' followed by manually re-indenting three comments.
Changes, partially to reduce build failures from external dependencies: - Upgrade Ubuntu and drop unnecessary third-party repos. - Properly clone apt config to ensure retries. - Upgrade to clang-9 from the standard repos. - Use Ubuntu 20.04 focal for the libssh build, use of ssh_get_publickey fails on -Werror=deprecated-declarations in Ubuntu 18.04. Do not use focal everywhere yet since Travis CI has not documented this option. In focal, python-impacket (Py2.7) has been removed, leaving only python3-impacket. Since it is only needed for SMB tests and not SSH, skip it for the libssh job since it might need more work. - apt: Remove gcc-8 and libstdc++-8-dev, already installed via g++-8. Non-functional cleanups: - Simplify test matrix, drop redundant os and compiler keys. - Deprecation fixes: remove sudo, rename matrix -> jobs. - Every job has an 'env' key, put this key first in a list item.
30c6192
to
7534bb4
Compare
Removed CC/CXX override for focal and used YAML anchors to simplify future clang version upgrades. The GCC override can be changed in a similar way in the future when the version is bumped. I like this diffstat (it can be cut down further if libpsl-dev/libbrotli-dev are moved to
diff since previous changediff --git a/.travis.yml b/.travis.yml
index eb66a2c9d..6f7e67bfb 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -61,7 +61,6 @@ jobs:
- libssh2-1-dev
- env:
- T=normal C=--with-libssh
- - OVERRIDE_CC="CC=gcc-8" OVERRIDE_CXX="CXX=g++-8"
# Avoid bionic, its pre-release libssh version triggers deprecation warnings.
dist: focal
addons:
@@ -162,64 +161,59 @@ jobs:
- libbrotli-dev
- env:
- T=debug
- - OVERRIDE_CC="CC=clang-9" OVERRIDE_CXX="CXX=clang++-9"
+ - &clang OVERRIDE_CC="CC=clang-9" OVERRIDE_CXX="CXX=clang++-9"
compiler: clang
addons:
apt:
<<: *common_apt
packages:
- - *common_packages
- - clang-9
+ - &clang_packages [*common_packages, clang-9]
- libpsl-dev
- libbrotli-dev
- env:
- T=debug C="--enable-alt-svc"
- - OVERRIDE_CC="CC=clang-9" OVERRIDE_CXX="CXX=clang++-9"
+ - *clang
compiler: clang
addons:
apt:
<<: *common_apt
packages:
- - *common_packages
- - clang-9
+ - *clang_packages
- libpsl-dev
- libbrotli-dev
- env:
- T=debug C="--with-mbedtls --without-ssl"
- - OVERRIDE_CC="CC=clang-9" OVERRIDE_CXX="CXX=clang++-9"
+ - *clang
compiler: clang
addons:
apt:
<<: *common_apt
packages:
- - *common_packages
- - clang-9
+ - *clang_packages
- libpsl-dev
- libbrotli-dev
- libmbedtls-dev
- env:
- T=debug C="--with-gnutls --without-ssl"
- - OVERRIDE_CC="CC=clang-9" OVERRIDE_CXX="CXX=clang++-9"
+ - *clang
compiler: clang
addons:
apt:
<<: *common_apt
packages:
- - *common_packages
- - clang-9
+ - *clang_packages
- libgnutls28-dev
- libpsl-dev
- libbrotli-dev
- env:
- T=debug C="--with-nss --without-ssl" NOTESTS=1 CPPFLAGS="-isystem /usr/include/nss"
- - OVERRIDE_CC="CC=clang-9" OVERRIDE_CXX="CXX=clang++-9"
+ - *clang
compiler: clang
addons:
apt:
<<: *common_apt
packages:
- - *common_packages
- - clang-9
+ - *clang_packages
- libnss3-dev
- libpsl-dev
- libbrotli-dev
@@ -241,15 +235,14 @@ jobs:
- libbrotli-dev
- env:
- T=cmake NGTCP2=yes C="-DUSE_NGTCP2=ON"
- - OVERRIDE_CC="CC=clang-9" OVERRIDE_CXX="CXX=clang++-9"
+ - *clang
- PKG_CONFIG_PATH="$HOME/ngbuild/lib/pkgconfig"
compiler: clang
addons:
apt:
<<: *common_apt
packages:
- - *common_packages
- - clang-9
+ - *clang_packages
- libpsl-dev
- libbrotli-dev
- env:
@@ -276,51 +269,47 @@ jobs:
- libbrotli-dev
- env:
- T=fuzzer
- - OVERRIDE_CC="CC=clang-9" OVERRIDE_CXX="CXX=clang++-9"
+ - *clang
compiler: clang
addons:
apt:
<<: *common_apt
packages:
- - *common_packages
- - clang-9
+ - *clang_packages
- libpsl-dev
- libbrotli-dev
- env:
- T=tidy
- - OVERRIDE_CC="CC=clang-9" OVERRIDE_CXX="CXX=clang++-9"
+ - *clang
compiler: clang
addons:
apt:
<<: *common_apt
packages:
- - *common_packages
- - clang-9
+ - *clang_packages
- clang-tidy-9
- libpsl-dev
- libbrotli-dev
- env:
- T=scan-build
- - OVERRIDE_CC="CC=clang-9" OVERRIDE_CXX="CXX=clang++-9"
+ - *clang
compiler: clang
addons:
apt:
<<: *common_apt
packages:
- - *common_packages
- - clang-9
+ - *clang_packages
- libpsl-dev
- libbrotli-dev
- env:
- T=debug CFLAGS="-fsanitize=address,undefined,signed-integer-overflow -fno-sanitize-recover=undefined,integer -Wformat -Werror=format-security -Werror=array-bounds -g" LDFLAGS="-fsanitize=address,undefined -fno-sanitize-recover=undefined,integer" LIBS="-ldl -lubsan"
- - OVERRIDE_CC="CC=clang-9" OVERRIDE_CXX="CXX=clang++-9"
+ - *clang
compiler: clang
addons:
apt:
<<: *common_apt
packages:
- - *common_packages
- - clang-9
+ - *clang_packages
- libpsl-dev
- libbrotli-dev
- env: |
Looks good to me! Unfortunately, g++-8 (and gcc-8) are hard to get rid of as they're set inside before_script.sh for building nghttp2, otherwise they could also be removed on focal. focal ships nghttp2 1.40.0 though, so it doesn't even need to be built there. Probably something for another PR. |
And we're not particularly married to 1.40.0 either... if a version is provide by the distro, I think that's a better and easier route! |
In that case, we don't need to build it at all - Bionic has 1.30.0. Then we only need a C++ compiler for the BoringSSL build. |
Changes, partially to reduce build failures from external dependencies: - Upgrade Ubuntu and drop unnecessary third-party repos. - Properly clone apt config to ensure retries. - Upgrade to clang-9 from the standard repos. - Use Ubuntu 20.04 focal for the libssh build, use of ssh_get_publickey fails on -Werror=deprecated-declarations in Ubuntu 18.04. Do not use focal everywhere yet since Travis CI has not documented this option. In focal, python-impacket (Py2.7) has been removed, leaving only python3-impacket. Since it is only needed for SMB tests and not SSH, skip it for the libssh job since it might need more work. - apt: Remove gcc-8 and libstdc++-8-dev, already installed via g++-8. Non-functional cleanups: - Simplify test matrix, drop redundant os and compiler keys. - Deprecation fixes: remove sudo, rename matrix -> jobs. - Every job has an 'env' key, put this key first in a list item. Closes #5370
A major cleanup of the travis job matrix. Before doing so, a libssh fix was needed to fix the Ubuntu 18.04 build. As libssh was missing in CMake, I had to add it in order to test it locally.
Changes:
Non-functional cleanups:
Simplify test matrix, drop redundant os and compiler keys.
Deprecation fixes: remove sudo, rename matrix -> jobs.
Every job has an 'env' key, put this key first in a list item.
Use consistent indentation, automatically done with: