[RFC]CI: add runtime test support #13315
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aparcar
force-pushed
the
runtime-tests
branch
2 times, most recently
from
9be157e
to
4742723
Compare
derekyerger
pushed a commit
to derekyerger/packages
that referenced
this pull request
Sep 6, 2020
Patches already merged and so removed: * 011-fix-ssl-build-use-have-npn.patch * 019-bpo-36216-Add-check-for-characters-in-netloc-that-normalize-to-separators-openwrtGH-12216.patch * 020-bpo-36216-Only-print-test-messages-when-verbose-openwrtGH-12291.patch * 021-2.7-bpo-35121-prefix-dot-in-domain-for-proper-subdom.patch * 022-bpo-30458-Disallow-control-chars-in-http-URLs-openwrtGH-13315.patch * 023-bpo-35907-Avoid-file-reading-as-disallowing-the-unnecessary-URL-scheme-in-urllib-openwrtGH-11842.patch * 027-bpo-38243-Escape-the-server-title-of-DocXMLRPCServer.patch * 028-bpo-34155-Dont-parse-domains-containing-openwrtGH-13079.patch Patches no longer necessary and so removed: * 017_lib2to3_fix_pyc_search.patch Signed-off-by: Jeffery To <jeffery.to@gmail.com> (cherry picked from 83b300a)
aparcar
force-pushed
the
runtime-tests
branch
9 times, most recently
from
07676f3
to
65e0a47
Compare
|
This PR fails because Go is not available on some architectures, however the new runtime tests work. The binaries for both x86/64 and armvirt/64 run, produce an output and exit with 0. The current Prometheus |
aparcar
force-pushed
the
runtime-tests
branch
7 times, most recently
from
a7351ef
to
64899e4
Compare
|
@jefferyto should Go packages fail for not supported architectures or silently skip? |
aparcar
force-pushed
the
runtime-tests
branch
8 times, most recently
from
08e3016
to
f9d7ba0
Compare
aparcar
force-pushed
the
runtime-tests
branch
17 times, most recently
from
88ad6c1
to
ac4c1cc
Compare
Run trivial check if the compiled binary works on the architecture. Signed-off-by: Paul Spooren <mail@aparcar.org>
aparcar
force-pushed
the
runtime-tests
branch
from
ac4c1cc
to
c5e17df
Compare
Additional to manual runtime tests this CI addition runs a custom test script per package. Ideally this lowers the errors of package bumps, something which is time consuming when done manually for multiple architectures. This CI uses the official OpenWrt containers and tries to install and run compiled packages. The run depends on the content of `test.sh`, which is an `ash` script. It's called with the *packge name* and *package version* as arguments. This allows different behaviour if a single package generates multiple IPK files. The version is usable for the most trivial runtime check, e.g. `tmux -V | grep "$2"`. The current approach uses the qus project[1] which contains multiple QEMU binaries to run various architectures. [1]: https://github.com/dbhi/qus Signed-off-by: Paul Spooren <mail@aparcar.org>
aparcar
force-pushed
the
runtime-tests
branch
from
c5e17df
to
d0c9cf4
Compare
|
This is labelled as "RFC" but I don't see any comments before this was merged. Also, is there any documentation on how to add a test.sh or what is should do? |
I talked a bit to @neheb via IRC who liked the idea.
Currently only in the commit message, I wasn't sure where to add this. Maybe on the bottom of the README.md? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a bit of a long shot but the idea is that binaries are tested inside a OpenWrt container with the specific architecture.
Currently we offer Docker contains in x86/64 and armvirt/{32,64}, however we could easily add more architectures.
The idea of
test.shis that it's a shell script that tests if the compiled tools has some minimal functionality.The attached example runs
prometheus -versionand checks if it exists cleanly. These tests could be more sophisticated. There is obviously no hardware emulation and much more is missing for authentic tests, but ideally it simplifies the lives of maintainers.It's not fully tested yet an more of a concept.