Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tests: run (some) servers on "any port" #5247

Merged
merged 9 commits into from Apr 19, 2020
Merged

tests: run (some) servers on "any port" #5247

merged 9 commits into from Apr 19, 2020

Conversation

@bagder
Copy link
Member

bagder commented Apr 16, 2020

Makes the test servers (sws and ftpserver) for HTTP, Gopher, FTP, IMAP, POP3, SMTP listen on a currently unused port and runtests adapts to that! Removed the need and use of a fixed port number for these tests.

Test case preprocessing

In order to work correctly with flexible port number for all test cases and all protocols, runtests now preprocesses every test case before it is started and the updated test case file is stored as log/test$num. Each protocol test server should now look for the preprocessed version first, then fall back to the original test file only if not found. This allows proper variable substitution in the entire test case and also makes it work for all protocols and test servers at once.

@bagder bagder added the tests label Apr 16, 2020
@bagder bagder force-pushed the bagder/tests-on-dynport branch from fda3f03 to ce5ccbc Apr 17, 2020
@bagder bagder marked this pull request as ready for review Apr 17, 2020
@xquery
Copy link
Member

xquery commented Apr 18, 2020

what is the interaction with absolute test numbers eg. take for example test2054 ... do we want to randomise that port ?

@bagder
Copy link
Member Author

bagder commented Apr 18, 2020

I don't think so. That test verifies that the logic works with host + (fixed) port number. I don't think that's a problem. It won't connect to those ports anyway. The port number it actually connects to will be "random" though...

@xquery
Copy link
Member

xquery commented Apr 18, 2020

apologies for being slow on the uptake - do these changes mean there is now no way to absolutely specify a port ?

@bagder
Copy link
Member Author

bagder commented Apr 18, 2020

Sure, just specify a number and that number will be used. But the port numbers for test servers should never be specified like that, not now and not before either. Up until now however, such mistakes were easily let through since it works for most test runs, while this PR will make them never work (well, except for one in many thousands).

bagder added 5 commits Apr 16, 2020
Makes the test servers for HTTP and Gopher pop up on a currently unused
port and runtests adapts to that!

Closes #5247
The runtests script now always performs variable replacement on the
entire test source file before the test gets executed, and saves the
updated version in a temporary file (log/test[num]) so that all test
case readers/servers can use that version (if present) and thus enjoy
the powers of test case variable substitution.

This is necessary to allow complete port number freedom.

Test 309 is updated to work with a non-fixed port number thanks to this.
@bagder bagder force-pushed the bagder/tests-on-dynport branch from a05fa6c to f99b007 Apr 18, 2020
bagder added 4 commits Apr 18, 2020
FTP, IMAP, POP3, SMTP and their IPv6 versions are now all on dynamic
ports

Test 842-845 are unfortunately a bit hard to move over to this concept
right now and require "default port" still...
@bagder bagder changed the title tests: run the sws server on "any port" tests: run (some) servers on "any port" Apr 19, 2020
@bagder bagder closed this in 80d6515 Apr 19, 2020
@bagder bagder merged commit 003a7b5 into master Apr 19, 2020
42 of 48 checks passed
42 of 48 checks passed
fuzzing
Details
macos normal
Details
macos debug
Details
macos libssh2
Details
macos c-ares
Details
macos HTTP only
Details
macos SecureTransport metalink
Details
macos OpenSSL metalink
Details
macos OpenSSL metalink
Details
macos LibreSSL metalink
Details
macos torture
Details
macos cmake clang OpenSSL
Details
macos cmake clang LibreSSL
Details
macos cmake gcc-8 OpenSSL
Details
macos cmake gcc-8 LibreSSL
Details
macos cmake gcc-9 OpenSSL
Details
macos cmake gcc-9 LibreSSL
Details
continuous-integration/appveyor/pr AppVeyor build failed
Details
curl.curl Build #20200418.5 failed
Details
curl.curl (windows msys1 mingw debug schannel) windows msys1 mingw debug schannel was canceled
Details
curl.curl (windows msys1 mingw64 debug openssl) windows msys1 mingw64 debug openssl was canceled
Details
curl.curl (windows msys1 mingw64 debug schannel) windows msys1 mingw64 debug schannel was canceled
Details
LGTM analysis: Python No code changes detected
Details
Codacy/PR Quality Review Up to standards. A positive pull request.
Details
FreeBSD FreeBSD:freebsd-11-3-snap Task Summary
Details
FreeBSD FreeBSD:freebsd-12-1 Task Summary
Details
FreeBSD FreeBSD:freebsd-13-0-snap Task Summary
Details
LGTM analysis: C/C++ No new or fixed alerts
Details
buildbot/curl-unthreaded-solaris10-i386 Build done.
Details
buildbot/curl-unthreaded-solaris10-sparc Build done.
Details
buildbot/curl_winssl_cross_x64 Build done.
Details
buildbot/curl_winssl_cross_x64_dbg Build done.
Details
buildbot/curl_winssl_cross_x86 Build done.
Details
buildbot/curl_winssl_cross_x86_dbg Build done.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
curl.curl (linux ubuntu HTTP only) linux ubuntu HTTP only succeeded
Details
curl.curl (linux ubuntu default) linux ubuntu default succeeded
Details
curl.curl (linux ubuntu sync resolver) linux ubuntu sync resolver succeeded
Details
curl.curl (linux ubuntu w/o HTTP/SMTP/IMAP) linux ubuntu w/o HTTP/SMTP/IMAP succeeded
Details
curl.curl (linux ubuntu w/o IPv6) linux ubuntu w/o IPv6 succeeded
Details
curl.curl (linux_torture ubuntu torture tests) linux_torture ubuntu torture tests succeeded
Details
curl.curl (windows msys1 mingw debug openssl) windows msys1 mingw debug openssl succeeded
Details
curl.curl (windows msys1 mingw32 debug openssl) windows msys1 mingw32 debug openssl succeeded
Details
curl.curl (windows msys1 mingw32 debug schannel) windows msys1 mingw32 debug schannel succeeded
Details
curl.curl (windows msys2 mingw32 debug openssl) windows msys2 mingw32 debug openssl succeeded
Details
curl.curl (windows msys2 mingw32 debug schannel) windows msys2 mingw32 debug schannel succeeded
Details
curl.curl (windows msys2 mingw64 debug openssl) windows msys2 mingw64 debug openssl succeeded
Details
curl.curl (windows msys2 mingw64 debug schannel) windows msys2 mingw64 debug schannel succeeded
Details
@bagder bagder deleted the bagder/tests-on-dynport branch Apr 19, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.