Add support for HTTP redirecti in ASIO-based http_client

[garethsb]

HTTP redirect is currently supported by the WinHttp back-end, but has to be handled manually in user code when C++ REST SDK is built with the ASIO-based back-end. This kind of platform difference is frustrating for users.

Solving this issue has been requested several times, e.g. #222. A previous PR #373 was abandoned.

This PR enables us to pass the 'outside tests' that were identified as failing in #27, even after reverting both PR #499 (which worked around the problem by permitting MovedPermanent in the test) and another similar case from commit 4e19c0c.

If this PR seems acceptable, it should be straightforward to use the new http_client_config options in the http_client_winhttp implementation also, which would solve #171 more elegantly.

Thoughts on how the headers sent with the redirect request are decided would be welcome!

[garethsb]

Thoughts on how the headers sent with the redirect request are decided would be welcome!

Current implementation in this PR is influenced by whatwg/fetch#977.
I note that the WinHTTP docs say

Headers are transferred across redirects. This can be a security issue. To avoid having headers transferred when a redirect occurs, use the WINHTTP_STATUS_CALLBACK callback to correct the specific headers when a redirect occurs.

Since this PR gives user code the control to disable automatic redirect (by http_client_config::set_max_redirects), such a callback seems unnecessary.

[garethsb]

it should be straightforward to use the new http_client_config options in the http_client_winhttp implementation also, which would solve #171 more elegantly.

I have attempted to prove this by adding the following snippet to winhttp_client::send_request...

        if (client_config().max_redirects() == 0)
        {
            // Disable auto redirects
            DWORD redirectPolicy = WINHTTP_OPTION_REDIRECT_POLICY_NEVER;
            if (!WinHttpSetOption(winhttp_context->m_request_handle,
                WINHTTP_OPTION_REDIRECT_POLICY,
                &redirectPolicy,
                sizeof(redirectPolicy)))
            {
                auto errorCode = GetLastError();
                request->report_error(errorCode, build_error_msg(errorCode, "Setting redirect policcy"));
                return;
            }
            // Note, using WINHTTP_OPTION_DISABLE_FEATURE with WINHTTP_DISABLE_REDIRECTS here doesn't seem to work
        }
        else
        {
            // Set max auto redirects
            DWORD maxRedirects = (DWORD)client_config().max_redirects();
            if (!WinHttpSetOption(winhttp_context->m_request_handle,
                                  WINHTTP_OPTION_MAX_HTTP_AUTOMATIC_REDIRECTS,
                                  &maxRedirects,
                                  sizeof(maxRedirects)))
            {
                auto errorCode = GetLastError();
                request->report_error(errorCode, build_error_msg(errorCode, "Setting max automatic redirects"));
                return;
            }

            // (Dis)allow HTTPS to HTTP redirects
            DWORD redirectPolicy = client_config().https_to_http_redirects()
                ? WINHTTP_OPTION_REDIRECT_POLICY_ALWAYS
                : WINHTTP_OPTION_REDIRECT_POLICY_DISALLOW_HTTPS_TO_HTTP;
            if (!WinHttpSetOption(winhttp_context->m_request_handle,
                                  WINHTTP_OPTION_REDIRECT_POLICY,
                                  &redirectPolicy,
                                  sizeof(redirectPolicy)))
            {
                auto errorCode = GetLastError();
                request->report_error(errorCode, build_error_msg(errorCode, "Setting redirect policcy"));
                return;
            }
        }

As noted in the comment, I found that WINHTTP_OPTION_REDIRECT_POLICY_NEVER works but WINHTTP_DISABLE_REDIRECTS doesn't in my environment.

Also, requesting a URL which requires more redirects than the WINHTTP_OPTION_MAX_HTTP_AUTOMATIC_REDIRECTS option unfortunately results in the WinHttpReceiveResponse exception being generated by the completion_callback rather than the last 3xx response being returned to the application. That's obviously the current behaviour (at the WinHTTP default of 10) but leaves something to be desired I feel?

[garethsb]

Also, requesting a URL which requires more redirects than the WINHTTP_OPTION_MAX_HTTP_AUTOMATIC_REDIRECTS option unfortunately results in the WinHttpReceiveResponse exception being generated by the completion_callback rather than the last 3xx response being returned to the application. That's obviously the current behaviour (at the WinHTTP default of 10) but leaves something to be desired I feel?

By adding WINHTTP_CALLBACK_FLAG_REDIRECT to the WinHttpSetStatusCallback call, we do get WINHTTP_CALLBACK_STATUS_REDIRECT in the completion_callback, at which point the response status code and headers can be read. But only up to the redirect before the specified WINHTTP_OPTION_MAX_HTTP_AUTOMATIC_REDIRECTS limit! So if we were to set that value one higher than specified by http_client_config::max_redirects(), and track the number of redirects in the request context, it would be possible to return this last redirect response to user code at that point. (In my tests it appears that WinHTTP unfortunately does not make the body of the redirect response available after WINHTTP_CALLBACK_STATUS_REDIRECT, which agrees with an aged microsoft.public.winhttp newsgroup post I found.)

[BillyONeal]

I would really like to see those test cases since there are a ton of extractable conditionals here; everything else are nitpicks. Thanks for your contribution, this is great!

[BillyONeal]

Since this PR gives user code the control to disable automatic redirect (by http_client_config::set_max_redirects), such a callback seems unnecessary.

If users are going to be relying on this for security reasons having some of the backends ignore it seems to be .... problematic :)

[garethsb]

I realised this needs reworking. If I hoist the whole thing outside of the user-code http_client it works nicely, but right now it performs the redirect checks redundantly, due to the tail recursion in follow_redirect and the nested call in asio_client::propagate as a result of constructing the inner http_client!

I'll come back...

[garethsb]

@BillyONeal I've resolved the issue I noted and addressed some of your questions. If you have a chance to review commit 6a4830b, that'd be great.

How do you think we should best surface http_redirect_follower to allow url_to_follow to be tested? There isn't a convenient matching header file to http_client_asio.cpp.

After that I can either make the config options specific to the ASIO backend, or potentially make the WinHTTP additions to http_client_winhttp.cpp to obey the http_client_config::max_redirects and https_to_http_redirects.

Thanks.

[garethsb]

potentially make the WinHTTP additions to http_client_winhttp.cpp to obey the http_client_config::max_redirects and https_to_http_redirects.

Done.

[garethsb]

Now that this PR supports configurable HTTP redirection for WinHTTP as well as Boost.ASIO, I have added a whole new suite of functional test cases. These test that the most common scenarios behave similarly for both underlying implementations, and then document where differences are currently unavoidable.

I think I'm done with this, thanks, @BillyONeal.

[garethsb]

Btw, this may be a can of worms, but why are the tests disabled for the Pipelines WinHttpPAL job?

cpprestsdk/azure-pipelines.yml

Lines 207 to 210 in b94bc32

	cd build.debug/Release/Binaries
	#./test_runner *test.so
	cd ../../../build.release/Release/Binaries
	#./test_runner *test.so

[BillyONeal]

Btw, this may be a can of worms, but why are the tests disabled for the Pipelines WinHttpPAL job?

cpprestsdk/azure-pipelines.yml

Lines 207 to 210 in b94bc32

	cd build.debug/Release/Binaries
	#./test_runner *test.so
	cd ../../../build.release/Release/Binaries
	#./test_runner *test.so

Because the owners of "WinHttpPAL" don't support or care about those cases.

[BillyONeal]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[garethsb]

Test redirect_tests:follows_permanent_redirect failure on VS2015 platform is due to WinHTTP not handling 308 response before Windows 10.

[BillyONeal]

I'll make the suggestion change and merge

[garethsb]

I wasn't sure if this ought to be a compile-time or run-time check.

[garethsb]

Hmm, not sure what's going on here, why VersionHelpers.h doesn't compile, @BillyONeal?

[BillyONeal]

Thanks for your contribution!

[franksinankaya]

Btw, this may be a can of worms, but why are the tests disabled for the Pipelines WinHttpPAL job?

cpprestsdk/azure-pipelines.yml

Lines 207 to 210 in b94bc32

	cd build.debug/Release/Binaries
	#./test_runner *test.so
	cd ../../../build.release/Release/Binaries
	#./test_runner *test.so

Because the owners of "WinHttpPAL" don't support or care about those cases.

Please mention my name next time you need attention from winhttppal. The reason test suite is not fully enabled is that there are known problems with compression support.

If there is a way to run the suite while suppressing some tests, please let me know. I can certainly post an update.

[garethsb]

@franksinankaya the test_runner executable seems only to have a way to include tests by name etc. not exclude them:

cpprestsdk/Release/tests/common/TestRunner/test_runner.cpp

Lines 36 to 39 in f4124a9

	std::cout
	<< "Usage: testrunner.exe <test_binaries> [/list] [/listproperties] [/noignore] [/breakonerror] [/detectleaks]"
	<< std::endl;
	std::cout << " [/name:<test_name>] [/select:@key=value] [/loop:<num_times>]" << std::endl;

On the other hand, looking through the test suites, there are many instances of test cases being #if-ed out based on platform/compile-time config, so that may be an option?

[franksinankaya]

Ok, I will post a PR if this is the directory @BillyONeal wants to go.

[franksinankaya]

Or we could add exclusion support to the tool.