Exception 'The server returned an invalid or unrecognized response.' when netcoreapp2.1 and System.ServiceModel.* > 4.4.4

[wiktor-golonka]

Hi,
We started getting this exception since we changed target framework from netcoreapp2.0 netcoreapp2.1.
Swapping it back to netcoreapp2.0 solves this issue.
The exception is thrown immediately on the first submitted request.

It seems to be related to dependencies:

<PackageReference Include="System.ServiceModel.Duplex" Version="4.5.1" />
<PackageReference Include="System.ServiceModel.Http" Version="4.5.1" />
<PackageReference Include="System.ServiceModel.NetTcp" Version="4.5.1" />
<PackageReference Include="System.ServiceModel.Security" Version="4.5.1" />

Changing these all back to version 4.4.4 also solves this issue.
So currently we know two ways of solving this:
a) downgrade these 4 dependencies to version 4.4.4
or
b) downgrade target framework from netcoreapp2.1 to netcoreapp2.0

WSDL file is available here:
https://service.bloomberg.com/assets/dl/dlws.wsdl

I'm not sure how helpful it will be as it may require authentication key/certificate which unfortunately I can't provide.

Our code looks like that:

Channel creating:

var result = new BasicHttpsBinding(BasicHttpsSecurityMode.Transport)
{
    MaxBufferSize = int.MaxValue,
    ReaderQuotas = System.Xml.XmlDictionaryReaderQuotas.Max,
    MaxReceivedMessageSize = int.MaxValue,
    AllowCookies = true,
    Security =
    {
        Transport = new HttpTransportSecurity
        {
            ClientCredentialType = HttpClientCredentialType.Certificate
        },
        Mode = BasicHttpsSecurityMode.Transport
    }
};

var endpointAddress = new EndpointAddress("https://endpoint.Address");

var channelFactory = new ChannelFactory<PerSecurityWS>(result, endpointAddress)
{
    Credentials =
    {
        ClientCertificate =
        {
            Certificate = new X509Certificate2(certificate, password)
        }
    }
};

this.webService = channelFactory.CreateChannel();

Exception is thrown here:

var submitResponse = await this.webService.submitGetDataRequestAsync(new submitGetDataRequestRequest(request));

where submitGetDataRequestAsync is

[System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Advanced)]
System.Threading.Tasks.Task<namespace.Dlws.submitGetDataRequestResponse> namespace.Dlws.PerSecurityWS.submitGetDataRequestAsync(namespace.Dlws.submitGetDataRequestRequest request)
{
    return base.Channel.submitGetDataRequestAsync(request);
}

Exception:

The server returned an invalid or unrecognized response.
Stack trace:

   at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result)
   at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
   at System.ServiceModel.Channels.ServiceChannelProxy.TaskCreator.<>c__DisplayClass1_0.<CreateGenericTask>b__0(IAsyncResult asyncResult)
--- End of stack trace from previous location where exception was thrown ---

Inner exception:

The server returned an invalid or unrecognized response.
Inner exception stack trace:

   at System.Net.Http.HttpConnection.ThrowInvalidHttpResponse()
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at System.ServiceModel.Channels.HttpChannelFactory`1.HttpClientRequestChannel.HttpClientChannelAsyncRequest.SendRequestAsync(Message message, TimeoutHelper timeoutHelper)

PackageReference:

<PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="2.6.1" />
<PackageReference Include="Microsoft.CodeAnalysis.FxCopAnalyzers" Version="2.6.1" />
<PackageReference Include="Microsoft.Extensions.Configuration" Version="2.1.1" />
<PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="2.1.1" />
<PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions" Version="2.1.1" />
<PackageReference Include="Newtonsoft.Json" Version="11.0.2" />
<PackageReference Include="System.Net.Http" Version="4.3.3" />
<PackageReference Include="System.ServiceModel.Duplex" Version="4.5.1" />
<PackageReference Include="System.ServiceModel.Http" Version="4.5.1" />
<PackageReference Include="System.ServiceModel.NetTcp" Version="4.5.1" />
<PackageReference Include="System.ServiceModel.Security" Version="4.5.1" />
<PackageReference Include="System.ValueTuple" Version="4.5.0" />

[wiktor-golonka]

I reopened this issue as it also seems to be related to these 4 dependencies:

<PackageReference Include="System.ServiceModel.Duplex" Version="4.5.1" />
<PackageReference Include="System.ServiceModel.Http" Version="4.5.1" />
<PackageReference Include="System.ServiceModel.NetTcp" Version="4.5.1" />
<PackageReference Include="System.ServiceModel.Security" Version="4.5.1" />

Downgrading all of them back to 4.4.4 solves the issue.

[wiktor-golonka]

It seems to be similar issue to: #2960

[mconnew]

@wiktor-golonka, do you know which server stack you are communication with? Without a valid client certificate, I'm unable to get any headers from the service to see what service stack is being used. The main change we made between 4.4.4 and 4.5.0 is adding the Expect: 100-Continue header. The purpose of this is to prevent the request body being sent to the server until after authentication has completed. Looking at the call stack HttpConnection.SendAsyncCore is calling HttpConnection.ThrowInvalidHttpResponse(). This only happens in one place in that method. The condition that causes this is when the server has responded with a 100 Continue response status line which is then not followed by a blank line.
This looks to be a bug in HttpClientHandler in corefx as according to RFC7231, headers are allowed to be sent after a 100 status code. I'll open an issue in corefx to track this problem.

[wiktor-golonka]

@mconnew:
I can't get from Fiddler the exact response with 100-Continue header but after looking into session state I see that definitely this header has been sent inside the response.

My post request:

POST https://dlws.bloomberg.com/dlps HTTP/1.1
Cache-Control: no-cache, max-age=0
SOAPAction: "submitGetHistoryRequest"
Expect: 100-continue
Accept-Encoding: gzip, deflate
Content-Type: text/xml; charset=utf-8
Content-Length: 1133
Host: dlws.bloomberg.com

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">    
    <s:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
        <submitGetHistoryRequest xmlns="http://services.bloomberg.com/datalicense/dlws/ps/20071001">
            (REQUEST)
        </submitGetHistoryRequest>
    </s:Body>
</s:Envelope>

and the response that I'm receiving:

HTTP/1.1 200 OK
Connection: Keep-Alive
Date: Fri, 27 Jul 2018 10:03:47+0000
Via: BAS/1.0 nybasv1
Content-Length: 630
Content-Type: text/xml; charset=utf-8
Server: Apache-Coyote/1.1
bb-request-id: (bb-requestId)
com.bloomberg.bas-recipientTask: machine=0; instance=0;
com.bloomberg.bas-serviceId: serviceId
com.bloomberg.bas-serviceVersion: 3.1
Set-Cookie: machineNumber=0; path=/
X-Powered-By: Servlet 2.4; JBoss-4.3.0.GA_CP04 (build: SVNTag=JBPAPP_4_3_0_GA_CP04 date=200902200048)/JBossWeb-2.0

<?xml version="1.0" encoding="UTF-8" ?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <soap:Body>
        <dlws:submitGetHistoryResponse 
            xmlns='http://services.bloomberg.com/datalicense/dlws/ps/20071001' 
            xmlns:dlws="http://services.bloomberg.com/datalicense/dlws/ps/20071001" 
            xmlns:env="http://schemas.xmlsoap.org/soap/envelope/">
            <dlws:statusCode>
                <dlws:code>0</dlws:code>
                <dlws:description>Success</dlws:description>
            </dlws:statusCode>
            <dlws:requestId>
                (bb-requestId)
            </dlws:requestId>
            <dlws:responseId>
                (bb-responseId)
            </dlws:responseId>
        </dlws:submitGetHistoryResponse>
    </soap:Body>
</soap:Envelope>

Fiddler session state:

SESSION STATE: Aborted.
Request Entity Size: 1133 bytes.
Response Entity Size: 630 bytes.

== FLAGS ==================
BitFlags: [IsHTTPS, ClientPipeReused, ServerPipeReused] 0x19
X-ABORTED-WHEN: SendingResponse
X-CLIENTIP: 127.0.0.1
X-CLIENTPORT: 52193
X-EGRESSPORT: 52194
X-FIDDLER-STREAM1XX: Returned a HTTP/100 message from the server.
X-HOSTIP: 69.191.193.90
X-PROCESSINFO: dotnet:14752
X-RESPONSEBODYTRANSFERLENGTH: 630
X-SERVERSOCKET: REUSE ServerPipe#3

== TIMING INFO ============
ClientConnected:	11:03:47.258
ClientBeginRequest:	11:03:47.618
GotRequestHeaders:	11:03:47.618
ClientDoneRequest:	11:03:48.635
Determine Gateway:	0ms
DNS Lookup: 		0ms
TCP/IP Connect:	0ms
HTTPS Handshake:	0ms
ServerConnected:	11:03:47.359
FiddlerBeginRequest:	11:03:48.635
ServerGotRequest:	11:03:48.635
ServerBeginResponse:	11:03:48.707
GotResponseHeaders:	11:03:48.929
ServerDoneResponse:	11:03:48.929
ClientBeginResponse:	11:03:48.929
ClientDoneResponse:	11:03:48.929

	Overall Elapsed:	0:00:01.311

The response was buffered before delivery to the client.

== WININET CACHE INFO ============
This URL is not present in the WinINET cache. [Code: 2]
* Note: Data above shows WinINET's current cache state, not the state at the time of the request.
* Note: Data above shows WinINET's Medium Integrity (non-Protected Mode) cache only.

[mconnew]

Did using Fiddler cause the request to work? The fiddler session state says the response was buffered before delivery to the client which means that Fiddler would handle any optional headers returned with the 100 continue response. In that situation, HttpClientHandler wouldn't be presented with the extra headers and the request would be successful.
One workaround that might work for you is to disable the usage of SocketsHttpHandler. That can be done by calling AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false).

[wiktor-golonka]

@mconnew: Thank you.
Calling AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false) also helps.
The request fails whether I'm using Fiddler or not. I tried to decrypt packets with Wireshark and Microsoft Message Analyzer but they both don't support this encryption method so I can't see any of the response headers.

[maxiptah]

Can confirm the issue. Our WCF service (client certificates authentication) stopped working after upgrade. Downgrading to 4.4 resolved the issue. Will the switch AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false) introduce some problems if we are using SignalR? This relies on sockets heavily.

[wiktor-golonka]

@maxiptah
We are also using SignalR and AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false) and we haven't noticed any problems so far.

[mconnew]

@wiktor-golonka and @maxiptah, if you update to the latest version of .NET Core, this problem should now be fixed. Can you please verify that you aren't seeing this problem any longer?