File upload with httpx from utf-8 str content (not encoded) leads to h11._util.LocalProtocolError: Too much data for declared Content-Length

[AntonOvsyannikov]

First check

• I added a very descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn't find it.

Example

Server

import uvicorn
from fastapi import FastAPI, File

app = FastAPI()


@app.put("/upload")
async def upload(file: bytes = File(...)):
    ...


if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=20000, log_level='trace')

Client

import asyncio

import httpx


async def main():
    cli: httpx.AsyncClient
    async with httpx.AsyncClient(base_url='http://127.0.0.1:20000') as cli:
        # await cli.put('/upload', files=[('file', ('sample.txt', 'Q'))])  # ok
        # await cli.put('/upload', files=[('file', ('sample.txt', 'Й'.encode()))])  # ok
        await cli.put('/upload', files=[('file', ('sample.txt', 'Й'))])  # exception, see below


if __name__ == "__main__":
    asyncio.run(main())

Description

Not sure it's about FastAPI or httpx, but suppose it should be useful here anyway.
Upload file with httpx.put (using files=) to FastAPI endpoint from not encoded utf-8 string content leads to exception, but should not, str is acceptable content type for httpx due to FileContent = Union[IO[str], IO[bytes], str, bytes]

Traceback (most recent call last):
  File "/Users/aovsyannikov/Documents/Projects/file_upload/client.py", line 15, in <module>
    asyncio.run(main())
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 583, in run_until_complete
    return future.result()
  File "/Users/aovsyannikov/Documents/Projects/file_upload/client.py", line 11, in main
    await cli.put('/upload', files=[('file', ('sample.txt', 'Й'))])  # exception, see below
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/httpx/_client.py", line 1680, in put
    timeout=timeout,
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/httpx/_client.py", line 1372, in request
    request, auth=auth, allow_redirects=allow_redirects, timeout=timeout
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/httpx/_client.py", line 1411, in send
    history=[],
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/httpx/_client.py", line 1448, in _send_handling_auth
    history=history,
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/httpx/_client.py", line 1476, in _send_handling_redirects
    response = await self._send_single_request(request, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/httpx/_client.py", line 1507, in _send_single_request
    ext={"timeout": timeout.as_dict()},
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/httpcore/_async/connection_pool.py", line 219, in arequest
    method, url, headers=headers, stream=stream, ext=ext
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/httpcore/_async/connection.py", line 105, in arequest
    return await self.connection.arequest(method, url, headers, stream, ext)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/httpcore/_async/http11.py", line 66, in arequest
    await self._send_request_body(stream, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/httpcore/_async/http11.py", line 112, in _send_request_body
    await self._send_event(event, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/httpcore/_async/http11.py", line 123, in _send_event
    bytes_to_send = self.h11_state.send(event)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/h11/_connection.py", line 469, in send
    data_list = self.send_with_data_passthrough(event)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/h11/_connection.py", line 502, in send_with_data_passthrough
    writer(event, data_list.append)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/h11/_writers.py", line 77, in __call__
    self.send_data(event.data, write)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/h11/_writers.py", line 97, in send_data
    raise LocalProtocolError("Too much data for declared Content-Length")
h11._util.LocalProtocolError: Too much data for declared Content-Length

Environment

• OS: macOS
• httpx Version: 0.16.1
• FastAPI Version: 0.61.2
• Python version: 3.7

[Kludex]

Probably about h11. Could you try the following command instead?

    uvicorn.run(app, host="0.0.0.0", port=20000, log_level='trace', http="httptools")

[AntonOvsyannikov]

Does not help. I forget to mention exception is on client side.

[ycd]

Can you try setting a content-type?

await cli.put('/upload', files=[('file', ('sample.txt', 'Й', 'text/plain'))]

If it doesn't work, please check the content-length of the outgoing request and incoming request using a debugger.

[AntonOvsyannikov]

If it doesn't work

It doesn't work.

check the content-length of the outgoing request and incoming request using a debugger

Sorry, I don't get how to do it, instead i captured outgoing packets from client using Wireshark, here them are.

Good, sending 'Q'.

E,@@¤N µ$Ý¢-÷4¯ëÿ
/ò=/ò7PUT /upload HTTP/1.1
host: 127.0.0.1:20000
accept: */*
accept-encoding: gzip, deflate
connection: keep-alive
user-agent: python-httpx/0.16.1
content-length: 173
content-type: multipart/form-data; boundary=2f1d3f2027a0826ff877afabffd2dbfa

Bad, sending 'Й'.

E,@@Â¥N ;âl ±fëÿ
/ò§/ò¡PUT /upload HTTP/1.1
host: 127.0.0.1:20000
accept: */*
accept-encoding: gzip, deflate
connection: keep-alive
user-agent: python-httpx/0.16.1
content-length: 173
content-type: multipart/form-data; boundary=2c94f4a2b7f73d0f2dba209aa27090fc

[tacozMacleo]

This seams to be a problem with how HTTPx calculates the content-length.
encode/httpx#1482

@AntonOvsyannikov Try using the Requests module instead and see if you can replicate the Problem. It happens with 'HTTPx' but not when I use 'requests':

import requests

def main():
    base_url='http://127.0.0.1:20000'
    # requests.put(f'{base_url}/upload', files=[('file', ('sample.txt', 'Q'))])  # ok
    # requests.put(f'{base_url}/upload', files=[('file', ('sample.txt', 'Й'.encode()))])  # ok
    requests.put(f'{base_url}/upload', files=[('file', ('sample.txt', 'Й'))])  # ok


if __name__ == "__main__":
    main()

[tiangolo]

I suspect this was changed recently, the content part has to be bytes now it seems.

Sorry for the long delay! 🙈 I wanted to personally address each issue/PR and they piled up through time, but now I'm checking each one in order.

[github-actions[bot]]

Assuming the original need was handled, this will be automatically closed now. But feel free to add more comments or create new issues or PRs.

[Kludex]

This was solved some time ago.