Multipart file encoding takes up a lot of memory

[mfleader]

Using httpx AsyncClient to post a file, takes up a lot of memory. Specifically posting a 10 GiB file caused the python process to go up to 20 GiB RAM for several seconds, and posting a 20 GiB file caused the python process to consume all of my available memory ~23 GiB for much longer.

Steps to reproduce the issue:

1. Create files for a given size

truncate -s 5M ~/5M.txt
truncate -s 10G ~/10G.txt
truncate -s 20G ~/20G.txt

2. Post a file

from pathlib import Path
import httpx

async def async_post_file_req(
    url: str, token: str, filepath: Path, filedir: str = ''):    
    
    async with httpx.AsyncClient(
        timeout=httpx.Timeout(write=None, read=None, 
            connect=None, pool=None)) as client:

        r = await client.post(
            url, 
            params={
                'filename': filepath.name,
                'filedir': filedir
            },
            files={
                'file': (filepath.name, filepath.open('rb'), 'application/octet-stream')
            },
            headers = {
                'Authorization': f'Bearer {token}'
            }
        )

3. Watch system processes

top

4. Make post

import os

url = os.getenv('FILE_SERVER_URL')
tkn = token(f'{url}/auth/jwt/login', 
    os.getenv('FILE_SERVER_UNAME'), 
    os.getenv('FILE_SERVER_PASS'))

asyncio.run(
    async_post_file_req(
        f'{url}/stream', tkn,
        Path('/home/mleader/10G.txt')
    )
)

[tomchristie]

That's surprising, since we do use incremental upload for multipart files...

httpx/httpx/_multipart.py

Line 136 in 776dbb5

for chunk in self.file:

Probably worth looking into some more.
First up: is anyone able to reproduce the behaviour described here?