[Bug]: Malformed attachment object if adding under the same key when using the Async client

[yassiezar]

Version

1.6.0

Operating System

Linux

Python Version

3.12

What happened?

I'm running into an error when adding multiple file attachments under the same key when using the Async client, e.g.

client = AsyncClient(
        auth=..., api_url="https://api.eu.mailgun.net"
    )
data = {
        "from": "source@mydomain.com",
        "to": "dest@mydomain.com",
        "subject": "Error",
        "text": "buggy test",
    }

attachments = [
    (
        "inline",
        ("test.txt", b"Hello, this is a test file.")
    ),
    (
        "inline",
        ("test2.txt", b"Hello, this is also a test file.")
    )
]

req = await client.messages.create(data=data, files=attachments, domain="mydomain.com")

This behaviour is supported in the requests library (see here) and the above seems to be valid. However, it looks like the mailgun SDK is transforming the input into an unsupported dict format that breaks when passed into the requests API.

E.g. the above is transformed into the dict below

{
    'inline': [
        ('test.txt', <_io.BytesIO object at 0x7fba431c7510>, 'application/octet-stream'), 
        ('test2.txt', <_io.BytesIO object at 0x7fba431c75b0>, 'application/octet-stream')
    ]
}

and requests isn't able to process the dict.

Weirdly, running the above with the sync client yields no errors. It might be because the async client is calling the _prepare_files() function first, which looks to be doing the malforming

Steps to reproduce

1. Setup test env (mailgun v1.6.0)
2. Initialise client
3. Setup data and attachments as above
4. Execute request with the async client

Relevant log output

File ".../venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 394, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../venv/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
    raise exc from None
  File ".../venv/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
    response = await connection.handle_async_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../venv/lib/python3.12/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 136, in handle_async_request
    raise exc
  File ".../venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 88, in handle_async_request
    await self._send_request_body(**kwargs)
  File ".../fc-api/venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 157, in _send_request_body
    async for chunk in request.stream:
  File ".../venv/lib/python3.12/site-packages/httpx/_multipart.py", line 299, in __aiter__
    for chunk in self.iter_chunks():
  File ".../venv/lib/python3.12/site-packages/httpx/_multipart.py", line 261, in iter_chunks
    yield from field.render()
  File ".../venv/lib/python3.12/site-packages/httpx/_multipart.py", line 221, in render
    yield from self.render_data()
  File ".../venv/lib/python3.12/site-packages/httpx/_multipart.py", line 214, in render_data
    chunk = self.file.read(self.CHUNK_SIZE)
            ^^^^^^^^^^^^^^
AttributeError: 'list' object has no attribute 'read'

Additional information

No response

[skupriienko]

Hi @yassiezar, I can replicate your error.

You can use unique keys instead:

    attachments = [
        (
            "inline0", # Unique key 1
            ("test.txt", Path("test.txt").read_bytes())
        ),
        (
            "inline1", # Unique key 2
            ("test2.txt", Path("test2.txt").read_bytes())
        )
    ]

or:

    attachments = [
        (
            "inline[0]", # Unique key 1
            ("test.txt", b"Hello, this is a test file.")
        ),
        (
            "inline[1]", # Unique key 2
            ("test2.txt", b"Hello, this is also a test file.")
        )
    ]

This example will attach files to the email and send it. I've just checked that it works. Does this solve your issue? Then we'll update our docs. If not, then we should handle it properly in _prepare_files(). @erz9engel, can you look at it?

If you send:

    attachments = [
        ("test.txt", b"Hello, this is a test file."),
        ("test2.txt", b"Hello, this is also a test file."),
    ]

Then the email will be sent, but attachments won't be included.

[yassiezar]

Hi @skupriienko, using unique keys does work. I also managed to bypass the issue by passing in the attachment as a tuple of tuples, instead of a list of tuples, e.g.,

attachments = (
    (
        "inline",
        ("test.txt", b"Hello, this is a test file.")
    ),
    (
        "inline",
        ("test2.txt", b"Hello, this is also a test file.")
    )
)

This makes it bypass the dict and list checks and passes the attachment object along to requests verbatim. I found the latter to be a nicer solution for our particular use-case with CID tags, but I suppose either can work.

Is there a technical reason for reconstructing the attachment object into a dictionary of lists? It's a bit surprising that there are different behaviours between the async and non-async clients. I'd have expected that passing in a requests/httpx-compatible attachment object would 'just work' without having to be modified

[skupriienko]

AsyncClient uses a different logic for handling the files than the non-async Client. Looks like we should revisit it.