Multipart uploading: explicit boundary ignored?

[los-floppos]

Hey,

I am trying to POST a request to upload a JSON file with multipart encoding.

This post by @dreinon helped me quite a bit in figuring out how to set the boundary properly. However, still, my request looks error-prone. What puzzles me most is that the explicit set boundary +++ seems to be ignored and a random string is used instead.

Here is my sample code:

import json
import httpx
import curlify2

data = {
  "integer": 123,
  "boolean": "True",
  "list": [
    "a",
    "b",
    "c"
  ]
}
json_string = json.dumps(data)
with open("foo.json", "w") as i :
   json.dump(json_string, i)


client = httpx.Client()
content_type_file = "application/json; charset=UTF-8"
content_type_request = "multipart/form-data; boundary=+++"
r = client.build_request(
    method="POST",
    url="https://example.com",
    files={"metadata": (None, open("foo.json", "rb"), content_type_file)},
    headers={"cacheControl": "3600", "content-type": content_type_request},
)

print(curlify2.to_curl(r))

response = client.send(r)
boundary = response.request.headers["Content-Type"].split("boundary=")[-1]
print(f"{boundary=}")

print(curlify2.to_curl(r)) returns:

curl -X POST -H "host: example.com" -H "accept: */*" -H "accept-encoding: gzip, deflate, br" -H "connection: keep-alive" -H "user-agent: python-httpx/0.22.0" -H "cachecontrol: 3600" -H "content-type: multipart/form-data; boundary=+++" -H "content-length: 250" -d '--72db69ac61f02f73f28cb2121e693b8f
Content-Disposition: form-data; name="metadata"
Content-Type: application/json; charset=UTF-8

"{\"integer\": 123, \"boolean\": \"True\", \"list\": [\"a\", \"b\", \"c\"]}"
--72db69ac61f02f73f28cb2121e693b8f--
' https://example.com

print(f"{boundary=}") returns:

boundary='+++'

Any suggestions are very much appreciated.

Thank you very much in advance 🙂

[los-floppos]

This workaround works for me:

#!/usr/bin/env python3

import json
import httpx
import curlify2

data = {
  "integer": 123,
  "boolean": "True",
  "list": [
    "a",
    "b",
    "c"
  ]
}
json_string = json.dumps(data)
with open("foo.json", "w") as i :
   json.dump(json_string, i)

client = httpx.Client()
boundary = b'+++'
content_type_request = "multipart/form-data; boundary=" + boundary.decode('utf-8')
multipart_header_json = b'application/json; charset=UTF-8'

# Constructing the binary message body
file = open("foo.json", "rb")
content = b'--' + boundary + b'\n' + \
          multipart_header_json + file.read() + b'\n' + \
          b'--' + boundary + b'--\n'

r = client.build_request(
    method="POST",
    url="https://example.com",
    content=content,
    headers={"cacheControl": "3600", "content-type": content_type_request},
)

print(curlify2.to_curl(r))

response = client.send(r)

[los-floppos]

However, this is still just a workaround IMHO.

[adriangb]

I think custom boundaries are not implemented, so this would be a feature request.

My main question is: should we add a boundary option to Client.post/put/request/etc or should we only allow setting it if the header is explicitly set or both?

I wrote a quick implementation for the latter, it's pretty straightforward, but I'm not sure it's what we want.

[tomchristie]

I think custom boundaries are not implemented, so this would be a feature request.

Correct.

My main question is: should we add a boundary option to Client.post/put/request/etc or should we only allow setting it if the header is explicitly set or both?

We've seen folks attempting to use this via setting the header value, so I think we ought to consider supporting that.

I think we'd want to see what the change footprint for it is like before making a yes/no call. I assume that we would want to support it unless it somehow ends up being more awkward than expected.

I'd suggest that any extra "boundary" API could be treated as a later follow-up question.

[adriangb]

I opened #2278