HTTP message body lost when uploading a file and a redirection occurs

[nicolasferreiralozano]

When the SDK makes an HTTP request to any of the upload URLs and the HTTP response is 307 Temporary Redirect, or possibly any other redirection, the SDK correctly follows the redirection but drops the HTTP message body, causing the server to respond with a client error.

A real-world scenario could be a proxy server sitting between the client and the server, needing to authenticate the upload URL, redirecting the client to perform authentication before allowing the client to proceed with the request. In this case, the SDK would POST the usual multipart/form-data to https://upload.box.com/api/2.0/files/content, receive a 307 Temporary Redirect, and POST to Location without the HTTP message body, which is incorrect.

A few observations:

• A quick glance at the SDK's source code suggests that the issue is that the stream used for uploads by the SDK is not rewindable.
• Requests works correctly. Requests correctly follows the redirection and maintains the HTTP message body. Requests had this bug in 2016, but it was fixed: requests gets stuck on post redirect where data is readable psf/requests#3079
• The auto-generated Python SDK has the same problem.

[arjankowski]

Hi @nicolasferreiralozano, thank you for reporting this issue.
This is the first time we are encountering such a problem, so to understand it better, we need more information.

When did you start noticing this issue and under what circumstances?
You mentioned that a real-world scenario could be a proxy server, but is that your case?

Could you send us a sample code that causes this issue and describe network configuration, meaning whether the traffic is being redirected through a proxy?

Regards,
Artur

[nicolasferreiralozano]

I would say this is a latent bug. I noticed it while I was investigating a customer-reported issue. They are using a well-known cloud proxy. So yes, this is my case.
I isolated the problem, found that it lies in the SDK and not Requests, which is what the SDK uses under the hood, and used Fiddler to replicate it.
I also found that Requests had this problem, and I agree with Cory Benfield that "this is a very real problem", so please look at psf/requests#3079, which I think has all the details to put the pieces together.
My suggestion would be to think abstractly and less in terms of scenarios. Maybe it helps if we agree that Box will never redirect from and endpoint that expects multipart/form-data, and that this is exclusive to man-in-the-middle applications. The reason being that man-in-the-middle applications can do anything, and as long as they comply with the spec, the SDK should handle the behaviors.
That said, here are some instructions:
Create a text file with a .dat file extension and the following content:

HTTP/1.1 307 Temporary Redirect
Location: https://www.google.com/56f2078f-49fe-40b2-a2ae-d4432a07126b
Content-Length: 0

Make sure there are two newline at the end of the file.

Using Fiddler Classic, create the following AutoResponder rule:
EXACT:https://upload.box.com/api/2.0/files/content
And point to the .dat file.

Here is some code:

import json
import uuid

import requests
from boxsdk import Client, OAuth2

CLIENT_ID = "…"
CLIENT_SECRET = "…"


def upload_sdk(access_token, file_path, file_name, folder_id):
    auth = OAuth2(
        client_id=CLIENT_ID, client_secret=CLIENT_SECRET, access_token=access_token
    )
    client = Client(auth)

    with open(file_path, "rb") as stream:
        new_file = client.folder(folder_id).upload_stream(stream, file_name=file_name)
        return new_file


def upload_requests(access_token, file_path, file_name, folder_id):
    attributes = {
        "name": file_name,
        "parent": {"id": folder_id},
        "description": None,
        "content_created_at": None,
        "content_modified_at": None,
    }
    url = "https://upload.box.com/api/2.0/files/content"
    headers = {"Authorization": f"Bearer {access_token}"}

    with open(file_path, "rb") as stream:
        payload = {
            "attributes": (None, json.dumps(attributes), "application/json"),
            "file": stream,
        }

        r = requests.post(url, files=payload, headers=headers)
        return r.text


if __name__ == "__main__":
    access_token = "…"
    file_path = "…"
    folder_id = "0"

    file_name = str(uuid.uuid4())
    new_file = upload_requests(access_token, file_path, file_name, folder_id)
    print(new_file)

    file_name = str(uuid.uuid4())
    new_file = upload_sdk(access_token, file_path, file_name, folder_id)
    print(new_file)

Run the code with the rule enabled and you will see that Requests returns immediately, whereas the SDK hangs until the server times out. This is because the SDK is replying the original request, including a > 0 Content-Length, but is dropping the body.

The issue seems to be that the "stream" is not rewindable:

box-python-sdk/boxsdk/session/session.py

Line 481 in e5c60b2

multipart_stream = MultipartStream(data, files)

[mwwoda]

Thanks for the sample. I was able to reproduce it with mitm-proxy by simulating 307 redirects. Let me discuss it internally, as I see that it affects not only our Python sdks (e.g. in .NET Gen exception is thrown), so we should agree how these kinds of redirects should be handled when stream is involved

[lukaszsocha2]

Hi @nicolasferreiralozano ,
requests library documentation states that:

In the event you are posting a very large file as a multipart/form-data request, you may want to stream the request. By default, requests does not support this, but there is a separate package which does - requests-toolbelt. (link):

when using Requests’ Multipart upload functionality all the data must be read into memory before being sent to the server. (link):

In the Box Python SDK, we use requests-toolbelt by default to avoid loading the entire file into memory before sending the request. This ensures better efficiency, especially for large files.

However, 307 Temporary Redirects introduce a challenge when using streamed file uploads. Unlike 302 Found, which can change a POST request into a GET, a 307 redirect requires that the method and request body remain unchanged. This becomes an issue when the request body is a file stream, as the stream may already be exhausted when the redirect occurs. Since requests automatically follows redirects, the request is resent, but the file content may no longer be available.

To address this issue we have introduced stream_file_content parameter to upload methods. This allows you to choose between streaming the file (for memory efficiency) or loading it into memory first (to handle redirects correctly).
Let me know of this solves your issue,
Best,
@lukaszsocha2