Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nested dict data get silently ignored in POST request #5058

Open
martimlobao opened this issue Apr 16, 2019 · 0 comments

Comments

Projects
None yet
1 participant
@martimlobao
Copy link

commented Apr 16, 2019

I just spent the last several hours trying to figure out why I was able to successfully send an API POST request through curl but not through requests.

After much searching on SO, I finally figured out the solution, but I'm still confused why this should be a problem at all. Furthermore, it's very strange that requests does such a good job of simplifying HTTP requests but then reveals intricacies that an end user likely won't know how to deal with.

It turns out that requests.post() already has a json argument which handles non-trivial data dictionaries, but I don't understand why a user should need to know when to use json instead of data. To a simple user like me (and others: #2885), data simply takes a dictionary and parses it into a request. Using json in some cases but not others adds confusion and is not obvious at all. I also don't understand why I wouldn't want to always use json instead of data.

At the very least, if there's a strong reason why it makes sense to have both data and json arguments, a warning should be raised in these cases telling the user that they might not be sending what they think they're sending.

Expected Result

Here's an example request of what I was sending through curl:

$ curl -X POST 'https://httpbin.org/post' -H 'Content-Type: application/json' -d '{"first": [{"second": {"third": "data"}}]}'

{
  "args": {}, 
  "data": "{\"first\": [{\"second\": {\"third\": \"data\"}}]}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "42", 
    "Content-Type": "application/json", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.54.0"
  }, 
  "json": {
    "first": [
      {
        "second": {
          "third": "data"
        }
      }
    ]
  }, 
  "origin": "123.56.7.890, 123.56.7.890", 
  "url": "https://httpbin.org/post"
}

Sending (what I thought was) an equivalent request through requests always returned an invalid json error:

import requests

headers = {"Content-Type": "application/json"}

data = {
    "first": [
        {
            "second": {
                "third": "data"
            }
        }
    ]
}

r = requests.post("https://httpbin.org/post", headers=headers, data=data)

print(r.json())

I expected the entire data dictionary to be sent in the request. However, when looking at the response from httpbin, it becomes clear that all the data from the second level is missing.

Actual Result

This is the response from httpbin, with almost all of the sent data missing:

{'args': {},
 'data': 'first=second',
 'files': {},
 'form': {},
 'headers': {'Accept': '*/*',
             'Accept-Encoding': 'gzip, deflate',
             'Content-Length': '12',
             'Content-Type': 'application/json',
             'Host': 'httpbin.org',
             'User-Agent': 'python-requests/2.19.1'},
 'json': None,
 'origin': '123.56.7.890, 123.56.7.890',
 'url': 'https://httpbin.org/post'}

Reproduction Steps

import requests

headers = {"Content-Type": "application/json"}

data = {
    "first": [
        {
            "second": {
                "third": "data"
            }
        }
    ]
}

r = requests.post("https://httpbin.org/post", headers=headers, data=data)

print(r.json())

Changing the request fixes the issue, but the solution is (to me) non-obvious:

r = requests.post("https://httpbin.org/post", headers=headers, json=data)

System Information

$ python -m requests.help
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": ""
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.6"
  },
  "platform": {
    "release": "18.5.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.19.1"
  },
  "system_ssl": {
    "version": "1000212f"
  },
  "urllib3": {
    "version": "1.23"
  },
  "using_pyopenssl": false
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.