GET request causes HTTP 400 error (OpenStack & potentially other providers)

[mspagon]

Summary

HTTP 400 errors are appearing in the logs of our OpenStack server: ERROR WSGI: code 400, Bad request syntax ('0').

They appear when sending a GET request.

Wireshark

Detailed Information

Libcloud version: v3.0.0
OS: Ubuntu 14.04.6 LTS
Provider: OpenStack Swift
Service: Storage

Reproducing

I'm not sure how to reproduce this without access to an Openstack server along with the server logs/wireshark to examine the HTTP error. But I would recommend just making a download request.

I traced down the bug to a single-line of code and in a subsequent post will isolate the reason and illustrate the behavior.

from libcloud.storage.providers import get_driver

import libcloud.security
libcloud.security.VERIFY_SSL_CERT = False

if __name__ == '__main__':
    cls = get_driver(Provider.OPENSTACK_SWIFT)

    driver = cls('<USER>', '<PASSWORD>',
                 ex_force_auth_url='http://10.110.5.30/auth/v1.0',
                 ex_force_auth_version='1.0',
                 ex_force_service_name='cloudFiles',
                 ex_force_service_type='object-store')

    container = driver.get_container('bucket01')

    # Get the object
    object = driver.get_object('bucket01', 'test-file.txt')

    # Download object as stream.
    for chunk in driver.download_object_as_stream(object):
        print(chunk)

[mspagon]

I traced the bug down to a single-line of code.

A GET request is sent with an empty body '' incorrectly as a chunked request and the 0 appears because it is supposed to signal the final chunk.

The Line of Code

The code is in libcloud/http.py and relates to how requests is being used to prepare requests and potentially affects every HTTP request libcloud makes.

A PreparedRequest is being constructed and then its body is being overwritten. The assignment seems redundant to me, as Requests does the job of translating it.

Line 244 of libcloud/http.py (LibcloudConnection class)

def prepared_request(self, method, url, body=None,
                     headers=None, raw=False, stream=False):
    headers = self._normalize_headers(headers=headers)

    req = requests.Request(method, ''.join([self.host, url]),
                           data=body, headers=headers)

    prepped = self.session.prepare_request(req)

    prepped.body = body  # <<<<<< This line.

    self.response = self.session.send(
        prepped,
        stream=stream,
        verify=self.ca_cert if self.ca_cert is not None else self.verify)

Why it's the Problem

When a request is prepared, if its body is an empty string '', the preparation step ignores the body and sets the resulting PreparedRequest body to None. Subsequently setting the body back to an empty string '' breaks requests logic when sending the request. Requests now thinks the request should be chunked because body='' breaks this logic:

Line 420 of requests/adapters.py

chunked = not (request.body is None or 'Content-Length' in request.headers)

A raw HTTP chunked request is sent with no body. As a result only a 0 is transmitted to signify the last chunk. This coincides with when our server receives the HTTP 400 error.

Line 469 of requests/adapters.py

for i in request.body:
    low_conn.send(hex(len(i))[2:].encode('utf-8'))
    low_conn.send(b'\r\n')
    low_conn.send(i)
    low_conn.send(b'\r\n')
low_conn.send(b'0\r\n\r\n')

Issue Appears when Default Value is ''

This problem originates from the default value for the data parameter within a provider's connection class request method. Some connection classes set the default value for data as None instead of '', and None does not trigger this issue.

#libcloud/common/openstack.py OpenStackBaseConnection
def request(self, action, params=None, data='', headers=None, ...)

#libcloud/common/base.py Connection
def request(self, action, params=None, data=None, headers=None, ...)

Replicating the Behavior

You can use this snippet to reproduce the same behavior outside the context of Libcloud.

import requests

session = requests.Session()

req = requests.Request('GET', 'http://www.google.com', data='')

# prepare_request() will ignore the empty string and set prepped.body to None.
prepped = session.prepare_request(req)

# Overwriting None with '' will break requests logic for sending requests.
prepped.body = ''

# Chunked request is sent with a zero length body. Only terminating 0 is sent. 
response = session.send(prepped, stream=False, verify=False)