Auth fails for responses that include multiple WWW-Authenticate headers. #1234

beruhan · 2020-08-28T13:24:24Z

Checklist

httpx==0.11.1

Describe the bug

I use httpx request camera mjpeg,The camera response two WWW-Authenticate headers,I used the right username and password,but It response 401. When I user chrome, everything is ok, I doubt the httpx.DigestAuth may have bug when handle digest.
WireShark Data as follows:

request with httpx

GET /cgi-bin/guest/Video.cgi?media=MJPEG&channel=0 HTTP/1.1
host: 192.168.12.88
user-agent: python-httpx/0.11.1
accept: */*
accept-encoding: gzip, deflate, br
connection: keep-alive

HTTP/1.1 401 Unauthorized
Connection: close
Content-Type: text/html; charset=ISO-8859-1
WWW-Authenticate: Digest realm="server", nonce="4f3b1a0c93d39d1cab4d5d2a8928f85a3cc41bd5e8180e5c37587a9a8aea5097"
WWW-Authenticate: Basic realm="server"
Content-Length: 211

<HTML><HEAD><TITLE>401 Unauthorized</TITLE></HEAD>
<BODY><H1>401 Unauthorized</H1>
Your client does not have permission to get URL /cgi-bin/guest/Video.cgi?media=MJPEG&channel=0 from this server.
</BODY></HTML>

GET /cgi-bin/guest/Video.cgi?media=MJPEG&channel=0 HTTP/1.1
host: 192.168.12.88
user-agent: python-httpx/0.11.1
accept: */*
accept-encoding: gzip, deflate, br
connection: keep-alive
authorization: Digest username="admin", realm="server", nonce="4f3b1a0c93d39d1cab4d5d2a8928f85a3cc41bd5e8180e5c37587a9a8aea5097", uri="/cgi-bin/guest/Video.cgi?media=MJPEG&channel=0", response="43265b82a45736ef943ae501f0234807", algorithm=MD5

HTTP/1.1 401 Unauthorized
Connection: close
Content-Type: text/html; charset=ISO-8859-1
WWW-Authenticate: Digest realm="server", nonce="4f3b1a0ca12e0b92dc8ccca22e84edf7ce355d7825ac0e21eedf8f97d82ec588"
WWW-Authenticate: Basic realm="server"
Content-Length: 211

<HTML><HEAD><TITLE>401 Unauthorized</TITLE></HEAD>
<BODY><H1>401 Unauthorized</H1>
Your client does not have permission to get URL /cgi-bin/guest/Video.cgi?media=MJPEG&channel=0 from this server.
</BODY></HTML>

request with chrome

GET /cgi-bin/guest/Video.cgi?media=MJPEG&channel=0 HTTP/1.1
Host: 192.168.12.88
Connection: keep-alive
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.9,en;q=0.8
Cookie: pbAudioFalg=ON; RESOLUTION=720P; LifeTime=0; captcha=4PTT5z50/JVrA

HTTP/1.1 401 Unauthorized
Connection: close
Content-Type: text/html; charset=ISO-8859-1
WWW-Authenticate: Digest realm="server", nonce="4f3b1c6b2dccae706af263fdc5001a702ae3512d3cc4e42ebf823c60431f64b6"
WWW-Authenticate: Basic realm="server"
Content-Length: 211

<HTML><HEAD><TITLE>401 Unauthorized</TITLE></HEAD>
<BODY><H1>401 Unauthorized</H1>
Your client does not have permission to get URL /cgi-bin/guest/Video.cgi?media=MJPEG&channel=0 from this server.
</BODY></HTML>

GET /cgi-bin/guest/Video.cgi?media=MJPEG&channel=0 HTTP/1.1
Host: 192.168.12.88
Connection: keep-alive
Cache-Control: max-age=0
Authorization: Digest username="admin", realm="server", nonce="4f3b1c6b2dccae706af263fdc5001a702ae3512d3cc4e42ebf823c60431f64b6", uri="/cgi-bin/guest/Video.cgi?media=MJPEG&channel=0", response="7411b325f99b1ef61db78f6a1ba3ff9d"
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.9,en;q=0.8
Cookie: pbAudioFalg=ON; RESOLUTION=720P; LifeTime=0; captcha=4PTT5z50/JVrA

HTTP/1.1 200 OK
Expires: 0
Pragma: no-cache
Cache-Control: no-cache
Content-Type: multipart/x-mixed-replace;boundary="boundary"

--boundary
Content-Type: image/jpeg
Content-Length: 59048

xxxxx

The text was updated successfully, but these errors were encountered:

florimondmanca · 2020-08-28T14:54:59Z

Hello!

You're using a rather old-ish HTTPX version (0.11.1). What does this look like on the latest HTTPX release (0.14.2)?
Can you share a minimal example that reproduces the issue on your side?
Are there any details about the server/device you're requesting worth sharing? (Make and model, server documentation, etc.)

tomchristie · 2020-08-28T15:14:05Z

Perhaps a server-side implementation that's treating the Authentication header in a case sensitive way, and so is failing to authenticate correctly?

beruhan · 2020-08-29T02:09:08Z

I switch to the latest version httpx==0.14.2,It seems bug still exists, I found a camera instance on public network, And I wrote a simple example to illustrate this problem， as follows, I used requests and httpx request the same url, the requests can authentication successfully,but httpx failed.


import httpx
import requests
from requests.auth import HTTPDigestAuth

url = "http://190.156.226.164:8083/cgi-bin/guest/Video.cgi?media=MJPEG&channel=0"


def test_requests():
    auth = HTTPDigestAuth("admin", "admin")
    req = requests.get(url, auth=auth, stream=True)
    print(req.status_code)
    req.close()


def test_httpx():
    auth = httpx.DigestAuth("admin", "admin")
    with httpx.stream("GET", url, auth=auth) as r:
        print(r.status_code)


if __name__ == '__main__':
    test_requests()
    test_httpx()

output:

200
401

beruhan · 2020-08-29T06:43:11Z

Perhaps a server-side implementation that's treating the Authentication header in a case sensitive way, and so is failing to authenticate correctly?

It might be like you said， Is there any way to modify it？

florimondmanca · 2020-08-29T07:41:28Z

@beruhan To confirm that case sensitivity is causing this issue, you can try adjusting the request exemple so that it sends authorization (lowercase) instead of Authorization. (I don't have a code snippet off the top of my head right now, but the Requests docs should have some examples of modifying headers before sending the request.)

If this is confirmed then this would be related to #538.

beruhan · 2020-08-31T02:29:30Z

like you said,I modified the requests code to lowercase Authorization header,It can also authenticated,

GET /cgi-bin/guest/Video.cgi?media=MJPEG&channel=0 HTTP/1.1
Host: 192.168.12.88
User-Agent: python-requests/2.24.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
authorization: Digest username="admin", realm="server", nonce="4f3e773156897b27b411c1e6e28dfc81ba65d826ceaad75908ef1194b749acd2", uri="/cgi-bin/guest/Video.cgi?media=MJPEG&channel=0", response="be2fc243ffba15e79854605b202a301a"

HTTP/1.1 200 OK
Expires: 0
Pragma: no-cache
Cache-Control: no-cache
Content-Type: multipart/x-mixed-replace;boundary="boundary"

I don't know way the problem,I debug the authenticate process in requests and httpx as follows:
requests authenticate process

www-authenticate header received======= Digest realm="server", nonce="4f3e79d568136064295acb9ac2a95ae3a635ba5807cfda9d74549b5a1d4a9947", Basic realm="server"

Authorization header======= Digest username="admin", realm="server", 
nonce="4f3e79d568136064295acb9ac2a95ae3a635ba5807cfda9d74549b5a1d4a9947", uri="/cgi-bin/guest/Video.cgi?media=MJPEG&channel=0", response="e1198d68b1e06ebe68109cb10dce09ea"

httpx authenticate process

www-authenticate header received======= Digest realm="server", nonce="4f3e79d526379f6c6b9c52bb4f74a864d3e4ee32f3ce72e59bc6b817cb804450", Basic realm="server"

Authorization header======= Digest username="admin", realm="server", nonce="4f3e79d526379f6c6b9c52bb4f74a864d3e4ee32f3ce72e59bc6b817cb804450", uri="/cgi-bin/guest/Video.cgi?media=MJPEG&channel=0", response="8e82811163fff43a2522d4f7a15b1f9f"

beruhan · 2020-08-31T02:44:30Z

I also modify the code to make camera response header as the same ,

www-authenticate header received======= Digest realm="server", nonce="4f3e79d568136064295acb9ac2a95ae3a635ba5807cfda9d74549b5a1d4a9947", Basic realm="server"
Authorization header======= Digest username="admin", realm="server", nonce="4f3e79d568136064295acb9ac2a95ae3a635ba5807cfda9d74549b5a1d4a9947", uri="/cgi-bin/guest/Video.cgi?media=MJPEG&channel=0", response="e1198d68b1e06ebe68109cb10dce09ea"
200
www-authenticate header received======= Digest realm="server", nonce="4f3e79d568136064295acb9ac2a95ae3a635ba5807cfda9d74549b5a1d4a9947", Basic realm="server"
Authorization header======= Digest username="admin", realm="server", nonce="4f3e79d568136064295acb9ac2a95ae3a635ba5807cfda9d74549b5a1d4a9947", uri="/cgi-bin/guest/Video.cgi?media=MJPEG&channel=0", response="1189f82b9463486970d32134d3e9f6d4", algorithm=MD5
401

tomchristie · 2020-08-31T08:08:55Z

Somehow I'd completely missed that this issue is specific to the case when the server sends two WWW-Authenticate headers in the response.

And yup, can see how that might pick out a bug on our side, and yup can see how we ought to resolve this.

Leaving a link to a useful part of Mozilla's multiple header handling... https://github.com/bnoordhuis/mozilla-central/blob/master/netwerk/protocol/http/nsHttpHeaderArray.h#L185 for my own later reference in this.

j178 · 2020-09-02T09:33:55Z

@beruhan Hi, a fix hanlding mulitiple auth headers has been merged, can you try to install from the master branch and confirm if the issue resolved?

$ pip install git+https://github.com/encode/httpx.git@master#egg=httpx --upgrade

florimondmanca added the needs confirmation The issue described has not yet been confirmed, or replicated locally. label Aug 28, 2020

beruhan changed the title ~~there may be a bug in authentication when the server response to www-auth headers~~ there may be a bug in authentication when the server response two www-auth headers Aug 29, 2020

florimondmanca added bug Something isn't working external Root cause pending resolution in an external dependency and removed needs confirmation The issue described has not yet been confirmed, or replicated locally. labels Aug 29, 2020

florimondmanca added needs confirmation The issue described has not yet been confirmed, or replicated locally. and removed bug Something isn't working external Root cause pending resolution in an external dependency labels Aug 29, 2020

tomchristie changed the title ~~there may be a bug in authentication when the server response two www-auth headers~~ Auth fails for responses that include multiple WWW-Authenticate headers. Aug 31, 2020

tomchristie added bug Something isn't working and removed needs confirmation The issue described has not yet been confirmed, or replicated locally. labels Aug 31, 2020

tomchristie mentioned this issue Aug 31, 2020

Handle multiple auth headers correctly #1240

Merged

tomchristie closed this as completed in #1240 Sep 1, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Auth fails for responses that include multiple WWW-Authenticate headers. #1234

Auth fails for responses that include multiple WWW-Authenticate headers. #1234

beruhan commented Aug 28, 2020

florimondmanca commented Aug 28, 2020 •

edited

tomchristie commented Aug 28, 2020

beruhan commented Aug 29, 2020

beruhan commented Aug 29, 2020

florimondmanca commented Aug 29, 2020 •

edited

beruhan commented Aug 31, 2020 •

edited

beruhan commented Aug 31, 2020

tomchristie commented Aug 31, 2020

j178 commented Sep 2, 2020

Auth fails for responses that include multiple WWW-Authenticate headers. #1234

Auth fails for responses that include multiple WWW-Authenticate headers. #1234

Comments

beruhan commented Aug 28, 2020

Checklist

Describe the bug

florimondmanca commented Aug 28, 2020 • edited

tomchristie commented Aug 28, 2020

beruhan commented Aug 29, 2020

beruhan commented Aug 29, 2020

florimondmanca commented Aug 29, 2020 • edited

beruhan commented Aug 31, 2020 • edited

beruhan commented Aug 31, 2020

tomchristie commented Aug 31, 2020

j178 commented Sep 2, 2020

florimondmanca commented Aug 28, 2020 •

edited

florimondmanca commented Aug 29, 2020 •

edited

beruhan commented Aug 31, 2020 •

edited