HAR server replay missing content length in HEAD response #6547
Comments
|
cc @stanleygvi — no pressure I just saw that you contributed this feature recently <3 |
|
The following patch resolves this issue for me diff --git a/mitmproxy/http.py b/mitmproxy/http.py
index 986e6b898..d53f345b8 100644
--- a/mitmproxy/http.py
+++ b/mitmproxy/http.py
@@ -378,7 +378,7 @@ class Message(serializable.Serializable):
# don't set content-length if a transfer-encoding is provided
pass
else:
- self.headers["content-length"] = str(len(self.raw_content))
+ self.headers.setdefault("content-length", str(len(self.raw_content)))
def get_content(self, strict: bool = True) -> bytes | None:
"""This behavior looks like it was intentionally added in #4827 but I'm confused by it's different with HAR vs the binary storage. |
|
Thanks for the detailed report! I think we should keep flow.request.content = flow.request.content.replace(b"a", b"bb")to work in user addons. Many years ago we had it so that Instead, we should fix HAR loading to not use Contributions welcome! |
mhils
added
kind/bug
help wanted
area/addons
and removed
kind/triage
|
Thanks for the pointers! I was thinking about this later and came to similar conclusions that we should not change the behavior of |
#### Description Closes #6547 Responses in flows constructed from HAR files were using the `Response.make` utility which resulted in the injection of `content-length` headers. When a `content-length` header existed already, this could cause failures during replay. #### Checklist - [x] I have updated tests where applicable. - [x] I have added an entry to the CHANGELOG. --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Problem Description
When I use a HAR file for server response replay, my application fails with an error about missing content in the expected zip file response. It appears that the HEAD request's response does not properly include the
content-lengthso the subsequent GET request is not made by the client? Perhaps there is another issue with chunked responses happening here.With hardump:
When I don't use HAR, you can see it sets the content length and there are subsequent requests:
The content length appears to be correctly encoded in the HAR file (expand)
{ "startedDateTime": "2023-12-12T05:23:27.346896+00:00", "time": 16.900062561035156, "request": { "method": "HEAD", "url": "https://files.pythonhosted.org/packages/00/e5/f12a80907d0884e6dff9c16d0c0114d81b8cd07dc3ae54c5e962cc83037e/tqdm-4.66.1-py3-none-any.whl", "httpVersion": "HTTP/2.0", "cookies": [], "headers": [ { "name": "accept", "value": "*/*" }, { "name": "user-agent", "value": "puffin" }, { "name": "accept-encoding", "value": "gzip, br" } ], "queryString": [], "headersSize": 91, "bodySize": 0 }, "response": { "status": 200, "statusText": "", "httpVersion": "HTTP/2.0", "cookies": [], "headers": [ { "name": "last-modified", "value": "Thu, 10 Aug 2023 11:39:00 GMT" }, { "name": "etag", "value": "\"a296c6e224c118b0d08cd77e8c08f4b1\"" }, { "name": "x-amz-request-id", "value": "aeb4d3335548af85" }, { "name": "x-amz-id-2", "value": "aN65jxTFgNrlm8zEJMNdk7mYLYwUwTzh0" }, { "name": "x-amz-version-id", "value": "4_z179c51e67f11a0ad8f6c0018_f10789ff3151435c8_d20230810_m113900_c005_v0501001_t0045_u01691667540984" }, { "name": "content-type", "value": "application/octet-stream" }, { "name": "cache-control", "value": "max-age=365000000, immutable, public" }, { "name": "accept-ranges", "value": "bytes" }, { "name": "date", "value": "Tue, 12 Dec 2023 05:23:27 GMT" }, { "name": "age", "value": "2295687" }, { "name": "x-served-by", "value": "cache-iad-kcgs7200038-IAD, cache-stp9222-STP" }, { "name": "x-cache", "value": "HIT, HIT" }, { "name": "x-cache-hits", "value": "21458, 104163" }, { "name": "x-timer", "value": "S1702358607.425873,VS0,VE0" }, { "name": "strict-transport-security", "value": "max-age=31536000; includeSubDomains; preload" }, { "name": "x-frame-options", "value": "deny" }, { "name": "x-xss-protection", "value": "1; mode=block" }, { "name": "x-content-type-options", "value": "nosniff" }, { "name": "x-permitted-cross-domain-policies", "value": "none" }, { "name": "x-robots-header", "value": "noindex" }, { "name": "x-pypi-file-python-version", "value": "py3" }, { "name": "x-pypi-file-version", "value": "4.66.1" }, { "name": "x-pypi-file-package-type", "value": "bdist_wheel" }, { "name": "x-pypi-file-project", "value": "tqdm" }, { "name": "content-length", "value": "78258" } ], "content": { "size": 0, "compression": 0, "mimeType": "application/octet-stream", "text": "" }, "redirectURL": "", "headersSize": 1187, "bodySize": 0 }, "cache": {}, "timings": { "connect": 5.411863327026367, "ssl": 6.536245346069336, "send": 0.23412704467773438, "receive": 0.20503997802734375, "wait": 4.512786865234375 }, "serverIPAddress": "199.232.28.223" }, { "startedDateTime": "2023-12-12T05:23:27.356972+00:00", "time": 6.1588287353515625, "request": { "method": "GET", "url": "https://files.pythonhosted.org/packages/00/e5/f12a80907d0884e6dff9c16d0c0114d81b8cd07dc3ae54c5e962cc83037e/tqdm-4.66.1-py3-none-any.whl", "httpVersion": "HTTP/2.0", "cookies": [], "headers": [ { "name": "range", "value": "bytes=61874-78257" }, { "name": "accept", "value": "*/*" }, { "name": "user-agent", "value": "puffin" } ], "queryString": [], "headersSize": 90, "bodySize": 0 }, "response": { "status": 206, "statusText": "", "httpVersion": "HTTP/2.0", "cookies": [], "headers": [ { "name": "last-modified", "value": "Thu, 10 Aug 2023 11:39:00 GMT" }, { "name": "etag", "value": "\"a296c6e224c118b0d08cd77e8c08f4b1\"" }, { "name": "x-amz-request-id", "value": "aeb4d3335548af85" }, { "name": "x-amz-id-2", "value": "aN65jxTFgNrlm8zEJMNdk7mYLYwUwTzh0" }, { "name": "x-amz-version-id", "value": "4_z179c51e67f11a0ad8f6c0018_f10789ff3151435c8_d20230810_m113900_c005_v0501001_t0045_u01691667540984" }, { "name": "content-type", "value": "application/octet-stream" }, { "name": "cache-control", "value": "max-age=365000000, immutable, public" }, { "name": "accept-ranges", "value": "bytes" }, { "name": "date", "value": "Tue, 12 Dec 2023 05:23:27 GMT" }, { "name": "age", "value": "2295687" }, { "name": "x-served-by", "value": "cache-iad-kcgs7200038-IAD, cache-stp9222-STP" }, { "name": "x-cache", "value": "HIT, HIT" }, { "name": "x-cache-hits", "value": "21458, 104164" }, { "name": "x-timer", "value": "S1702358607.435964,VS0,VE0" }, { "name": "strict-transport-security", "value": "max-age=31536000; includeSubDomains; preload" }, { "name": "x-frame-options", "value": "deny" }, { "name": "x-xss-protection", "value": "1; mode=block" }, { "name": "x-content-type-options", "value": "nosniff" }, { "name": "x-robots-header", "value": "noindex" }, { "name": "access-control-allow-methods", "value": "GET, OPTIONS" }, { "name": "access-control-allow-headers", "value": "Range" }, { "name": "access-control-allow-origin", "value": "*" }, { "name": "x-pypi-file-python-version", "value": "py3" }, { "name": "x-pypi-file-version", "value": "4.66.1" }, { "name": "x-pypi-file-package-type", "value": "bdist_wheel" }, { "name": "x-pypi-file-project", "value": "tqdm" }, { "name": "content-range", "value": "bytes 61874-78257/78258" }, { "name": "content-length", "value": "16384" } ], "content": { "size": 16384, "compression": 0, "mimeType": "application/octet-stream", "text": "oJXk4Pn9+zNUNfvAMb1jD1cyGfag9xosQqBN3efa4rfTZT9G+M9Hy4QUs4/..." } } }Reproduction
I can work on a reproduction in the mitmproxy test suite as well as producing an example extracted from my application.
Roughly:
System Information
The text was updated successfully, but these errors were encountered: