Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Content Missing when getting a large file(>1mb) #6724

Closed
cannonLCK opened this issue Mar 8, 2024 · 1 comment
Closed

Content Missing when getting a large file(>1mb) #6724

cannonLCK opened this issue Mar 8, 2024 · 1 comment
Labels
kind/triage Unclassified issues

Comments

@cannonLCK
Copy link

Problem Description

  • MitMProxy resets stream with an INTERNAL_ERROR, when the response length is bigger than about 1mb.
    (I haven't figured out where the exact length breaks, but it's somewhere around 1mb)

Steps to reproduce the behavior:

  • Settings: mitmdump -s ./test.py --set stream_large_bodies=2m --set ssl_insecure=true --set anticache=true
    stream_large_bodies differs 1m to 2m to reproduce an error.
    anticache, lazy option were not relevant
    test.py just captures partial content(206) and attempts to make a full pdf.
    2 files tested: one about 900kb, the other about 1.3mb
  1. GET <1m file: downloaded well with partial content
[02:55:05.695] [+] Content captured
[02:55:05.832] [+] Downloading [redacted filename] 65536/845812(+65536)
[::1]:55498: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[02:55:05.950] [+] Content captured
[02:55:05.951] [+] Downloading [redacted filename] 124916/845812(+59380)
[::1]:55498: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 58.0k
[02:55:06.100] [+] Content captured
[02:55:06.101] [+] Downloading [redacted filename] 190452/845812(+65536)
[::1]:55498: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[02:55:06.252] [+] Content captured
[02:55:06.252] [+] Downloading [redacted filename] 255988/845812(+65536)
[::1]:55498: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[02:55:06.402] [+] Content captured
[02:55:06.403] [+] Downloading [redacted filename] 321524/845812(+65536)
[::1]:55498: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[02:55:06.526] [+] Content captured
[02:55:06.526] [+] Downloading [redacted filename] 387060/845812(+65536)
[::1]:55498: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[02:55:06.642] [+] Content captured
[02:55:06.643] [+] Downloading [redacted filename] 452596/845812(+65536)
[::1]:55498: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[02:55:06.791] [+] Content captured
[02:55:06.791] [+] Downloading [redacted filename] 518132/845812(+65536)
[::1]:55498: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[02:55:06.925] [+] Content captured
[02:55:06.925] [+] Downloading [redacted filename] 583668/845812(+65536)
[::1]:55498: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[02:55:07.081] [+] Content captured
[02:55:07.082] [+] Downloading [redacted filename] 649204/845812(+65536)
[::1]:55498: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[02:55:07.191] [+] Content captured
[02:55:07.192] [+] Downloading [redacted filename] 714740/845812(+65536)
[::1]:55498: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[02:55:07.297] [+] Content captured
[02:55:07.297] [+] Downloading [redacted filename] 780276/845812(+65536)
[::1]:55498: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[02:55:07.470] [+] Content captured
[02:55:07.473] [+] Download completed: [redacted filename] 1
[::1]:55498: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
  1. GET >1m file, stream buffer cut < 1m: getting file is cut, all partial contents but last chunk were downloaded.
[03:02:47.844][[::1]:60412] Streaming response from [redacted url].
[::1]:60412: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 200 OK (content missing)
 << stream reset by client (CANCEL)
[03:02:47.970] [+] Content captured
[03:02:48.073] [+] Downloading [redacted filename]:65536/1359373(+65536)
[::1]:60412: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[03:02:48.182] [+] Content captured
[03:02:48.182] [+] Downloading [redacted filename]:114189/1359373(+48653)
[::1]:60412: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 47.5k
[03:02:48.357] [+] Content captured
[03:02:48.358] [+] Downloading [redacted filename]:179725/1359373(+65536)
[::1]:60412: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[03:02:48.509] [+] Content captured
[03:02:48.509] [+] Downloading [redacted filename]:245261/1359373(+65536)
[::1]:60412: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[03:02:48.622] [+] Content captured
[03:02:48.623] [+] Downloading [redacted filename]:310797/1359373(+65536)
[::1]:60412: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[03:02:48.740] [+] Content captured
[03:02:48.740] [+] Downloading [redacted filename]:376333/1359373(+65536)
[::1]:60412: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[03:02:48.955] [+] Content captured
[03:02:48.955] [+] Downloading [redacted filename]:441869/1359373(+65536)
[::1]:60412: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[03:02:49.098] [+] Content captured
[03:02:49.098] [+] Downloading [redacted filename]:507405/1359373(+65536)
[::1]:60412: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[03:02:49.200] [+] Content captured
[03:02:49.200] [+] Downloading [redacted filename]:572941/1359373(+65536)
[::1]:60412: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[03:02:49.325] [+] Content captured
[03:02:49.325] [+] Downloading [redacted filename]:638477/1359373(+65536)
[::1]:60412: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[03:02:49.583] [+] Content captured
[03:02:49.583] [+] Downloading [redacted filename]:704013/1359373(+65536)
[::1]:60412: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[03:02:49.704] [+] Content captured
[03:02:49.705] [+] Downloading [redacted filename]:769549/1359373(+65536)
[::1]:60412: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[03:02:49.935] [+] Content captured
[03:02:49.936] [+] Downloading [redacted filename]:900621/1359373(+131072)
[::1]:60412: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 128k
[03:02:50.078] [+] Content captured
[03:02:50.079] [+] Downloading [redacted filename]:966157/1359373(+65536)
[::1]:60412: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[03:02:50.362] [+] Content captured
[03:02:50.363] [+] Downloading [redacted filename]:1031693/1359373(+65536)
[::1]:60412: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[03:02:50.624] [+] Content captured
[03:02:50.624] [+] Downloading [redacted filename]:1097229/1359373(+65536)
[::1]:60412: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[03:02:51.046] [+] Content captured
[03:02:51.047] [+] Downloading [redacted filename]:1162765/1359373(+65536)
[::1]:60412: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[03:02:51.543] [+] Content captured
[03:02:51.544] [+] Downloading [redacted filename]:1228301/1359373(+65536)
[::1]:60412: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
[03:02:51.699] [+] Content captured
[03:02:51.700] [+] Downloading [redacted filename]:1293837/1359373(+65536)
[::1]:60412: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 206 Partial Content 64.0k
  • Last chunk(256kb) was not transmitted.
  1. GET >1m file, stream buffer cut > 2m: getting file fails, no further partial contents were downloaded
[::1]:57502: GET https://[redacted url]/[redacted pdf].PDF HTTP/2.0
 << HTTP/2.0 200 OK (content missing)
 << stream reset by client (INTERNAL_ERROR)
  • No further partial contents were downloaded.

System Information

Mitmproxy: 10.2.4
Python:    3.12.2
OpenSSL:   OpenSSL 3.2.1 30 Jan 2024
Platform:  Linux-5.15.146.1-microsoft-standard-WSL2-x86_64-with-glibc2.37
  • Version of each Mitmproxy and Python may not be relevant.
    I've already suffered this about 1 years ago.
@cannonLCK cannonLCK added the kind/triage Unclassified issues label Mar 8, 2024
@mhils
Copy link
Member

mhils commented Mar 11, 2024

Can you provide a reproducer that does not rely on redacted URLs?

@mhils mhils closed this as not planned Won't fix, can't repro, duplicate, stale Mar 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/triage Unclassified issues
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mhils @cannonLCK