Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stream large bodies warn with modify body #6514

Merged
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
717071a
stream_large_bodies_warn_with_modify_body: Add change log message
rosydawn6 Dec 1, 2023
54487eb
stream_large_bodies_warn_with_modify_body: Clarify how streaming inte…
rosydawn6 Dec 1, 2023
7753062
stream_large_bodies_warn_with_modify_body: Clarify how streaming inte…
rosydawn6 Dec 1, 2023
83cd967
stream_large_bodies_warn_with_modify_body: Add alert log with suitabl…
rosydawn6 Dec 1, 2023
209f63c
stream_large_bodies_warn_with_modify_body: Clarify modify_mody option…
rosydawn6 Dec 1, 2023
1d6bae0
stream_large_bodies_warn_with_modify_body: Add alert log with suitabl…
rosydawn6 Dec 1, 2023
a0ee48c
stream_large_bodies_warn_with_modify_body: Clarify stream_large_bodie…
rosydawn6 Dec 1, 2023
7a2004a
Revert "stream_large_bodies_warn_with_modify_body: Add alert log with…
rosydawn6 Dec 2, 2023
10bc309
Revert "stream_large_bodies_warn_with_modify_body: Clarify modify_mod…
rosydawn6 Dec 2, 2023
b3427a8
Update CHANGELOG.md
rosydawn6 Dec 2, 2023
6d755a1
[autofix.ci] apply automated fixes
autofix-ci[bot] Dec 2, 2023
0321c2a
Merge remote-tracking branch 'downstream_fork/stream_large_bodies_war…
rosydawn6 Dec 2, 2023
40cea6d
stream_large_bodies_warn_with_modify_body: Only emit alert if both mo…
rosydawn6 Dec 4, 2023
3bec0de
stream_large_bodies_warn_with_modify_body: Only emit alert if both mo…
rosydawn6 Dec 4, 2023
71c14c7
Merge remote-tracking branch 'origin/main' into pr-6514
mhils Dec 5, 2023
7b3c3ff
simplify
mhils Dec 5, 2023
a93e47c
[autofix.ci] apply automated fixes
autofix-ci[bot] Dec 5, 2023
ed9147d
nits
mhils Dec 5, 2023
83378c1
tests++
mhils Dec 5, 2023
365fc09
[autofix.ci] apply automated fixes
autofix-ci[bot] Dec 5, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGELOG.md
Expand Up @@ -11,7 +11,8 @@
([#5084](https://github.com/mitmproxy/mitmproxy/pull/5084), @Speedlulu)
* Scripts with relative paths are now loaded relative to the config file and not where the command is ran
([#4860](https://github.com/mitmproxy/mitmproxy/pull/4860), @Speedlulu)

* Enhance documentation and add alert log messages when stream_large_bodies and modify_body are set
([#6514](https://github.com/mitmproxy/mitmproxy/pull/6514), @rosydawn6)

## 14 November 2023: mitmproxy 10.1.5

Expand Down
8 changes: 5 additions & 3 deletions docs/src/content/overview-features.md
Expand Up @@ -205,7 +205,9 @@ if a modify hook is triggered on server response, the replacement is
only run on the Response object leaving the Request intact. You control
whether the hook triggers on the request, response or both using the
filter pattern. If you need finer-grained control than this, it's simple
to create a script using the replacement API on Flow components.
to create a script using the replacement API on Flow components. Body
modifications have no effect on streamed bodies. See
[Streaming]({{< relref "#streaming" >}}) for more detail.

#### Examples

Expand Down Expand Up @@ -359,8 +361,8 @@ indicated manipulations on it, and then send the message on to the other party.
This can be problematic when downloading or uploading large files. When
streaming is enabled, message bodies are not buffered on the proxy but instead
sent directly to the server/client. This currently means that the message body
will not be accessible within mitmproxy. HTTP headers are still fully buffered before
being sent.
will not be accessible within mitmproxy, and body modifications will have no
effect. HTTP headers are still fully buffered before being sent.

Request/response streaming is enabled by specifying a size cutoff in the
`stream_large_bodies` option.
Expand Down
13 changes: 13 additions & 0 deletions mitmproxy/addons/modifybody.py
Expand Up @@ -6,6 +6,7 @@
from mitmproxy import exceptions
from mitmproxy.addons.modifyheaders import ModifySpec
from mitmproxy.addons.modifyheaders import parse_modify_spec
from mitmproxy.log import ALERT


class ModifyBody:
Expand Down Expand Up @@ -37,6 +38,18 @@ def configure(self, updated):

self.replacements.append(spec)

stream_and_modify_conflict = (
ctx.options.modify_body
and ctx.options.stream_large_bodies
and ("modify_body" in updated or "stream_large_bodies" in updated)
)
if stream_and_modify_conflict:
logging.log(
ALERT,
"Both modify_body and stream_large_bodies are active. "
"Streamed bodies will not be modified.",
)

def request(self, flow):
if flow.response or flow.error or not flow.live:
return
Expand Down
5 changes: 3 additions & 2 deletions mitmproxy/addons/proxyserver.py
Expand Up @@ -150,8 +150,9 @@ def load(self, loader):
None,
"""
Stream data to the client if response body exceeds the given
threshold. If streamed, the body will not be stored in any way.
Understands k/m/g suffixes, i.e. 3m for 3 megabytes.
threshold. If streamed, the body will not be stored in any way,
and such respones cannot be modified. Understands k/m/g
suffixes, i.e. 3m for 3 megabytes.
""",
)
loader.add_option(
Expand Down