Seeking help on how to dump flows and start mitmproxy all from a script

[securisec]

Steps to reproduce the problem:

Not an issue. I am failing to understand how i can start the proxy from a script, and then write any flows on a file. Here is my script so far. Script is a put together form various script ideas by googling around. didnt want to create an issue, but couldnt really get in help on twitter/slack/irc etc. I understand this is simper to do using subprocess, but i am not trying to use subprocess

import os
import sys

from mitmproxy import http
# from mitmproxy.master import Master
import mitmproxy.master as master
from mitmproxy.addons import core
from mitmproxy.proxy import ProxyConfig, ProxyServer
from mitmproxy.options import Options
from mitmproxy import io, http


class Writer(master.Master, http.HTTPFlow):
    def __init__(self, options, server, file_path):
        master.Master.__init__(self, options)
        self.server = server
        self.file_path = file_path
        self.f = open(self.file_path, 'wb+')
        self.w = io.FlowWriter(self.f)

    def run(self):
        try:
            master.Master.run(self)
        except KeyboardInterrupt:
            self.shutdown()
            sys.exit(0)
    
    def writestuff(self):
        flo = http.HTTPFlow
        print(flo)
        self.w.add(flo)


def start_proxy(port, file_path):
    options = Options(
        listen_port=port,
    )
    config = ProxyConfig(
        options=options,
    )
    server = ProxyServer(config)
    # print(dir(http.HTTPFlow))
    m = Writer(options, server, file_path)
    m.addons.add(core.Core())
    m.run()
    m.writestuff()

start_proxy(8080, '/tmp/dumptest')

System information

Mitmproxy: 4.0.4
Python: 3.7.0
OpenSSL: OpenSSL 1.1.0h 27 Mar 2018
Platform: Darwin-16.7.0-x86_64-i386-64bit

[securisec]

Trying a different approach. New test script following issues here is

from mitmproxy import proxy, options
from mitmproxy.tools.dump import DumpMaster
from mitmproxy.addons import core


class AddHeader:
    def __init__(self):
        self.num = 0

    def response(self, flow):
        self.num = self.num + 1
        print(self.num)
        flow.response.headers["count"] = str(self.num)


addons = [
    AddHeader()
]

opts = options.Options(listen_host='127.0.0.1', listen_port=8080)
pconf = proxy.config.ProxyConfig(opts)

m = DumpMaster(None)
m.server = proxy.server.ProxyServer(pconf)
print(m.addons)
m.addons.add(addons)
# m.addons.add(core.Core())

try:
    m.run()
except KeyboardInterrupt:
    m.shutdown()

This creates an error AttributeError: No such option: body_size_limit which seems to be mitigated with master.addons.add(core.Core) but this core addon already exists in DumpMaster so that fires a different error.

I also see that the the save add on is loaded in DumpMaster, but how can i work with it to specify a path to write on?

[YancyFrySr]

anyone had any luck with this?

[mhils]

You'd definitely need to do master.addons.add(core.Core()). Take a look at the initialization procedures in mitmproxy/tools. That being said, subclassing your own master is currently "no support" area.

[0xEmanuel]

This worked for me:
(remove mode='transparent', if you dont want to run mitmproxy in transparent mode)

from mitmproxy import proxy, options
from mitmproxy.tools.dump import DumpMaster

class AddHeader:
    def request(self, flow):
        if flow.request.pretty_host == "example.org":
            flow.request.host = "mitmproxy.org"

def start():
    myaddon = AddHeader()
    opts = options.Options(listen_host='0.0.0.0', listen_port=8080, mode='transparent', confdir='/home/user/.mitmproxy')
    pconf = proxy.config.ProxyConfig(opts)
    m = DumpMaster(opts)
    m.server = proxy.server.ProxyServer(pconf)
    m.addons.add(myaddon)

    try:
        m.run()
    except KeyboardInterrupt:
        m.shutdown()

start()

Another alternative without subprocess, but probably not what you want:

from mitmproxy.tools import _main
_main.mitmdump(["--mode", "transparent", "--set", "confdir=/home/user/.mitmproxy", "-s", "/home/user/myscript_containing_addons.py" ])

[securisec]

This is great @0xEmanuel or @mhils . How can i go about writing the flows from dumpmaster and write to a file similar to how i can pass the -w to mitmproxy itself? that is the main goal that i am trying to achieve.

[xuelliu]

options.Options(block_global=False, listen_host='0.0.0.0', listen_port=listen_port, mode = mode, output_path = output_path, scripts = scripts)

using this way, error raised:
'Unknown options: block_global, output_path, scripts'

how to add those option supports which are not in Options?

[xuelliu]

options.Options(block_global=False, listen_host='0.0.0.0', listen_port=listen_port, mode = mode, output_path = output_path, scripts = scripts)

using this way, error raised:
'Unknown options: block_global, output_path, scripts'

how to add those option supports which are not in Options?

Using this way, it works. Cool~~~

    from mitmproxy.addons import block

    pconf = proxy.config.ProxyConfig(opts)
    m = DumpMaster(opts)
    m.addons.add(block)
    m.options.set('block_global=false')
    m.server = proxy.server.ProxyServer(pconf)

[pankajthekush]

You are looking for this

https://gist.github.com/akhil-reni/61afdf945cbad04614caa35ddaecb92b