New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Session store #2175

Open
ujjwal96 opened this Issue Mar 17, 2017 · 1 comment

Comments

Projects
None yet
2 participants
@ujjwal96
Member

ujjwal96 commented Mar 17, 2017

Having sessions in mitmproxy so that one can resume the work from where they left.
As told by @cortesi on mail

Things session store would do:

  • stores current live configuration so session can resume easily
  • has an immutable on-disk log of all flows in a session
  • provides space for addons to add annotations and store secondary data about flows. All analysis results should go into the session store

Once this is in place we can work on implementing higher-level functionality like security scanners, reconnaissance tools and end-point discovery mechanisms.
@mhils

@cortesi

This comment has been minimized.

Member

cortesi commented Mar 18, 2017

Let's flesh out this proposal somewhat. A rough outline of what I have in mind here:

Outline

A session is a mutable data store that can be accessed by addons and by the mitmproxy core.
Every mitmproxy process has an attached session. If the user doesn't explicitly specify a session path, one is created in a tmpdir and cleaned up on exit.

Annotations

Addons can add add and read annotations to flows using the flow unique identifier. So a security analysis addon can add notes to a flow that might then be shown to uses, or indeed used by other addons. I think we can be relaxed about the structure of these annotations to begin with, and firm things up by convention alter on.

This is also the right place to add user annotations, which we've wanted in our interface for a long time.

Options

The session should contain a store of the current invocation options, so the user can easily start up where they left off.

Flow record

The session contains an on-disk flow dump that is a complete immutable record of all flows that occurred during the session. This has a number of effects:

  • The backup/restore mechanism we currently have in the flow dump format should be moved to the session store. We should think about this more clearly than we have - if we have an immutable flow store, then a backup could just be the unique identifier of a previous flow, rather than a complete duplication of data. The same goes for replay, which can carry a reference to a previous flow.
  • Flow marking should be moved to the session store.
  • We can solve the long-standing issue we've had where mitmproxy and mitmweb memory use grows infinitely. We have an immutable flow store on disk in the session, so we can build a secondary in-memory index that takes much less space. To do this, we'd extend our flow writing mechanism to give us a file offset whenever a flow is written, which we can use for random access into the flow dump file.

Implementation

Each session is a directory containing a mitmdump and a storage file. The storage file is a SQLite database (or similar) that contains all the auxiliary data. The interface presented to addons and the mitmproxy core should be light-weight and storage agnostic.

@cortesi cortesi changed the title from [Feature] Have a session store to Session store Mar 18, 2017

@cortesi cortesi added the RFC label Mar 18, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment