New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP][web]Static viewer converter for mitmweb #2510

Merged
merged 13 commits into from Aug 21, 2017

Conversation

Projects
None yet
2 participants
@MatthewShao
Contributor

MatthewShao commented Aug 10, 2017

This PR is trying to add a static viewer converter, which can convert the dumped flows file into a group of static file, so that we could view the flows without backend server support.

This is about the first step, copying the resource files to a destination path. Also modifying index.html a bit, so we can have the interface by opening this html file.

If I understand the thing correctly, the next step is about converting the flows file into a bunch of json files in certain structure, so that our frontend could fetch the flow information from those local files.

However, there are two troubles here as far as I can see:

  1. The fetch api we used in StaticBackend does not support the file:/// protocol, ref: github/fetch#92, I tried XMLHttpRequest too, but it only works in Firefox, Chrome prevents us to fetch from local file. We need to change the design of StaticBackend, maybe importing the json files directly from our js code.
  2. Since we will mimic the API calls in a folder structure of local files, updatesStoreFromUrl and updateUrlFromStore can not work as excepted, we need to find some workarounds for them.

MatthewShao added some commits Aug 10, 2017

[web] Import static.js in index.html
This is necessary after we introducing the static mode, or it will raise
an "undefined" exception.
@mhils

This comment has been minimized.

Show comment
Hide comment
@mhils

mhils Aug 10, 2017

Member

The fetch api we used in StaticBackend does not support the file:/// protocol, ref:

Doesn't matter for the final use case. For testing we can just start a webserver (python3 -m http.server or devd .), ultimately those files would be uploaded to our testing server anyways.

Since we will mimic the API calls in a folder structure of local files, updatesStoreFromUrl and updateUrlFromStore can not work as excepted, we need to find some workarounds for them.

Good catch. AFAIK they currently work on the location hash though, so this should just work. 😃

Other than that, this is going in the right direction. Let's continue with the flows! :)

Member

mhils commented Aug 10, 2017

The fetch api we used in StaticBackend does not support the file:/// protocol, ref:

Doesn't matter for the final use case. For testing we can just start a webserver (python3 -m http.server or devd .), ultimately those files would be uploaded to our testing server anyways.

Since we will mimic the API calls in a folder structure of local files, updatesStoreFromUrl and updateUrlFromStore can not work as excepted, we need to find some workarounds for them.

Good catch. AFAIK they currently work on the location hash though, so this should just work. 😃

Other than that, this is going in the right direction. Let's continue with the flows! :)

MatthewShao added some commits Aug 12, 2017

[web] Update static_viewer addon.
When the addon is triggered, we first copy the static resource files to
the output dir, then load the `flows` file and dump json files.
Show outdated Hide outdated web/src/js/flow/utils.js
Show outdated Hide outdated web/src/js/utils.js
@MatthewShao

This comment has been minimized.

Show comment
Hide comment
@MatthewShao

MatthewShao Aug 16, 2017

Contributor

If there is anything that I'm not satisfied with this implementation yet, that should be the speed. We are doing a lot of IO operations here, basically creating a mass of static files, so it is kind of slow... It takes over 30s to process a flows file which is only 2MB on my computer. How could we improve this?
I will write the tests for this addon once we nail down the design of it.

Contributor

MatthewShao commented Aug 16, 2017

If there is anything that I'm not satisfied with this implementation yet, that should be the speed. We are doing a lot of IO operations here, basically creating a mass of static files, so it is kind of slow... It takes over 30s to process a flows file which is only 2MB on my computer. How could we improve this?
I will write the tests for this addon once we nail down the design of it.

@MatthewShao

Looks so great! Thanks! 🍰

Show outdated Hide outdated mitmproxy/addons/static_viewer.py
def save_flows_content(path: pathlib.Path, flows: typing.Iterable[flow.Flow]) -> None:
for f in flows:

This comment has been minimized.

@MatthewShao

MatthewShao Aug 17, 2017

Contributor

The variable f has a collision with the filetype var on line 51, let's make it flow here.

@MatthewShao

MatthewShao Aug 17, 2017

Contributor

The variable f has a collision with the filetype var on line 51, let's make it flow here.

@mhils

mhils approved these changes Aug 21, 2017

🎉 🎉 🎉

@mhils mhils merged commit 7fcc945 into mitmproxy:master Aug 21, 2017

4 checks passed

codecov/patch 100% of diff hit (target 88.54%)
Details
codecov/project 88.6% (+0.05%) compared to 9d37597
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment