Network dumps docs

[jonbarrow]

Adds docs for creating and submitting network dumps to help with development.

Incomplete as of right now. Missing HTTP proxy docs, and Bandwidth has not been updated yet either.

Bandwidth has a PR open now PretendoNetwork/Bandwidth#9

@DaniElectra Can you write the mitmproxy docs, since you use it as your proxy?
@CaramelKat Can you write the Fiddler docs, since you use it as your proxy?
@InternalLoss Can you write the Charles docs, since you use it as your proxy?

The PR is on a branch in this repo, so you can just push to this branch with your changes

[DaniElectra]

Before wroking on the mitmproxy section I wanted to comment on some common parts of the docs

[MatthewL246]

In case you're interested, I created a (somewhat hacky) Dockerfile for running a mitmproxy Docker container with OpenSSL 1.1.1 in my mitmproxy-nintendo fork. I know that using the system-installed OpenSSL version is a common annoyance with mitmproxy. I've tested it, and it works with the Wii U and 3DS with Inkay/Nimbus enabled.

It's also published at https://ghcr.io/matthewl246/mitmproxy-pretendo. This version isn't currently suitable for network dumps because it redirects requests from Nintendo to Pretendo by default, but this could be changed easily if you're interested.

[jonbarrow]

Using the system OpenSSL is indeed the biggest annoyance I've had with using mitmproxy for these console and is the biggest reason I stick with Charles. I'm definitely interested in something like this. Would you be willing to also PR these changes back into the upstream repo?

@SuperMarioDaBom and I have been working towards Dockerizing all of our services anyway

[jonbarrow]

Bandwidth officially has a PR open to add his side of things PretendoNetwork/Bandwidth#9

[jonbarrow]

@hauntii I believe you also use Fiddler, correct? I know Jemma has a full time job outside of this. Would you mind contributing to the Fiddler docs in her absence?

Or maybe we should scrap all of that and only focus on mitmproxy, seeing as it's free and cross platform, and @MatthewL246 has been doing some upkeep on our old wrapper for it including that Dockerfile. Using that should make things pretty easy for new users, since it only really requires using Docker?

[MatthewL246]

Sure, I would be interested in making a PR for that. I was thinking that I could set up builds for multiple versions of the container using different tags - one that redirects Nintendo to Pretendo as originally intended, one for Wii U packet captures (doesn't modify any traffic and sends the Wii U client certificate), and one for 3DS packet captures (sends the CTR client certificate).

Update: Here's an example network dump generated by mitmdump -w flows.txt: flows.txt. It can easily be loaded by running mitmweb -n to start the web UI without starting the proxy server and opening the file.

[jonbarrow]

Sure, I would be interested in making a PR for that. I was thinking that I could set up builds for multiple versions of the container using different tags - one that redirects Nintendo to Pretendo as originally intended, one for Wii U packet captures (doesn't modify any traffic and sends the Wii U client certificate), and one for 3DS packet captures (sends the CTR client certificate).

This works for me. Though for less tech-savvy users, would it be possible to still provide a "combined" version that doesn't modify traffic? So that way users can capture traffic from both consoles at once?

I've had people who needed help even just installing software like Fiddler, which has an installation wizard. So anything we can do to help minimize friction is a plus in my book. Having things be separated would be useful for development though on our end

Update: Here's an example network dump generated by mitmdump -w flows.txt: flows.txt. It can easily be loaded by running mitmweb -n to start the web UI without starting the proxy server and opening the file.

Awesome! I gave your container a try just now and it seems to run well. The only thing I'm concerned about is exporting the traffic. I see you uploaded your flows as flows.txt, but the web UI (which is what I'll be having users use in these docs) exports as an extensionless flows file. This works fine for us, but wouldn't be able to be submitted through our Discord bot. Our Discord bot checks for network dumps by their extension, which would break here (unless we also check for a file just named flows? But then anyone who renames the file, which is something we suggest to do, wouldn't be able to submit them)

Does mitmproxy support exporting as a HAR in the web UI? I remember hearing that this was coming at some point a while ago but I'm unsure if this ever happened. Our Discord bot is already setup to detect HAR dumps so that would fix the issue

[DaniElectra]

You can export mitmproxy dumps to HAR by specifying the output file on the hardump option. More info: https://mitmproxy.org/posts/har-support/

[jonbarrow]

on the hardump option

I did see this, but this only seems to apply to when using the CLI. I don't see anywhere that mentions support for exporting as HAR in the web UI, which is what these docs will focus on as a GUI is likely easier for random people rather than dealing with a CLI

[MatthewL246]

Hardump can be enabled in the config file. I think it would apply to all 3 of the mitmproxy modes but I haven't tested it (I just learned about it now).

Edit: I tested it and it works for mitmproxy, mitmweb, and mitmdump.

[jonbarrow]

That sounds perfect then 👍

[DaniElectra]

The option is on the GUI too

[jonbarrow]

You're right, I missed that. I didn't see it mentioned in the changelog so I didn't think to look for it there. We can either enable it in the config automatically or have the docs tell users to set this. Either works for me, whatever you think would be easiest

[MatthewL246]

The HAR doesn't seem to include all of the information that the flows file includes (like the certificate info), but I guess it has what you need? I can absolutely enable hardump by default in the network dumping config.

Direct comparison of the formats: flows.txt and dump.har.txt (by the way I just renamed them to .txt because GitHub wouldn't let me upload them here otherwise).

[jonbarrow]

So long as it has the headers, paths, bodies, etc, that should be enough for our needs? @DaniElectra what say you

[MatthewL246]

Though for less tech-savvy users, would it be possible to still provide a "combined" version that doesn't modify traffic? So that way users can capture traffic from both consoles at once?

I'm not sure. Mitmproxy does provide a way to specify a different client certificate per domain, but this would require a comprehensive list of every Nintendo domain that requires a client certificate. It also would have the limitation that only one client certificate could be specified per domain. Is there any domain that both the Wii U and 3DS need to connect to and requires a different client certificate depending on the console? That would be a problem.

I think it would be easier to just tell people to run docker run mitmproxy-pretendo:wiiu or docker run mitmproxy-pretendo:3ds depending on the console.

[jonbarrow]

I think it would be easier to just tell people to run docker run mitmproxy-pretendo:wiiu or docker run mitmproxy-pretendo:3ds depending on the console.

That sounds reasonable to me given the circumstances. I'm not that familiar with mitmproxy so I wasn't sure how possible something like that was

[DaniElectra]

So long as it has the headers, paths, bodies, etc, that should be enough for our needs? @DaniElectra what say you

All the relevant data is present, so I think it should be enough

[DaniElectra]

Mitmproxy does provide a way to specify a different client certificate per domain, but this would require a comprehensive list of every Nintendo domain that requires a client certificate

We probably need this anyway for the Wii U as it has a different "common" and "account" client certificate, but I agree with keeping the consoles separate

[jonbarrow]

@DaniElectra I added the final bit of docs. I believe this should be everything? I did notice however in my Wii U testing that POST requests to the SOAP server did not save their request bodies. I am unsure if that is a bug in mitmproxy or a fluke on my end but it happened with all SOAP requests in my test capture. The Content-Length header is present and claims to have sent data, but none is saved. Since we don't really support the SOAP server anyway, and it's required for the eShop so I doubt it will go down in April, I think this is fine? Just something we need to keep an eye on in the future I suppose

EDIT: Looks like a bug with Charles again. Charles seems to have pretty lackluster HAR support

[DaniElectra]

did notice however in my Wii U testing that POST requests to the SOAP server did not save their request bodies

That's weird, I tested on the 3DS and the SOAP request bodies are being saved. Can you send me the dump on discord?

[jonbarrow]

I sent it in the dumps channel

[DaniElectra]

LGTM, only one thing

[jonbarrow]

@MatthewL246 @DaniElectra Aight, added WSL notes and updated the commands. Should be good?

[MatthewL246]

I think it should be good.

[DaniElectra]

Looks good