Network monitoring and HAR export

[motin]

Work in progress - do not merge yet.

Fixes #3

To try this out, following the clone, install and start instructions on https://github.com/motin/dataleak-pioneer-shield-study/blob/develop/docs/DEV.md#details

After Firefox has started, open a new tab (only this tab will be instrumented atm), navigate to an URL and inspect the Browser Console for what is going on.

[motin]

HTTP activity does not seem to be recorded anywhere. The HAR returned from NetMonitor is always null. What is missing here?

[ochameau]

It will only start recording from here, when calling this.getNetMonitorAPI().connect.
So you will have to reload the current page or load a new one in the tab in order to get some data out of NetMonitorAPI.getHar() (and wait for the end of load).

[motin]

It doesn't seem to work currently. I can reload the page twenty times, write new URLs in the address bar of the monitored tab etc after this statement - without anything but empty HAR contents being returned by NetMonitor afterwards.

[ochameau]

Hum. I don't know then.

It would be helpful to trace what is happening here:
https://searchfox.org/mozilla-central/source/devtools/client/netmonitor/src/api.js#107
options.items should be a non empty array.

Then ensure that this code is trigerred:
https://searchfox.org/mozilla-central/source/devtools/client/netmonitor/src/connector/firefox-connector.js#71
Without the call to addListeners here, nothing will be recorded.

Finally, you could also ensure the redux actions are correctly dispatched for each request, from here:
https://searchfox.org/mozilla-central/source/devtools/client/netmonitor/src/connector/firefox-data-provider.js#76-77
Otherwise, the store won't be populated.

If all that is correct, then I'm really out of clue.

[motin]

Stalled atm pending feedback / help with firefox-devtools/har-export-trigger#13 (comment)

[lonnen]

@rhelmer can you have a look at this and the corresponding issue?

[rpl]

@janodvarko any idea about what can still be missing or what is going wrong with the issue mentioned in #4 (comment)?

[rpl]

@motin I took another look at this, to check what may still be missing from the code that initialize the network monitor API component, and I guess that it may be related at least to the following:

• the "NetworkActivity" listener has to be started on the console actor to make it able to emit the "networkEvent" RDP events (which, if I'm not wrong, are the once that are collected by the Netmonitor API and then converted into the HAR logs), e.g. it seems that we do something similar here:
  https://searchfox.org/mozilla-central/rev/05d91d3e02a0780f44599371005591d7988e2809/devtools/client/netmonitor/initializer.js#80-83
  and so my guess is that in the experiment API it has to be done near here:

  OpenWPM-WebExtension-Experiment/src/api.js/Monitor.js

  Line 104 in 626fcbb

  await netMonitor.connect(MockToolbox);

  
  by adding something like:

    await target.makeRemote();
    await target.activeConsole.startListeners(["NetworkActivity"]);
    await netMonitor.connect(MockToolbox);

• it seems to me that all the collected requests are cleared by default when the target tab navigates if "devtools.netmonitor.persistlog" preference is not true, which (at least from what I see) it is happening from here: https://searchfox.org/mozilla-central/rev/05d91d3e02a0780f44599371005591d7988e2809/devtools/client/netmonitor/src/connector/firefox-connector.js#171-174, to avoid the requests to be cleared on navigation we may need to make it configurable per "NetMonitor API" instance, which will likely require some changes in mozilla-central first (e.g. optionally allow to pass a parameter to the connect method, something like await netMonitor.connect(MockToolbox, {persistLog: true});)

@janodvarko what do you think about the above two comments? may they be the reasons for the empty HAR archive?