HAR Export Trigger to work with single HTTP event & example page not working

[thegreyfawkes]

Per previous issue report (firebug/netexport#21), I am attempting to extract a single HAR export to capture the traffic that results in a tracking pixel returning a query string parameter.

Previous to HAR Export Trigger, I had this setup sort-of working with latest (non-developer) Firefox, Firebug 2.0.12, and NetExport 0.9b7 (code for this in the link above). The two main issues being that:

1. There would be 3 HAR exports: one for the initial page visit, one for logging into the site, and one for the product search that would result in the desire HAR export.
2. The HAR files exported would have a naming convention default of "(url)(date/time).HAR", and with no real way to arbitrarily name the file. This was (is) preventing me from further automating my regression testing. The real benefit from the previous Firefox/Firebug/NetExport is that I did not need to insert any HTML or script within the pages I wanted to trigger a HAR export for. Unless I'm mistaken, I think that this is a requirement in order to properly use HAR Export Trigger, which is effectively leaving my dead in the water with my automated regression testing efforts.

Lastly, I cannot seem to get the HAR Export Trigger example page to work, even after enabling the API within Firefox. After clicking "Execute XHR", I see an event in the dev panel, but clicking "HAR Export" gives me the message saying that I have not enabled the API ("Security check didn't pass. You need to set 'devtools.netmonitor.har.contentAPIToken' pref to match the string token passed into HAR object API call (browser restart is required").

Thank you in advance for your time!

EDIT: Forgot to add that I also have the devtools.netmonitor.har.enableAutoExportToFile preference enabled as well.

[janodvarko]

HAR Export Trigger 0.5.0 beta 4 is available on AMO
https://addons.mozilla.org/en-US/firefox/addon/har-export-trigger/
(downloadable, but still awaiting review process)
Or you can download the XPI from release notes: https://github.com/firebug/har-export-trigger/releases

I cannot seem to get the HAR Export Trigger example page to work,

Please try the test page again. The error message was wrong, you need to set
extensions.netmonitor.har.contentAPIToken

Follow instructions on this page:
http://janodvarko.cz/har/tests/har-export-trigger/har-export-api.html
and let me know if it works.

The test page shows how to use the HAR API and you can:

• Configure result file name using formatters, see:
  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/toLocaleFormat
  (there is an example in the test page)
• If you don't want to have the local HAR files to be generated set the extensions.netmonitor.har.enableAutomation preference (not the devtools.netmonitor.har.enableAutoExportToFile)
• You can also generate HAR files after every page load using HAR API. See an example on the test page.
• Of course, you can have both, auto generated HAR files (after every page load) and also HAR files generated when you call triggerExport() and pass fileName into it. If you skip the fileName, and pass 'getData' the file isn't created and you only get back the HAR as a string.

First make the example page working and then let me know if all your use cases are covered.

Honza

[thegreyfawkes]

Okay, let's start at the top.

1. Install/Enable HAR Export Trigger 0.5.0 beta 4 - done
   
2. Enable "extensions.netmonitor.har.contentAPIToken" - done
   
3. From http://janodvarko.cz/har/tests/har-export-trigger/har-export-api.html, enabling other test-page expected values - done
   
4. Other preferences that can be set are set as such:
5. Have a HAR file created after every page load
   

• have HAR automation active, but no files created
  

Ensuring that I have all of the above taken care of, I then attempted to load the test page again and received the same error:

Am I doing something wrong?

--Travis

[janodvarko]

extensions.netmonitor.har.contentAPIToken must be set to "test" (no quotes)
This is the string token passed into API calls.

Honza

[thegreyfawkes]

Changing the value to "test" made a difference, as the security check error no longer appears. However, nothing else is shown in the console either. Looking at your instructions, I also navigated to my firefox profile folders (C:\Users\me\AppData\Roaming\Mozilla\Firefox\Profiles) and there appears to be no "\har\logs" folder present. Just to be sure, I've searched my entire C:\ drive in case I somehow missed them...but the only .har files that I have are the ones from my firebug/netexport script.

[janodvarko]

Is there any error displayed in the Browser Console? Ctrl+Shift+J

Perhaps you need to refresh the page?

Honza

[thegreyfawkes]

I refreshed the page (in fact, I restarted the browser after refreshing and checking) and see "HAR log: " in the Ctrl-Shift-J console:

Literally right after I posted this, I now see this in the console:

Is this what I'm supposed to be seeing?

[janodvarko]

Yes! It's the har log (JSON string) fetched through the API (parsed and logged as an object, see the page script in the Debugger panel, F12).

You can also go to /har/logs and see HAR files (in case you have the enableAutoExportToFile pref set to true. Or if you pass fileName in the triggerExport method (e.g. on your test page)

Honza

[thegreyfawkes]

Okay, awesome....I can see the log files created. I have two questions, though.

1. Your previous work with firebug and netexport allowed me to collect HAR files without having to do anything but visit a site. I'm not sure, but If I understand correctly, don't you need to put something into the site you want to collect from to trigger the HAR collection?
2. I was hoping the example site would give me a better look at how to use the "var options" syntax, as all I would really want to do is specify a static name for the file (easier to automate that way) and a local folder to store it in (ideally, a local storage folder that syncs to a cloud service, like Box or OneDrive).

Here's my use-case, and why I want to collect the HAR's to begin with:

• I need to verify that a header value displayed in a query string parameter equals an expected value. Since this happens from a tracking pixel, I have to monitor the HTTP request made when a user searches for a product, which returns the data I'm looking for.
• I noticed that I was able to manually collect the HAR files and check them with the online HAR viewer (http://www.softwareishard.com/har/viewer/), and that the value I needed was definitely there.
• In order to automate the testing completely, I need to have a script that
  • visits the site
  • logs in
  • searches for a product
  • records the ensuing search/HTTP event to a HAR file, and
  • send that HAR file to the online viewer where I could further script the clicking needed to expose the value that I am looking for within the viewer, and
  • record that value to a file

Will I ultimately be able to do this with the HAR Export Trigger?

[janodvarko]

Your previous work with firebug and netexport allowed me to collect HAR files without having to do anything but visit a site. I'm not sure, but If I understand correctly, don't you need to put something into the site you want to collect from to trigger the HAR collection?

You can still collect HAR without putting a script into a site. Just set devtools.netmonitor.har.enableAutoExportToFile pref and a HAR file will be created automatically for ever loaded page.

Using HAR API is not a requirement it's just an additional feature that makes some complex system a lot easier to integrate.

I was hoping the example site would give me a better look at how to use the "var options" syntax, as all I would really want to do is specify a static name for the file (easier to automate that way) and a local folder to store it in (ideally, a local storage folder that syncs to a cloud service, like Box or OneDrive).

I am not sure what you mean by 'var options', but you can specify default name for the generated HAR file in devtools.netmonitor.har.defaultFileName. For example having "archive" (no quotes) in the pref will generate "archive.har" file (no quotes). You can also specify a local folder where these files should be stored in devtools.netmonitor.har.defaultLogDir pref.

Note that if the target folder already contains a file with the same name it won't be overridden. For example, if your default name is "archive" and the folder already contains "archive.har" the next har file name will be "archive-1.har", the next one "archive-2.har", etc.

In order to automate the testing completely, I need to have a script that
visits the site
logs in
searches for a product
records the ensuing search/HTTP event to a HAR file, and
send that HAR file to the online viewer where I could further script the clicking needed to expose the value that I am looking for within the viewer, and record that value to a file.

I don't understand this part. Why you need to include the viewer in the automated process?
Are you trying to get some data from the viewer UI? This is up to you, but it would be a lot easier and safer to inject a script into the monitored page (e.g. Selenium allows that easily as far as I know) get the HAR data as a string and process with JavaScript.

Just for the record, here is an example:

var options = {
  token: "test",  // this tokenvalue must equal to what you have in preferences
  getData: true,  // true == I want to get HAR data in the callback below
  title: "my custom title", // Title of the HAR file
  fileName: "archive" // default name of the HAR file automatically stored, if empty no file is created.
};

// Trigger export now. Since 'getData' option is true, the  HAR data will be passed in the callback.
HAR.triggerExport(options).then(result => {
  var har = JSON.parse(result.data);

  // Now I can access query string like as follows.
  // and iterate its values
  var queryString = har.log.entries[0].request.queryString;
  for (var i=0; i<queryString.length; i++) {
    var param = queryString[i];
    console.log(param.name + " = " + param.value);

    // Check for specific parameter
    if (param.name == "This is the parameter I am looking for!") {
      // OK, I've found the query string created by a tracking pixel. let's send it to my server.
      // (it could be also somehow picked up by Selenium at this point)
      var request = new XMLHttpRequest();
      request.open("POST", "my-server.php", true);
      request.setRequestHeader("Content-Length", param.value.length);
      request.send(param.value);
    }
  }
});

Will I ultimately be able to do this with the HAR Export Trigger?

Yes, I think so. HAR Export Trigger supports the same set of features as Firebug+NetExport combo.

Does that make sense?

Honza

[thegreyfawkes]

HI Honza,

Yes, this makes sense...mostly. My java knowledge is about 3-4 weeks old, so your confusion as to why I wanted to use the online HAR viewer to extract the info is because it was the easiest way I could think of and not knowing a better way. Kind of like taking 3 left turns instead of 1 right - you get to the same place, but if you don't know how to "turn right", then the long, hard way is the only way you can go. :)

I'll see what I can do to work your example above into extracting the query string parameter that I'm looking for. I'm fairly sure I'll be spending a good deal of time googling how to effectively apply it, but at least NOW I have a much more solid foot to walk on than I did last week.

And again, thank you very much. I truly appreciate your time helping me to get this figured out.

[janodvarko]

Excellent

I am closing this report, since all should be working fine now (and btw. the latest release is 0.5.0 beta 5 atm). Feel free to create a new report any time you are experiencing an issue.

Honza

[batmantest7]

Hi thegreyfawkes,

"There would be 3 HAR exports: one for the initial page visit, one for logging into the site, and one for the product search that would result in the desire HAR export."

Were you able to successfully export har files using harexport manual trigger to a single .har file?

Thanks in advance.