E-Commerce Reports should not aggregate data to "-1"

[anonymous-matomo-user]

We are using the ecommerce feature of Piwik for our websites and have noticed that items are aggregated to "-1". This occurs in both UI and API for "Product SKU" and "Product Name" reports.

This piece of code simulates just some product views and a purchase so that the ecommerce report shows up in the UI:

var piwikTracker = Piwik.getTracker("http://localhost/piwik/piwik.php", 1);

for (i = 0; i < 100; i++) {
  piwikTracker = Piwik.getTracker("http://localhost/piwik/piwik.php", 1);
  piwikTracker.setEcommerceView("id_book"+i, "Book"+i, "Books", 7.50);
  piwikTracker.trackPageView();
}

piwikTracker = Piwik.getTracker("http://localhost/piwik/piwik.php", 1);
piwikTracker.addEcommerceItem("id_book1",  "Book1", "Books", 7.50, 1);
piwikTracker.trackEcommerceOrder("order1",7.50);
piwikTracker.trackPageView();

The data attached in the CSV has been generated using the code above. As you can see, some product views are missing, e.g. everything after id_book52 as well as id_book6 to id_book9. The information itself is correctly in the database.

The huge negative impact of this aggregation is not visible: There are 10 visits in sum to id_book60, but it also aggregates to "-1" and does not show up in the report. So you cannot see products which are converting bad if you have a lot of products in your store.

If the config value "datatable_archiving_maximum_rows_subtable_referers" is set higher than 50, data is no longer aggregated and we see all product views correctly. However I assume raising this value might affect the UI in other views in a negative way.

Would it be possible to implement one of the following options?

1. Ignore the parameter, i.e. do not aggregate the data in ecommerce reports
2. Introduce an own parameter for the ecommerce reports
3. Only aggregate data with "visits = 1" and rename "-1" to something more meaningful like "Products with only 1 Visit"

Keywords: ecommerce, aggregation, -1

[anonymous-matomo-user]

Attachment: piwik csv export example data
Piwik Export _ Product SKU _ Friday 20 April 2012.csv

[robocoder]

In your example, you created 102 tracker objects, with 101 instances each trying to send an asynchronous request to the server. Due to request pipelining, the browser may restrict the number of concurrent requests causing some to be queued up, others to timeout, etc.

Also, if your server is processing concurrent requests for the same visitor, there may be race conditions when aggregating because the scripts run in separate threads of execution.

So, try making only 1 call to:

piwikTracker = Piwik.getTracker("http://localhost/piwik/piwik.php", 1);

[anonymous-matomo-user]

Thanks for your response. The code was just an example. We can see this aggregation to "-1" in our production environment as well when using the Piwik tracker as intended. I'm also sending the requests and then viewing it in the UI (e.g. one hour later) so I don't think the behaviour is related to a race condition.

It's also correct that the requests are sent asynchronously according to the database, but when viewing their data in UI they are aggregated and ordered - everything from id_book1 to id_book51 is visible, everything afterwards and in between missing.

IMO something should be changed. I don't know how aggregation is intended to work but at least ecommerce data should not be aggregated by alphabet. We are missing bad performing products. I was really surprised that "datatable_archiving_maximum_rows_subtable_referers" helped me out (no documentation on the website) but I don't know anything about its side-effects.

[mattab]

Thanks for the report.

The fact that ecommerce data are truncated and that datatable_archiving_maximum_rows_subtable_referers works (which as you point out is "surprising") is because of the implementation: ecommerce uses Custom Variables report to get the number of views on ecommerce product pages. Therefore we will never truncate ecommerce products which have sold at least once, however we will truncate products which were seen but not bought.

Still we can make things better:

• Bug: Ecommerce product reports show -1 instead of "Others"
• Bug: Could we "truncate" less the Custom Variable Ecommerce-specific subtables?

[diosmosis]

In e6b6d8a: Refs #3116, make summary row label readable in ecommerce goals reports and add tests for those changes.

[diosmosis]

@matt, I think it might be possible to increase the limit for ecommerce-specific reports, but it would take a while for me to figure out what the code is doing.

[mattab]

In e1cbf1d: Fixes #3116 If website is ecommerce, do not limit custom variables reports to 1,000 rows. Limit to 50,000 instead.