[ML] Nginx remote_ip_url_count results regression

[sophiec20]

Forst noticed in: { "build" : { "hash" : "a47564f", "date" : "2018-04-09T21:45:51.347364Z" },

The ML Module Nginx jobs in 6.3 are producing significantly different results to those found in 6.0. Job config:

high_dc(nginx.access.url) over nginx.access.remote_ip
bucket_span=1h
incluencer=nginx.access.remote_ip

6.0.1 Overall score

6.3.0 Overall score

The 6.0.1 top influencer is 72.57.0.53 on 2017-02-23. This is not in the top 10 list for 6.3.0.

The top 10 influencer lists only share 2 remote_ips: 196.89.216.188 and 154.187.183.122

A very notable difference is for 95.142.104.2...

95.142.104.2 6.0.1

95.142.104.2 6.3.0

Is this an intended or unintended consequence of recent improvements?

[tveasey]

I have done some initial investigation of this issue. The change in behaviour is incidentally caused by us detecting periodicity in 6.3 due to improvements in the periodicity testing methodology. However, this has exposed some underlying issues:

1. We allow clusters to form with a lower fraction of the data in the limit we have observed a lot of data and have decreased the decay rate, so increased the model memory, because the residual errors are stable. However, this is being misapplied for models which have many updates per bucket, i.e. population models. The upshot is we can potentially fit a mode to outliers and this is what happens during the course of this run.
2. There is instability in model selection, such that sometimes it chooses a clustered model for the data and sometimes not. The data values are significantly better described by the clustered model so it is not clear what is driving it to sometimes choose the non-clustered model. This needs further investigation.

[tveasey]

I get good results on this data set (better than 6.0.1 IMO) by only using the multimodal residual model. This takes top level model selection out of the equation (we still select the appropriate tail characteristics for each mode) and so works around issue 2 above.

This is something I had considered in the past as an optimisation for population analysis, since we can always choose not to cluster the data and I expect the chance of wanting to cluster to be high. It appears it may also be a good idea from QoR perspective.

The choice of whether to cluster or not can very significantly alter the probability we assign to outlying values. This means that if this model is intermittently selected the analysis can effectively suppress some of the most interesting events. Making a change to exclude unimodal models only increases the stability of the modelling in this respect and so mitigates this effect. On this data set the clustering itself is stable.

I will continue to investigate the underlying cause for issue 2, since it is appropriate to include explicitly include univariate models for individual analysis, but we should separately assess whether excluding unimodal models from population analysis is a good idea.

[tveasey]

The latest behaviour is broadly similar to 6.0.1:

I think there is a case for always selecting a model using population analysis, but this is a significant change which we would need to assess across our full regression suite.