Cross Node Metric Correlations

[andrewm4894]

Creating this public discussion for Cross Node MC ideas and initial approaches.

Wanted to move some cross node mc discussion from this PR that optimizes the agent based single node MC quite a lot into a GH discussion here where we can discuss ideas and approaches in a more public and suitable place.

[andrewm4894]

Just Aggregate the p-values in some way

The dumbest idea I can think of would be to just average the p-values from the kstwo results up to some limited number of nodes - eg maybe 5 (no real reason why 5 apart from gut feel).

https://stats.stackexchange.com/questions/78596/when-combining-p-values-why-not-just-averaging

So idea here is that we would just have cloud BE average the p-values for each node and then do the normalization step in the cloud on the fly. I think this might actually work for smallish numbers of nodes as really all we want to do with that averaged p-value is to rank the metrics, its ok that this averaged p-value is not really a p-value anymore.

So on the overview screen if up to 5 nodes are selected the metric correlation button is available and cloud will just average the results as above - in the case of a single node this will just be an average of 1 and so revert back to single node MC.

[shyamvalsan]

But what would we do with more than 5 (or whatever X) nodes? Averaging becomes less useful the more we average over in this case, doesn't it?

Do other aggregate functions make sense? max? sum?

And does this maybe work better for the volume based approach @ktsaou proposed since we are not using p-values there.

[andrewm4894]

need to do some more research to see if is a way to either aggregate ks results in some more principled way. or if nodes can provide some part of the calc already (eg their CDF's and then if they can easily be aggregated for the ks test to be done without having to centrally agg all the data in the one place to begin with).

if all nodes are on the same parent then we can do it all proper at the edge but that's typically going to be just a special case we might want to explicitly handle somehow.

https://stats.stackexchange.com/questions/548514/aggregating-individual-ks-2-samp-results