Timeseries visualisation

[baentsch]

Moving a discussion from a PR #20 to an issue:

@baentsch : Question to @dstebila We're collecting all data since Oct 4 into the website display (and the file site.tgz): Shall we keep doing this or become more selective? "Oct 4" is actually a parameter in gen_website.sh...

@dstebila : Unclear to me. Let's reflect on the purpose of presenting it as a time series. I understand it serves two purposes: seeing how performance changed after important commits were merged, and providing a visual average over time (effectively adding more data points).

Agreed. So what about adding a mechanism to selectively set dates that shall become part of the visualization export (encoded in gen_website.sh? ) and which could be manually amended/checked in to github as and when substantial changes to the code base occur and as and when the current number of test runs shown becomes too large to derive meaningful insight from), e.g., a file containing a list of dates (instead of the single start date currently passed as parameter to the HTML-generator python script)? This would be a whitelist approach at the generating end (dropping data from export).
An alternative could be a simple "pruning" script deleting unwanted/redundant data/date points from the current daily exports, i.e., a blacklist approach that could be done at the receiving end (still retaining all data points without visualizing all of them). My preference would be on this, second option.
@dstebila: Any preference? Alternative suggestions?

[dstebila]

So we'd prune flawed / extremely old data points in the data exports on the server side. The clients would download all the non-pruned data but then only show say the last 2 weeks?

[baentsch]

So we'd prune flawed / extremely old data points in the data exports on the server side.

Yes, that'd be part of proposal 1.

The clients would download all the non-pruned data but then only show say the last 2 weeks?

Not quite: Proposal 2 would prune delivery (to say the last 2 weeks in your example) on the server side, too: This is driven by a file listing the to-be-displayed data points to the client logic (example here): The client only downloads those (days listed in the control file), while the server may have more.

So, indeed, a third option could be to also add a client-side pruning option (made accessible via something like a "download more" button)

[dstebila]

Proposal 2 that prunes delivery on the server side is good.

[baentsch]

Resolved by adding trivial script to server-side deploy

#!/bin/sh

for f in *.list; do
   mv $f orig-$f
   tail orig-$f > $f
done