Upload perf data

[mtfriesen]

Upload perf data to an Azure append storage blob. Azure takes care of appending concurrent blocks on our behalf. The results are publicly available.

[nibanks]

Why are you uploading to Azure for this? I still think GitHub is the way to go.

[mtfriesen]

Yeah, however I don't want to permanently trash our repo's commit history until we commit to that.

[maolson-msft]

"commit to that", good one

[maolson-msft]

what's this about?

[mtfriesen]

I updated the csv schema (for lack of a better word) to include two new columns, at 0-based indices 1 and 2. array[1] is now commit hash, and array[2] is unix timestamp.

[maolson-msft]

Have you considered using the time of commit as opposed to the time of test?

(I'm not sure whether this is a good idea, just that it's an interesting idea, and I know you can derive the time of commit from the commit hash separately but it could be a real convenience)

(maybe I'm overthinking things here)

[mtfriesen]

Yeah I did consider that (in terms of is the time of test redundant?) and I figured we should collect the actual time the test was run, because the environment we're running in may have its configuration change. Obviously that should not happen very often, but it is best to have an unambiguous record of when the code was tested.

This may be especially valuable if we re-baseline old code on new machines, for example.

[mtfriesen]

If the suggestion here is that the time of commit is also useful, I agree. Looking that up from source control is probably best until proven inadequate (a single source of truth is best) and we could look that up and retroactively update our data files if we need to change the schema.

[mtfriesen]

I suppose a practical dashboard will want to render the data onto a time scale, though, and in that case, is the time of commit the most relevant time? It probably is. But is the most important relationship that one of parent commit and child commit, independent of the number of days between commits?

In other words, do we want to impose a total ordering on the commits for a given branch and display them along the X axis in that order? How should we store that ordering in our data files, if we don't want to perform a git log lookup for each commit on the graph?

[mtfriesen]

Now I'm thinking a real database might actually be a good solution, because it makes sorting by commit/time and filtering by branches easy.

[maolson-msft]

Do you think there would be any problems with having an invariant in the data file where entries are always in git-graph-order?

[maolson-msft]

(i.e. shasum A comes after shasum B in the csv if and only if A comes after B in the git graph)

[mtfriesen]

It would definitely be problematic for this solution, which depends on Azure appending atomically to the end of the data file. We'd need something providing transactions (hmm, git? a real DB?) if we want to modify anything before the end of the data file.

[maolson-msft]

Yeah, makes sense. I was basically asking whether we will ever test a commit that isn't the latest, but that would be a pretty restrictive assumption.

[nibanks]

I wouldn't worry too much about this right now. As you improve what you collect and store, my expectation is that you won't be able to store everything in just one simple file. For instance, per-run latency graphs, are something that have a ton of data points per run, and can't be stored in one file easily with everything else (IMO).

As a result, you inevitably have a model where you have all the raw data, and then usually some post processed version just for the dashboard (though you may be able to do this processing dynamically, but at a cost to download more files and process more data).