Move output storage out of git and keep historic estimates

[seabbs]

At the moment estimates are stored in git and overwritten with each new update. This causes two issues: 1. Historic estimates are not available and 2. the size of the git repo grows over time meaning that periodically the history must be cleaned.

The ideal solution to this would allow programmatically pushing results for each update with an easy to use bash friendly login process.

[joeHickson]

There had been some discussion about the possibility of publishing the results to harvard dataverse - that can hold a version controlled (ergo history) set of results. Do you know what the outcome of that idea was?

[seabbs]

No aside from my own investigation. I couldn't find anything with an API that would allow you to push in an automated fashion (or at least semi-automated fashion) but tbh it was not a very exhaustive look.

[joeHickson]

http://guides.dataverse.org/en/4.20/api/native-api.html#publish-dataset-api was flagged by Richard as being a possible starting point.
Theres a few possible ways to structure it but it should be as simple as posting the latest dataset and then calling the publish api
It might also be possible to use this as the source for the last run dates / data, pulling it back down if needed.

[seabbs]

both of those sound like good ideas and this looks very viable!

[joeHickson]

Once we start to get this in place we may want to update EpiNow2 to only output into a location folder, not x/latest and x/2020-...

[joeHickson]

I think I might be able to get this to work independently from the i41 restructuring of the data - if so it can go ahead of it.