Do we want to track changes to package locations etc?

[stephlocke]

Why: For Science!
How: Google Big Query or something as part of the build
Pros: Insight into CRAN
Cons: Could be judgey

[maelle]

Technically, the build on Travis would update a Google Big Query database and the take_snapshot function would access that or would the Big Query data be for the archiving only? In any case by build it'd be cool to not query CRAN twice (once for the database and once for the dashboard). 🤔

[stephlocke]

big query for archiving only, tho it could also be an alright source if we wanted to read the dashboard from it

[maelle]

@stephlocke why big query btw?

[stephlocke]

seemed like a cheap storage util for this sort of simple accumulating data, plus it has some native ML capabilities built in so we would get to play 😉

[maelle]

So we need to run a code every hour to create the snapshot (with a bit more info cf #9 ) and send it to a Big Query project. Probaby with bigrquery + DBI.

[maelle]

The data is in the commit history of gh-pages at the moment, I suppose.

[stephlocke]

If the dashboard.Rmd could append the data to a csv as the start of our data capture mechanism, that'd be good.

[maelle]

So two tasks here

• Use GH commits to retrieve past data

• Add the appending of a csv to the data capture.

[stephlocke]

I'd focus on the append first. The looking through git versions of a html file (I believe we gotta look at the compiled html file?) sounds a lot harder and it'd be better to get fresh data capturing sooner. Backfilling is a nice to have!

[maelle]

Bad workflow at the moment. https://github.com/lockedata/cransays/blob/master/.github/workflows/master.yml branch https://github.com/lockedata/cransays/tree/history

• how to I create an orphan branch and then add CSV files to it w/o adding the rest? @stephlocke

• add the actual submission time to the csv.

once the workflow is improved, add it to cron.yml

[hadley]

I think this was done in aa0ab6b