Maybe split gold standards to a separate project?

[eerolinna]

I've started to think that maybe the gold standards should be in a separate project. This idea is not yet fully formed but I thought it would be good to put it out now. I'll use the name goldstandards for this hypothetical project.

• posteriordb provides posterior names that are used as unique identifiers in goldstandards
• goldstandards can get greater flexibility in how it is implemented: perhaps a central database and a web server along with client packages would work best for that project instead of the git repo with PRs model that we currently use in posteriordb
• Moving to a separate project doesn't mean we can't keep the same API, so gs <- gold_standard(po) could still work (however there are some caveats)
• One advantage could also be that we could first focus on getting many posteriors to posteriordb and only later add the gold standards. Of course this doesn't necessarily require two separate projects

Going more general

We dont have to restrict the other project to just gold standards, it could include posterior samples for any method. I'll call this more general project posteriorsamples

Selecting a gold standard

Having the ability to upload multiple posterior samples for a posterior might make it easier to come up with a good gold standard

• Contributors could upload several potential gold standards
• An expert user could decide which if any of these should be granted the status of an gold standard
• To help them decide we could have an user interface that allows them to see marginal density plots, diagnostic values etc.

Developing inference methods

• A researcher develops a new variational inference method niceinference and runs it on 50 posteriors from posteriordb. They publish a paper about their method that includes comparisons to gold standards. They also upload the posterior samples to posteriorsamples.
• You are working on an improved version of niceinference. You run your method on the same 50 posteriors. You also download the posterior samples of niceinference. Now in your paper you can include both comparisons to the gold standard and between the two methods.

Next steps

So as a reminder, this idea is not yet fully developed. Maybe keeping gold standards under posteriordb is the best idea. Maybe it's not. I might write a follow-up post later if the idea develops further, right now I just wanted to put this out here.

I'll tag you here so you notice this, but commenting right now is not necessary (but if you got some ideas from this of course feel free to comment) @MansMeg @paul-buerkner

[MansMeg]

Im not sure about this. The stan team (with paul as lead) is looking into a posteriors R package that is similar. We should build on that instead is my guess. But @paul-buerkner paul knows more about this.

[eerolinna]

Yes if we can build on it that would be ideal. Looking forward to hearing more about it.

[eerolinna]

@paul-buerkner can you give a short summary on what the posteriors R package is about?

[paul-buerkner]

From the current README:

The posterior R package will provide various tools for working with posterior distributions that are represented by draws.

Goals:

• provide efficient methods for converting between different representations of posterior draws
• provide summaries of posterior draws with names that can be adopted to standardize conventions in the bayesian R package world
  provide R implementations of the most useful MCMC diagnostics without depending on RStan

[eerolinna]

Thanks! Ah so I guess it's a public repo, can you link to it? I couldn't find it under either stan_dev or your user

[paul-buerkner]

Will be public in a few days. Am Do., 31. Okt. 2019 um 16:19 Uhr schrieb Eero Linna < notifications@github.com>:

…

Thanks! Ah so I guess it's a public repo, can you link to it? I couldn't find it under either stan_dev or your user — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#63>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADCW2AE6F6GDMMB5442P74LQRLZP7ANCNFSM4JEDM5NA> .

[MansMeg]

Now, this is public. https://github.com/jgabry/posterior

So I close this issue.