Move diagnostics to separate, light-weight package

[sethaxen]

Following this tweet, there was discussion of moving the diagnostics to a lightweight package that does not assume Chains storage, which could then be used by various PPLs. There's also MCMCDiagnostics.jl, but this package only has a subset of the diagnostics available here and is missing some of the recent improvements to R-hat.

A new package could perhaps make fewer assumptions about the types passed to it, e.g. assuming only the iterator interface, and packages that provide structures for storing posterior samples could then provide convenient overloads so their structures could be passed to the diagnostic functions.

[cscherrer]

Thanks @sethaxen , I'd very much like to have something like this I could easily use with SampleChains.jl

[sethaxen]

Yes, this would be great. I think the first step of this would be a really easy port, since the diagnostics are already implemented as wrappers around implementations that only require real array inputs. Then we could explore generalizations.

So if we move ahead on this, the first big decision is the name of the package. MCMCDiagnostics is a good name, but already taken. I believe @avehtari now uses the term "inference diagnostics." (see e.g. https://youtu.be/HKPm6txxxQM)

[cscherrer]

"ChainDiagnostics"?

[cscherrer]

Since both of these packages have "Chain", and it's shorter to type than "Inference" :)

[cpfiffer]

"ChainDiagnostics"?

I like this one.

[sethaxen]

My pushback is that "Chain" is a very generic term (MCMCChains is fine because it specifies that we're talking about markov chain monte carlo chains). I'm not certain that every diagnostic we might want to include would only be applicable to Markov chains. "SampleDiagnostics" seems too generic. Likewise, "ConvergenceDiagnostics" seems maybe too specific.

I guess we could always call it ChainDiagnostics and then rename later if we expand the scope, but it would be nice to avoid the hassle.

[avehtari]

believe @avehtari now uses the term "inference diagnostics."

Especially when I'm talking diagnostics for generic Monte Carlo, importance sampling, MCMC, variational inference, Laplace, EP, etc. As there are overlap in the algorithms used for diagnosing different inference methods, you may consider whether you would have just one package or later make a separate package if you need diagnostics for methods that don't have chains.

[devmotion]

I just went on and created a new repo: https://github.com/devmotion/InferenceDiagnostics.jl I'll try to extract the parts from MCMCChains that are not related to Chains (ie the diagnostics) while preserving the git history. It's easy to change the name or discard the project as long as it's not registered. I am also curious how lightweight (or heavy) the package will be when all diagnostics are extracted.

[sethaxen]

This sounds like a good first step.

In the end, a priority of mine is that the diagnostics in the package be eventually harmonized with those in ArviZ. That is, where they disagree, we identify why and whether they need to be fixed here or in ArviZ. I raised also in arviz-devs/ArviZ.jl#129 (comment) the question about where such a package might live (I argue it might ultimately fit better under the arviz-devs org), but this is not an important issue right now.

[ParadaCarleton]

This should probably be closed now.