ENH: Add plot_states and Pandas filtered state functions to MLEModel

[jengelman]

A common part of post-estimation work for statespace models is looking at how the state variable means/variances change over time. Both UnobservedComponents and RecursiveLS have really nice helper functions for plotting state means. It'd be great to get something similar for SARIMAX, VARMAX, et al and custom models without having to subclass all of them, by adding this to MLEModel. Additionally, functions that return the filtered states and state covariances in Pandas dataframes would be really convenient.

As far as I can tell, this could be done by reusing the existing code from RecursiveLS and a bit of refactoring for the existing two models. Happy to submit a PR for either or both parts of this if there aren't further complications I'm not aware of.

[ChadFulton]

On Tue, Dec 26, 2017 at 7:28 PM, Joshua Engelman ***@***.***> wrote: A common part of post-estimation work for statespace models is looking at how the state variable means/variances change over time. Both UnobservedComponents and RecursiveLS have really nice helper functions for plotting state means. It'd be great to get something similar for SARIMAX, VARMAX, et al and custom models without having to subclass all of them, by adding this to MLEModel. Additionally, functions that return the filtered states and state covariances in Pandas dataframes would be really convenient.

It's definitely a good idea to make them a little more accessible. At a bare minimum we should have them wrapped in Pandas objects with the appropriate index. A couple of questions: - For SARIMAX and VARMAX, the states are (as far as I know) a little less interpretable than e.g. UnobservedComponents; are there particular states you use for a particular purpose? - You mention returning covariances in Pandas DataFrames - are you thinking of a MultIndex, or something else? Best, Chad

[jengelman]

@ChadFulton For autoregressive or lag coefficients in in VARMAX and SARIMAX, it'd be nice to be able to visually check for changes in the "strength" of relationships without breaks_cusumolsresid (which doesn't show you the changepoints) or recursively running acorr_ljungbox on every set of residuals. I agree that it's a little bit less interpretable than level, trend, etc, but it's not really any less so than whatever endog used in RecursiveLS. Maybe plot functions could take an optional mapping from coefficients to display names in case people want to customize the displays.

For the second point, a multiindex sounds great for covariances.

Edit: somewhat related, just noticed VECM doesn't expose filtered_states etc at all (despite having the filtered probability plot function)

[jengelman]

Just noticed that the Markov models/results, which also doesn't expose filtered state (just probabilities) don't inherit from the MLE equivalents. Are you planning on changing that down the road, or keeping them separate?

[ChadFulton]

(My mistake on closing, hit the wrong button)

Just noticed that the Markov models/results, which also doesn't expose filtered state (just probabilities) don't inherit from the MLE equivalents. Are you planning on changing that down the road, or keeping them separate?

The Markov models aren't linear Gaussian state space models, so in general I don't think it would be useful to extend statespace.MLEModel, although perhaps some of the functionality in the latter should be placed into a Mixin, if it is also useful / valid in the Markov model case.