ENH: Markov-switching regression and autoregression models (non-state space)

[ChadFulton]

This is a cleaned up / rewritten version of the Markov switching models from my 2013 GSOC. Allows:

• Markov regression models
• Markov autoregression models
• Fitting via EM algorithm as well as by scoring
• Tests against Stata / Eviews

My thought is to finish developing this alongside @ValeryTyumen who is doing the state space Markov switching models so that we can maintain a constant interface, etc.

[coveralls]

Coverage increased (+0.0005%) to 85.506% when pulling 988623f on ChadFulton:markov-tsa into e62551c on statsmodels:master.

[josef-pkt]

partially in response to #2921 (comment)

@ChadFulton @ValeryTyumen Would it help to merge this into master?

Given that this has good test coverage and looks pretty complete, we could merge it, possibly with a Warning that this is currently under refactoring during GSOC.

On the other hand #2921 has another unmerged PR in between.

General policy could be to allow merges of master into a feature branch and remove it again during an interactive rebase before the final review and merge. The main problem for reviewing is that the merged code shows up as part of the PR, AFAIK and makes the change set less useful.

[ChadFulton]

Lots of improvements and bug fixes, along with example notebooks that can be found here:

• Markov regression models
• Markov autoregression models

With reference to the question about merging, this can probably be merged. The only features missing right now are predict and forecast, but otherwise they are pretty complete models.

[ChadFulton]

@ValeryTyumen Sorry about this, but due to a name change I suggested for you, I renamed the transition to regime_transition in these models, so the MarkovSwitchingParams class changed slightly (instead of transition, the transition probability-related parameters should be regime_transition now).

[ChadFulton]

Rebased on master.

[coveralls]

Coverage decreased (-1.1%) to 84.386% when pulling 2e5d5b2 on ChadFulton:markov-tsa into e62551c on statsmodels:master.

[coveralls]

Changes Unknown when pulling 95714ca on ChadFulton:markov-tsa into * on statsmodels:master*.

[ChadFulton]

Rebased on master. Probably not ready to be merged in as "stable", although the only thing it is actually missing right now is out-of-sample forecasting.

It could be included in 0.8 if there was a way to indicate that it was still new / unstable.

[josef-pkt]

If it's safe enough, then it would be good to get it in.

We can add an "experimental" or similar status qualifier to the docstring.
I need to do this also for some other parts. I would also like to add a blanket warning to the release notes that newly merged code is often not stable or not guaranteed to be stable.

[ChadFulton]

I will plan to merge it this weekend then, but I will improve the test coverage first.

[josef-pkt]

why an object array and not a list of strings?

[ChadFulton]

It's because below I use indexing with lists of integers (the self.parameters[i, 'autoregressive'])

[josef-pkt]

This looks fine based on a very very brief skimming. I leave the details to you and @ValeryTyumen

You refer to EViews in some of the unit tests, but as far as I could see, there is no origin or source for the reference numbers mentioned in other tests. Can you add some comment, so we can find it again later?

[josef-pkt]

This errors on appveyor but not on TravisCI

several files in tests/results cannot be found, e.g.
OSError: File b'C:\\projects\\statsmodels\\statsmodels\\tsa\\regime_switching\\tests\\results\\mar_filardo.csv' does not exist

(BTW: current master has the failing imputation test disabled, and PRs should run green after rebase.)

[coveralls]

Changes Unknown when pulling 404f065 on ChadFulton:markov-tsa into * on statsmodels:master*.

[coveralls]

Coverage decreased (-1.2%) to 84.302% when pulling 1d7fab8 on ChadFulton:markov-tsa into 03f2331 on statsmodels:master.

[coveralls]

Coverage decreased (-1.2%) to 84.302% when pulling 1d7fab8 on ChadFulton:markov-tsa into 03f2331 on statsmodels:master.

[ChadFulton]

Looks green to me, but I can't figure out how to interpret the coverage details; it doesn't seem like the coverage report has anything to do with the changes in this PR.

[josef-pkt]

@ChadFulton add a module __init__.py to the test directory.

I think TravisCI is not running any tests here.
clues:
when the csv files were missing, then only appveyor errored but not travis
comparing Ran 4667 tests which is the same as a previous run in master https://travis-ci.org/statsmodels/statsmodels/jobs/138652624

I don't know why appveyor and travis behave differently with respect to this.

[ChadFulton]

Well that would make sense. Thanks for catching that!

[josef-pkt]

I don't see why now travis complains about missing results/xxx.csv files. Those should be automatically included as datafiles in the source distribution (all csv, txt and dta files in results directories)
https://travis-ci.org/statsmodels/statsmodels/jobs/138804778

[josef-pkt]

@ChadFulton you could try locally setup.py sdist and look at what files are included in the archive file

I just realized that the travis error message has a b in front of the filename. Is pandas or your path composition using binary instead of string for the path/name ?

[josef-pkt]

try using os.path.join instead of explicit composition, which is cleaner anyway

[josef-pkt]

Since appveyor doesn't have problems, it's os (Linux) specific os.path issue, not a missing file in the sdist

[ChadFulton]

Thanks for debugging this, I haven't run into problems like this before.

[coveralls]

Coverage increased (+0.3%) to 85.78% when pulling 588b5f2 on ChadFulton:markov-tsa into 03f2331 on statsmodels:master.

[coveralls]

Coverage increased (+0.3%) to 85.767% when pulling 3c19b15 on ChadFulton:markov-tsa into 03f2331 on statsmodels:master.

[josef-pkt]

I don't know why we need the __init__.py also in results. If setup.py is correctly configured, then it shouldn't be necessary.

merging given it's all green

Thanks Chad

[WUhailing]

predict or forecast are not included in the package? I couldn't deploy neither of them after model fitting.

[ChadFulton]

Only in-sample predictions are available right now. Out-of-sample forecasting has not yet been added, although pull requests adding that functionality would be greatly appreciated.

[WUhailing]

The one-step forward forecast of time T+1 value is not a problem, but how to forecast the hidden status of the new observation? or any suggestions on deriving hidden status from time T+1 to T' of the new observations y_T+1,t_T+2,...y_T' after building Markov autoregression model on observations y_1,y_2,... y_T?

[ValeryTyumen]

I described and implemented this algorithm in my pull request - #2921 (comment).

[WUhailing]

Thanks. I have looked at the description, if I understand them correctly, the forecast algo is for unknown regimes and unknown observations, what if the new observations are known, can we still derive the hidden regimes?

[ValeryTyumen]

what if the new observations are known, can we still derive the hidden regimes?

Sounds like a usual Kim filter routine to me.

[WUhailing]

Ok, I will take a close look at Kim Filter later, still new to this.

[NowanIlfideme]

Am also interested in out-of-sample forecasting, both conditional on regime/exogenous and with unknown regime.