Inconsistent cktest behaviour depending on accessing eigenvectors

[cwehmeyer]

An MSM's cktest() method (PyEMMA-2.5.2, py36_1, conda-forge) shows inconsistent behaviour: if I estimate an MSM and directly perform the CK test I get the expected result. If instead I estimate the MSM, access its (right) eigenvectors first, and then perform the CK test, the test "fails".

See https://gist.github.com/cwehmeyer/1f0dc471da16aef30697f641f0597791 for a minimal example.

My conda environment

# Name                    Version                   Build  Channel
absl-py                   0.1.10                     py_0    conda-forge
appnope                   0.1.0                    py36_0    conda-forge
asn1crypto                0.24.0                   py36_0  
astor                     0.6.2                      py_0    conda-forge
attrs                     17.4.0                     py_0    conda-forge
backports                 1.0                      py36_1    conda-forge
backports.functools_lru_cache 1.5                      py36_0    conda-forge
bhmm                      0.6.2                    py36_1    conda-forge
blas                      1.0                         mkl  
bleach                    1.5.0                    py36_0    conda-forge
bzip2                     1.0.6                         1    conda-forge
ca-certificates           2018.4.16                     0    conda-forge
certifi                   2018.4.16                py36_0    conda-forge
cffi                      1.11.4           py36h342bebf_0  
chardet                   3.0.4            py36h96c241c_1  
conda                     4.5.3                    py36_0    conda-forge
conda-env                 2.6.0                         0    conda-forge
cryptography              2.1.4            py36h842514c_0  
cycler                    0.10.0                   py36_0    conda-forge
decorator                 4.2.1                    py36_0    conda-forge
deprecation               2.0.2                      py_0    conda-forge
dill                      0.2.7.1                  py36_0    conda-forge
entrypoints               0.2.3                    py36_1    conda-forge
freetype                  2.8.1                         0    conda-forge
gast                      0.2.0                    py36_0  
grpcio                    1.11.0           py36hd9629dc_0  
h5py                      2.7.1                    py36_3    conda-forge
hdf5                      1.10.1                        2    conda-forge
html5lib                  0.9999999                py36_0    conda-forge
humanfriendly             4.12                       py_0    conda-forge
icu                       58.2                          0    conda-forge
idna                      2.6              py36h8628d0a_1  
intel-openmp              2018.0.0             h8158457_8  
ipykernel                 4.8.2                    py36_0    conda-forge
ipython                   6.2.1                    py36_1    conda-forge
ipython_genutils          0.2.0                    py36_0    conda-forge
ipywidgets                7.1.2                    py36_0    conda-forge
jedi                      0.11.1                   py36_0    conda-forge
jinja2                    2.10                     py36_0    conda-forge
jpeg                      9b                            2    conda-forge
jsonschema                2.6.0                    py36_1    conda-forge
jupyter                   1.0.0                      py_1    conda-forge
jupyter_client            5.2.2                    py36_0    conda-forge
jupyter_console           5.2.0                    py36_0    conda-forge
jupyter_contrib_core      0.3.3                    py36_1    conda-forge
jupyter_contrib_nbextensions 0.5.0                    py36_0    conda-forge
jupyter_core              4.4.0                      py_0    conda-forge
jupyter_highlight_selected_word 0.2.0                    py36_0    conda-forge
jupyter_latex_envs        1.4.4                    py36_0    conda-forge
jupyter_nbextensions_configurator 0.4.0                    py36_0    conda-forge
kiwisolver                1.0.1                    py36_1    conda-forge
libcxx                    4.0.1                h579ed51_0  
libcxxabi                 4.0.1                hebd6815_0  
libedit                   3.1                  hb4e282d_0  
libffi                    3.2.1                h475c297_4  
libgfortran               3.0.1                h93005f0_2  
libiconv                  1.15                          0    conda-forge
libpng                    1.6.34                        0    conda-forge
libprotobuf               3.5.2                         0    conda-forge
libsodium                 1.0.15                        1    conda-forge
libtiff                   4.0.9                         0    conda-forge
libxml2                   2.9.8                         0    conda-forge
libxslt                   1.1.32                        0    conda-forge
llvmlite                  0.22.0           py36h35728e8_0  
lxml                      4.2.1                    py36_0    conda-forge
markdown                  2.6.11                     py_0    conda-forge
markupsafe                1.0                      py36_0    conda-forge
matplotlib                2.2.2                    py36_1    conda-forge
mdshare                   0.3.1                    py36_0    conda-forge
mdtraj                    1.9.1                    py36_1    conda-forge
mistune                   0.8.3                      py_0    conda-forge
mkl                       2018.0.1             hfbd8650_4  
mock                      2.0.0                    py36_0    conda-forge
msmtools                  1.2.1                    py36_1    conda-forge
multiprocess              0.70.5                   py36_0    conda-forge
nbconvert                 5.3.1                      py_1    conda-forge
nbformat                  4.4.0                    py36_0    conda-forge
ncurses                   6.0                  hd04f020_2  
ninja                     1.8.2                         1    conda-forge
notebook                  5.4.0                    py36_0    conda-forge
numba                     0.37.0          np113py36h6d7bb6e_0  
numexpr                   2.6.4                    py36_1    conda-forge
numpy                     1.13.3           py36ha726252_3  
olefile                   0.45.1                   py36_0    conda-forge
openblas                  0.2.20                        7    conda-forge
openssl                   1.0.2o                        0    conda-forge
packaging                 17.1                       py_0    conda-forge
pandas                    0.22.0                   py36_0    conda-forge
pandoc                    2.1.2                         0    conda-forge
pandocfilters             1.4.1                    py36_0    conda-forge
parso                     0.1.1                      py_0    conda-forge
pathos                    0.2.1                    py36_1    conda-forge
patsy                     0.5.0                    py36_0    conda-forge
pbr                       3.1.1                    py36_0    conda-forge
pexpect                   4.4.0                    py36_0    conda-forge
pickleshare               0.7.4                    py36_0    conda-forge
pillow                    5.0.0                    py36_0    conda-forge
pip                       9.0.3                    py36_0    conda-forge
pluggy                    0.6.0                      py_0    conda-forge
pox                       0.2.3                    py36_0    conda-forge
ppft                      1.6.4.7.1                py36_0    conda-forge
prompt_toolkit            1.0.15                   py36_0    conda-forge
protobuf                  3.5.2                    py36_0    conda-forge
psutil                    5.4.3                    py36_0    conda-forge
ptyprocess                0.5.2                    py36_0    conda-forge
py                        1.5.2                      py_0    conda-forge
pycosat                   0.6.3            py36hee92d8f_0  
pycparser                 2.18             py36h724b2fc_1  
pyemma                    2.5.2                    py36_1    conda-forge
pygments                  2.2.0                    py36_0    conda-forge
pyopenssl                 17.5.0           py36h51e4350_0  
pyparsing                 2.2.0                    py36_0    conda-forge
pyqt                      5.6.0                    py36_4    conda-forge
pysocks                   1.6.7            py36hfa33cec_1  
pytables                  3.4.2                    py36_7    conda-forge
pytest                    3.4.2                    py36_0    conda-forge
python                    3.6.4                hc167b69_1  
python-dateutil           2.6.1                    py36_0    conda-forge
python.app                2                py36h54569d5_7  
pytorch                   0.4.0           py36_cuda0.0_cudnn0.0_1    pytorch
pytz                      2018.3                     py_0    conda-forge
pyyaml                    3.12                     py36_1    conda-forge
pyzmq                     17.0.0                   py36_3    conda-forge
qt                        5.6.2                h9e3eb04_4    conda-forge
qtconsole                 4.3.1                    py36_0    conda-forge
readline                  7.0                  hc1231fa_4  
requests                  2.18.4           py36h4516966_1  
ruamel_yaml               0.15.35          py36h1de35cc_1  
scipy                     1.0.0            py36h1de22e9_0  
seaborn                   0.8.1                    py36_0    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                38.4.0                   py36_0  
simplegeneric             0.8.1                    py36_0    conda-forge
sip                       4.18                     py36_1    conda-forge
six                       1.11.0           py36h0e22d5e_1  
sqlite                    3.22.0               h3efe00b_0  
statsmodels               0.8.0                    py36_0    conda-forge
tae                       0.1.3.dev10+g2c3974b.d20180430           <pip>
tensorboard               1.6.0                    py36_0    conda-forge
tensorflow                1.6.0                    py36_0    conda-forge
termcolor                 1.1.0                    py36_1    conda-forge
terminado                 0.8.1                    py36_0    conda-forge
testpath                  0.3.1                    py36_0    conda-forge
thermotools               0.2.6                    py36_2    conda-forge
tk                        8.6.7                h35a86e2_3  
torchvision               0.2.1                    py36_1    pytorch
tornado                   4.5.3                    py36_0    conda-forge
tqdm                      4.19.6                     py_0    conda-forge
traitlets                 4.3.2                    py36_0    conda-forge
urllib3                   1.22             py36h68b9469_0  
wcwidth                   0.1.7                    py36_0    conda-forge
webencodings              0.5                      py36_0    conda-forge
werkzeug                  0.14.1                     py_0    conda-forge
wheel                     0.30.0           py36h5eb2c71_1  
widgetsnbextension        3.1.4                    py36_0    conda-forge
xz                        5.2.3                h0278029_2  
yaml                      0.1.7                hc338f04_2  
zeromq                    4.2.3                         2    conda-forge
zlib                      1.2.11               hf3cbc9b_2

[thempel]

Similar issues were fixed in #1255, must have missed something.

[marscher]

Mhm it seems that the estimation does not reset the RDL decomposition. Even if we copy the testing model, we will still have the old RDL in place. So my guess is, that we just need to reset it in _estimate to avoid this.