vader.cluster method fails with ValueError: array must not contain infs or NaNs

[yalchik]

Some jobs fail with the following error:

Job failed: 0a1c51a1-2ce1-423e-92ca-0f04db8cbf3e with err=array must not contain infs or NaNs, Traceback: Traceback (most recent call last):
  File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/hp_opt/vader_hyperparameters_optimizer.py", line 198, in run_cv_full_job
    result = job.run()
  File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/hp_opt/job/abstract_optimization_job.py", line 88, in run
    cv_fold_result = self._cv_fold_step(X_train, X_val, W_train, W_val)
  File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/hp_opt/job/full_optimization_job.py", line 42, in _cv_fold_step
    test_latent_loss) = clustering_func(X_train, X_val, W_train, W_val)
  File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/hp_opt/job/full_optimization_job.py", line 126, in _single_clustering
    effective_k = len(Counter(vader.cluster(X_train, W_train)))
  File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/vader.py", line 562, in cluster
    return self._cluster(mu_tilde, mu_c, sigma2_c, phi_c)
  File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/vader.py", line 425, in _cluster
    p = np.array([f(mu_t, mu[i], sigma2[i], phi[i]) for i in np.arange(mu.shape[0])])
  File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/vader.py", line 425, in <listcomp>
    p = np.array([f(mu_t, mu[i], sigma2[i], phi[i]) for i in np.arange(mu.shape[0])])
  File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/vader.py", line 424, in f
    return np.log(self.eps + phi) + np.log(self.eps + multivariate_normal.pdf(mu_t, mean=mu, cov=np.diag(sigma2)))
  File "/home/iyalchyk/vader_venv/lib/python3.7/site-packages/scipy/stats/_multivariate.py", line 525, in pdf
    psd = _PSD(cov, allow_singular=allow_singular)
  File "/home/iyalchyk/vader_venv/lib/python3.7/site-packages/scipy/stats/_multivariate.py", line 158, in __init__
    s, u = scipy.linalg.eigh(M, lower=lower, check_finite=check_finite)
  File "/home/iyalchyk/vader_venv/lib/python3.7/site-packages/scipy/linalg/decomp.py", line 445, in eigh
    a1 = _asarray_validated(a, check_finite=check_finite)
  File "/home/iyalchyk/vader_venv/lib/python3.7/site-packages/scipy/_lib/_util.py", line 263, in _asarray_validated
    a = toarray(a)
  File "/home/iyalchyk/vader_venv/lib/python3.7/site-packages/numpy/lib/function_base.py", line 499, in asarray_chkfinite
    "array must not contain infs or NaNs")
ValueError: array must not contain infs or NaNs

This error might be related to the interim calculations, since the input data set does not contain ins or NaNs.

[yalchik]

Fixed in 00d2f84