Variable referenced before assignment in 3D KDE

[lpsinger]

I occasionally get this traceback:

/home/user/local/lib/python2.7/site-packages/sky_area/sky_area_clustering.py:632: RuntimeWarning: divide by zero encountered in log
  return np.sum(np.log(self.posterior(pts))) - nparams/2.0*np.log(self.kde_pts.shape[0])
Traceback (most recent call last):
  File "/home/user/local/bin/run_sky_area.py", line 170, in <module>
    skypost3d = sac.Clustered3DKDEPosterior(np.column_stack((data['ra'], data['dec'], data['dist'])))
  File "/home/user/local/lib/python2.7/site-packages/sky_area/sky_area_clustering.py", line 581, in __init__
    self._set_up_optimal_k()
  File "/home/user/local/lib/python2.7/site-packages/sky_area/sky_area_clustering.py", line 295, in _set_up_optimal_k
    bic = self._set_up_optimal_kmeans(k, self.ntrials)
  File "/home/user/local/lib/python2.7/site-packages/sky_area/sky_area_clustering.py", line 343, in _set_up_optimal_kmeans
    self._set_up_kmeans(k, means=best_means, assign=best_assign)
UnboundLocalError: local variable 'best_means' referenced before assignment

Looking at the _set_up_kmeans function, the immediate cause is probably that bic evaluates to np.NINF on every loop iteration. Strictly speaking, it's also possible that the loop count ntrials is zero and the loop body never runs, but this is unlikely because ntrials seems to come straight from the program's command line arguments.

This is event 317140 from the 2016 scenario in the first2years paper.

I'll set the random seed to a fixed value from now on so that the results are reproducible.

[lpsinger]

I just got this again with event 833111 from 2016. I get it reliably when I run with --enable-distance-map and set --seed=0. It's very odd how many events fail when I set --seed=0. I wonder if somehow that seed is an especially bad one and produces very repetitive pseudorandom numbers?

Simply changing bic > best_bic to bic >= best_bic would protect against this exception:

if bic >= best_bic:
    best_means = self.means
    best_assign = self.assign
    best_bic = bic

What, if any, undesirable consequences might this have?

[farr]

I think your suggestion for solving this problem would result in the possibility of no successful trials for the clustering step.

Will

On 29 Sep 2015, at 23:13, Leo Singer notifications@github.com wrote:

I just got this again with event 833111 from 2016. I get it reliably when I run with --enable-distance-map and set --seed=0. It's very odd how many events fail when I set --seed=0. I wonder if somehow that seed is an especially bad one and produces very repetitive pseudorandom numbers?

Simply changing bic > best_bic to bic >= best_bic would protect against this exception:

if bic >= best_bic:
best_means = self.means
best_assign = self.assign
best_bic = bic

What, if any, undesirable consequences might this have?

—
Reply to this email directly or view it on GitHub.

[lpsinger]

OK. The more immediate cause is that the for the 3D KDE, the BIC is occasionally coming up as -inf because the posterior is identically zero at some of the sample points. Why would that happen? Is this just underflow?

[farr]

If those points are in the clusters that we just cut out of the posterior, then they can be far enough away from the other points that their posterior underflows to zero, yes. I'm not sure what to do about this....

Will

On 30 Sep 2015, at 03:32, Leo Singer notifications@github.com wrote:

OK. The more immediate cause is that the for the 3D KDE, the BIC is occasionally coming up as -inf because the posterior is identically zero at some of the sample points. Why would that happen? Is this just underflow?

—
Reply to this email directly or view it on GitHub.