Display optimization progress for find_MAP #1813

bwengals · 2017-02-23T07:57:28Z

If disp=True in call to find_MAP, then optimization progress is printed out.

If ipython notebook is being used, an updating table showing parameter progress is displayed. Colors are taken from PyMC3 logo.
If ipython notebook is not being used, more basic info (number of iterations, log posterior value, ||grad||, runtime) is displayed every 2 seconds.

Also in this PR is a small change to appearances of tqdm. If the user is developing in a notebook, the better looking tqdm notebook iteration bar is displayed.

Let me know if there are any changes, cosmetic or otherwise, you would like or would prefer.

twiecki · 2017-02-23T08:32:50Z

pymc3/tests/test_starting.py

+    with Model():
+        mu = Normal('mu', 0, 1, testval=100)
+        # test with verbosity turned off (it is on by default)
+        map_est1 = starting.find_MAP(fmin=starting.optimize.fmin_bfgs,   disp=True, callback = lambda x: 1.0)


no space for kwargs

what do you mean?

should be callback=lambda x: 1.0

and fmin=starting.optimize.fmin_bfgs, disp=True

twiecki · 2017-02-23T08:32:58Z

pymc3/tests/test_starting.py

+    #   should not error
+    with Model():
+        mu = Normal('mu', 0, 1, testval=100)
+        map_est1 = starting.find_MAP(fmin=starting.optimize.fmin_bfgs,   disp=False)


no extra spacing

twiecki · 2017-02-23T08:34:01Z

pymc3/tuning/starting.py

+            """.format(parameter, values["size"], values["valstr"])
+        html += "</table>"
+        self.param_table.value = html
+


empty-lines

twiecki · 2017-02-23T08:35:04Z

Do you have a screenshot of what the Monitor in find_MAP() looks like?

bwengals · 2017-02-23T09:02:31Z

@twiecki added screenshots to the original comment. One thing you'll notice is that values of untransformed variables are printed. Is there a clean way to transform them back? I had a hard time finding that poking around.

twiecki · 2017-02-23T09:19:00Z

I'm not sure about Monitor. find_MAP() is not really a strong focus of this package. We could use it in other parts too (MCMC) but not sure how useful that really is.

Then I think there is the problem of creating a lot of html output in a cell. If you look at the raw NB after an optimization run, does it have a huge amount of html code in it?

bwengals · 2017-02-23T09:54:40Z

It looks like the monitor does not persist across saves. Since it uses the interactive widgets, one needs to do this to save it.

And additionally, I imagine a version for MCMC chains could get messy pretty fast. I guess my arguments in favor would be that finding the MAP point is a good first step in (not all!) but many situations, like GPs. It is helpful (for me at least) to understand subtle problems in model specification by seeing the parameters veer off somewhere, without waiting around till it terminates. But very true, it's not an optimization package!

Could dial back the css to not seem so "branded", since its not the focus.

fonnesbeck · 2017-02-23T13:36:27Z

I'm not necessarily opposed, as long as it stays relatively lightweight. I do think more sophisticated monitoring would be nice for NUTS and ADVI. One could make it optional, with a monitor=True flag, or similar.

I've thought about using Bokeh for some graphical monitoring, similar to what dask does for its dashboard. But you wouldnt necessarily want it all the time.

bwengals · 2017-02-27T03:55:08Z

Could do something similar for the samplers and ADVI. For something lightweight, maybe it would look similar but show marginal mean and standard deviation in the "Current Value" column?

Yeah, default should be monitor=False. Like this?

twiecki · 2017-03-04T13:28:50Z

pymc3/sampling.py

@@ -187,7 +187,11 @@ def _sample(draws, step=None, start=None, trace=None, chain=0, tune=None,
    sampling = _iter_sample(draws, step, start, trace, chain,
                            tune, model, random_seed)
    if progressbar:
-        sampling = tqdm(sampling, total=draws)
+        try:


We should refactor this to our own tqdm function which does this.

That sounds good. Where would you recommend it goes?

twiecki · 2017-03-04T13:29:12Z

pymc3/tests/test_starting.py

+    with Model():
+        mu = Normal('mu', 0, 1, testval=100)
+        # test with verbosity turned off (it is on by default)
+        map_est1 = starting.find_MAP(fmin=starting.optimize.fmin_bfgs, disp=True, callback = lambda x: 1.0)


no white-space around callback kwarg =.

twiecki · 2017-03-04T13:29:47Z

pymc3/tuning/starting.py

+
+
+
+class Monitor(object):


Does there exist nothing in the jupyter (or bokeh) universe that does this already?

Gosh, not that I've seen. Couldn't find anything similar after a quick look around, but I'll check further. I took the idea of monitoring optimization progress from GPy, theirs was made from scratch also.

…inished formatting html table

…arting.py

aseyboldt · 2017-03-05T16:15:53Z

I think a couple of calls to tqdm.close() are missing. Otherwise it sometimes prints weird things into the next output in the notebook if the iterator is not exhausted (because of an exception or a keyboard interrupt). Something like this should do it:

vals = tqdm.tqdm(some_iter)
try:
    for val in vals:
        pass
finally:
    vals.close()

Or alternatively using contextmanager.closing.
Most of them were missing before this patch...

aseyboldt · 2017-03-05T16:16:33Z

pymc3/sampling.py

@@ -401,7 +405,11 @@ def sample_ppc(trace, samples=None, model=None, vars=None, size=None, random_see
    seed(random_seed)

    if progressbar:
-        indices = tqdm(randint(0, len(trace), samples), total=samples)
+        try:


close missing

aseyboldt · 2017-03-05T16:17:29Z

pymc3/variational/advi.py

        uw_i, elbo_current = f()
        if np.isnan(elbo_current):
            raise FloatingPointError('NaN occurred in ADVI optimization.')
+        # set up progress bar
+        try:


missing close

aseyboldt · 2017-03-05T16:18:24Z

pymc3/variational/advi.py

@@ -400,7 +404,14 @@ def rvs(x):
    trace = pm.sampling.NDArray(model=model, vars=vars_sampled)
    trace.setup(draws=draws, chain=0)

-    range_ = trange(draws) if progressbar else range(draws)
+    if progressbar:


missing close

aseyboldt · 2017-03-05T16:18:54Z

pymc3/variational/advi_minibatch.py

@@ -518,9 +518,14 @@ def is_shared(t):
    updates = OrderedDict(optimizer(loss=-1 * elbo, param=params))
    f = theano.function(tensors, elbo, updates=updates, mode=mode)

+    # set up progress bar
+    try:


missing close

bwengals · 2017-03-06T00:10:55Z

Thanks @aseyboldt, nice catch. Will include this.

bwengals · 2017-03-07T21:34:33Z

Want to give a heads up that, in addition to the suggestions, I'm working on having it print untransformed variables, ie. the value of sigma instead of sigma_log_ .

twiecki reviewed Feb 23, 2017

View reviewed changes

twiecki reviewed Mar 4, 2017

View reviewed changes

bwengals added 10 commits March 4, 2017 20:04

first crack at optimization monitor

dbdba68

notebook version done

d7836c5

added try, except block for using ipython notebook display of tqdm, f…

420227f

…inished formatting html table

started terminal version of monitor

f747a24

progress bars appear to be working for advi

5e242e7

started string formatting in display_terminal

8524f8e

finished formatting terminal output

c47aecc

added basic test of api

265ae1c

removed emtpy lines from starting.py, remove extra spacing in test_st…

bd54682

…arting.py

fix so it prints logpost, instead of -logpost

a72da5b

bwengals force-pushed the find_MAP-monitor branch from 7296412 to a72da5b Compare March 5, 2017 04:08

aseyboldt reviewed Mar 5, 2017

View reviewed changes

This was referenced Mar 25, 2017

tqdm progress bar display for ipython notebooks #1951

Closed

find_MAP monitor for notebooks #1980

Merged

bwengals closed this Apr 1, 2017

bwengals deleted the find_MAP-monitor branch June 9, 2017 20:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Display optimization progress for find_MAP #1813

Display optimization progress for find_MAP #1813

bwengals commented Feb 23, 2017 •

edited

Loading

twiecki Feb 23, 2017

bwengals Feb 23, 2017

twiecki Feb 23, 2017

twiecki Feb 23, 2017 •

edited

Loading

twiecki Feb 23, 2017

twiecki Feb 23, 2017

twiecki commented Feb 23, 2017

bwengals commented Feb 23, 2017 •

edited

Loading

twiecki commented Feb 23, 2017

bwengals commented Feb 23, 2017 •

edited

Loading

fonnesbeck commented Feb 23, 2017

bwengals commented Feb 27, 2017

twiecki Mar 4, 2017

bwengals Mar 5, 2017

twiecki Mar 4, 2017

twiecki Mar 4, 2017 •

edited

Loading

bwengals Mar 5, 2017 •

edited

Loading

aseyboldt commented Mar 5, 2017 •

edited

Loading

aseyboldt Mar 5, 2017

aseyboldt Mar 5, 2017

aseyboldt Mar 5, 2017

aseyboldt Mar 5, 2017

bwengals commented Mar 6, 2017

bwengals commented Mar 7, 2017 •

edited

Loading




		class Monitor(object):

Display optimization progress for find_MAP #1813

Display optimization progress for find_MAP #1813

Conversation

bwengals commented Feb 23, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

twiecki Feb 23, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

twiecki commented Feb 23, 2017

bwengals commented Feb 23, 2017 • edited Loading

twiecki commented Feb 23, 2017

bwengals commented Feb 23, 2017 • edited Loading

fonnesbeck commented Feb 23, 2017

bwengals commented Feb 27, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

twiecki Mar 4, 2017 • edited Loading

Choose a reason for hiding this comment

bwengals Mar 5, 2017 • edited Loading

Choose a reason for hiding this comment

aseyboldt commented Mar 5, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bwengals commented Mar 6, 2017

bwengals commented Mar 7, 2017 • edited Loading

bwengals commented Feb 23, 2017 •

edited

Loading

twiecki Feb 23, 2017 •

edited

Loading

bwengals commented Feb 23, 2017 •

edited

Loading

bwengals commented Feb 23, 2017 •

edited

Loading

twiecki Mar 4, 2017 •

edited

Loading

bwengals Mar 5, 2017 •

edited

Loading

aseyboldt commented Mar 5, 2017 •

edited

Loading

bwengals commented Mar 7, 2017 •

edited

Loading