[WIP] Fix for #74

[kudkudak]

Otherwise this happens :) :

import edward
from edward.models import Variational
from edward.models import Normal

print Variational().layers
Variational().add(Normal(2))
print Variational().layers

prints

[]
[<edward.models.variationals.Normal instance at 0x117cc0680>]

[coveralls]

Coverage remained the same at 51.845% when pulling c7ead3b on kudkudak:master into 85f833d on blei-lab:master.

[dustinvtran]

Thanks! Looks like there's still an error for that issue though:

iter 0 loss -3.71
shape:
[ 0.90207928]
scale:
[ 0.07558481]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-8bceaa96d93e> in <module>()
     22 
     23 inference = ed.MFVI(model, variational, data)
---> 24 inference.run(n_iter=500)

/Users/dvt/Dropbox/github/edward/edward/inferences.pyc in run(self, *args, **kwargs)
     65         self.initialize(*args, **kwargs)
     66         for t in range(self.n_iter):
---> 67             loss = self.update()
     68             self.print_progress(t, loss)
     69 

/Users/dvt/Dropbox/github/edward/edward/inferences.pyc in update(self)
    154     def update(self):
    155         sess = get_session()
--> 156         feed_dict = self.variational.np_sample(self.samples, self.n_minibatch)
    157         _, loss = sess.run([self.train, self.loss], feed_dict)
    158         return loss

/Users/dvt/Dropbox/github/edward/edward/models/variationals.py in np_sample(self, samples, size)
     93         for sample,layer in zip(samples, self.layers):
     94             if sample.name.startswith('Placeholder'):
---> 95                 feed_dict[sample] = layer.sample(size)
     96 
     97         return feed_dict

/Users/dvt/Dropbox/github/edward/edward/models/variationals.py in sample(self, size)
    356         z = np.zeros((size, self.num_vars))
    357         for d in range(self.num_vars):
--> 358             z[:, d] = beta.rvs(a[d], b[d], size=size)
    359 
    360         return z

/Users/dvt/Dropbox/github/edward/edward/stats/distributions.pyc in rvs(self, a, b, size)
     84 class Beta:
     85     def rvs(self, a, b, size=1):
---> 86         return stats.beta.rvs(a, b, size=size)
     87 
     88     def logpdf(self, x, a, b):

/Users/dvt/Envs/venv/lib/python2.7/site-packages/scipy/stats/_distn_infrastructure.pyc in rvs(self, *args, **kwds)
    854         cond = logical_and(self._argcheck(*args), (scale >= 0))
    855         if not np.all(cond):
--> 856             raise ValueError("Domain error in arguments.")
    857 
    858         # self._size is total size of all output values

ValueError: Domain error in arguments.

[kudkudak]

@dustinvtran could you please share code to reproduce it? On my machine both beta_bernoulli_tf.py and normal_two.py work.

[dustinvtran]

Here's the source for the jupyter notebook I tried. If I run the code block twice I get the error with your fix. (Without the fix I get the error reported in #74).

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter 0 loss -3.71\n",
      "shape:\n",
      "[ 0.90207928]\n",
      "scale:\n",
      "[ 0.07558481]\n"
     ]
    },
    {
     "ename": "ValueError",
     "evalue": "Domain error in arguments.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-2-a6cbf5b79b8d>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     22\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     23\u001b[0m \u001b[0minference\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0med\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mMFVI\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvariational\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 24\u001b[0;31m \u001b[0minference\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn_iter\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m300\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/Users/dvt/Dropbox/github/edward/edward/inferences.pyc\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     65\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minitialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     66\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mn_iter\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 67\u001b[0;31m             \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     68\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprint_progress\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mloss\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     69\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/dvt/Dropbox/github/edward/edward/inferences.pyc\u001b[0m in \u001b[0;36mupdate\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    154\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    155\u001b[0m         \u001b[0msess\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_session\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 156\u001b[0;31m         \u001b[0mfeed_dict\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvariational\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnp_sample\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msamples\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mn_minibatch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    157\u001b[0m         \u001b[0m_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msess\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloss\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    158\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mloss\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/dvt/Dropbox/github/edward/edward/models/variationals.py\u001b[0m in \u001b[0;36mnp_sample\u001b[0;34m(self, samples, size)\u001b[0m\n\u001b[1;32m     93\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0msample\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mlayer\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msamples\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlayers\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     94\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0msample\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstartswith\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Placeholder'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 95\u001b[0;31m                 \u001b[0mfeed_dict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0msample\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlayer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msample\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     96\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     97\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/dvt/Dropbox/github/edward/edward/models/variationals.py\u001b[0m in \u001b[0;36msample\u001b[0;34m(self, size)\u001b[0m\n\u001b[1;32m    356\u001b[0m         \u001b[0mz\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnum_vars\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    357\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0md\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnum_vars\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 358\u001b[0;31m             \u001b[0mz\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0md\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbeta\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrvs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0md\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mb\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0md\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msize\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    359\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    360\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mz\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/dvt/Dropbox/github/edward/edward/stats/distributions.pyc\u001b[0m in \u001b[0;36mrvs\u001b[0;34m(self, a, b, size)\u001b[0m\n\u001b[1;32m     84\u001b[0m \u001b[0;32mclass\u001b[0m \u001b[0mBeta\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     85\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mrvs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mb\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msize\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 86\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mstats\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbeta\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrvs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mb\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msize\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     87\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     88\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mlogpdf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/dvt/Envs/venv/lib/python2.7/site-packages/scipy/stats/_distn_infrastructure.pyc\u001b[0m in \u001b[0;36mrvs\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m    854\u001b[0m         \u001b[0mcond\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlogical_and\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_argcheck\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mscale\u001b[0m \u001b[0;34m>=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    855\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mall\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcond\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 856\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Domain error in arguments.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    857\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    858\u001b[0m         \u001b[0;31m# self._size is total size of all output values\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: Domain error in arguments."
     ]
    }
   ],
   "source": [
    "import edward as ed\n",
    "import tensorflow as tf\n",
    "\n",
    "from edward.models import Variational, Beta\n",
    "from edward.stats import bernoulli, beta\n",
    "\n",
    "class BetaBernoulli:\n",
    "    \"\"\"\n",
    "    p(x, z) = Bernoulli(x | z) * Beta(z | 1, 1)\n",
    "    \"\"\"\n",
    "    def log_prob(self, xs, zs):\n",
    "        log_prior = beta.logpdf(zs, a=1.0, b=1.0)\n",
    "        log_lik = tf.pack([tf.reduce_sum(bernoulli.logpmf(xs, z))\n",
    "                           for z in tf.unpack(zs)])\n",
    "        return log_lik + log_prior\n",
    "\n",
    "ed.set_seed(42)\n",
    "model = BetaBernoulli()\n",
    "variational = Variational()\n",
    "variational.add(Beta())\n",
    "data = ed.Data(tf.constant((0, 1, 0, 0, 0, 0, 0, 0, 0, 1), dtype=tf.float32))\n",
    "\n",
    "inference = ed.MFVI(model, variational, data)\n",
    "inference.run(n_iter=300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}

[kudkudak]

@dustinvtran It seems it is Python3 issue (it works fine on Python2). I will look into it

[kudkudak]

@dustinvtran So it seems it is stability issue, for instance seed=100 works well. So to fix it, clearly we need:

1. Look into stability of the example, any ideas?
2. Make seeding work properly (clearly it doesn't as we get different results each run, also one can check that by adding variational.layers[0].print_params() just before running inference to see initialization values)

About seeding: currently code is globally seeded, which is often considered a bad practice (for instance user can accidentally reseed code), but I assume this should be left unchanged for now. The core of the problem is that when single session is shared, seeding of tf.random_normal_initializer is quite counterintuitive:

ed.set_seed(42)
z = tf.random_normal_initializer()
init_op = z([1]) # shape [1] random tensor 
ed.get_session().run(init_op)

prints different number each time we are in the same session (for more information see https://www.tensorflow.org/versions/r0.8/api_docs/python/constant_op.html#set_random_seed). This is used in https://github.com/blei-lab/edward/blob/master/edward/util.py#L214.

So best solution would be to create new session each time in inference.initialize. There are many hacks around it, like storing initial values in a separate dict in VarStoreMethod and refreshing them on ed.set_seed() and somehow assigning them during execution of tf.initialize_all_variables() op. What would you suggest?

Btw. sorry for not adhering to the PR formatting policy - noticed it just now.

[dustinvtran]

Thanks for spotting this. So it sounds like there are a few issues. They all revolve around the fact that when a notebook runs a code block again, the Python environment is the same, and currently as a product of Edward's internals, the TensorFlow session is the same. This causes downstream issues such as

1. variational re-using a mutable object (fixed here;
2. set_seed not working within the same TensorFlow session (the technical details with TensorFlow op seeds, although it currently works as intended when re-running Python scripts from scratch;
3. other internals that are mysteriously sharing things when running the code block and thus causing the domain error I showed above.

Having the same TensorFlow session is ideal across the same Python environment and inference algorithms so we shouldn't change this. We could have set_seed work at the ops-level as well, that is, we also explicitly set the seed for all random TensorFlow ops and not only the global TensorFlow seed and NumPy seed. And we also have to find out the other internals that involve mishaps as it seems to not only be variational and set_seed?

[kudkudak]

Thanks for the reply.

We could have set_seed work at the ops-level as well, that is, we also explicitly set the seed for all random TensorFlow ops and not only the global TensorFlow seed and NumPy seed. And we also have to find out the other internals that involve mishaps as it seems to not only be variational and set_seed?

Yup, this is more or less what I meant. I will try to come up with something, but just to warn, I am afraid there is no "elegant" solution here :)

Minor comment:

other internals that are mysteriously sharing things when running the code block and thus causing the domain error I showed above.

This is just matter of instability - so it is a separate issue as well. I am not sure how to fix it. For instance for seed=0 even the first run prodcues Domain error (caused by np.nan passed to scipy.stats).

[dustinvtran]

Sounds great.

I'm not sure I understand the instability comment though (or the Python 3 comment). For example, with the Python script

#!/usr/bin/env python
import edward as ed
import tensorflow as tf

from edward.models import Variational, Beta
from edward.stats import bernoulli, beta

class BetaBernoulli:
    """
    p(x, z) = Bernoulli(x | z) * Beta(z | 1, 1)
    """
    def log_prob(self, xs, zs):
        log_prior = beta.logpdf(zs, a=1.0, b=1.0)
        log_lik = tf.pack([tf.reduce_sum(bernoulli.logpmf(xs, z))
                           for z in tf.unpack(zs)])
        return log_lik + log_prior

ed.set_seed(0)
model = BetaBernoulli()
variational = Variational()
variational.add(Beta())
data = ed.Data(tf.constant((0, 1, 0, 0, 0, 0, 0, 0, 0, 1), dtype=tf.float32))

inference = ed.MFVI(model, variational, data)
inference.run(n_iter=500)

I run it in on Python 2.7.10 and TensorFlow 0.8.0, and it always produces the same output (only the address for the function initializer changes):

INFO:tensorflow:Created variable alpha:0 with shape (1,) and init <function _initializer at 0x101344b18>
INFO:tensorflow:Created variable alpha:0 with shape (1,) and init <function _initializer at 0x101344b18>
INFO:tensorflow:Created variable beta:0 with shape (1,) and init <function _initializer at 0x101344b18>
INFO:tensorflow:Created variable beta:0 with shape (1,) and init <function _initializer at 0x101344b18>
iter 0 loss -6.56
shape:
[ 0.28207624]
scale:
[ 0.96621323]
iter 100 loss -8.06
shape:
[ 0.39867195]
scale:
[ 1.25165844]
iter 200 loss -6.52
shape:
[ 0.67361927]
scale:
[ 1.5545224]
iter 300 loss -5.40
shape:
[ 0.69744116]
scale:
[ 2.3602047]
iter 400 loss -6.04
shape:
[ 0.95770347]
scale:
[ 2.53179455]

[dustinvtran]

Epsilon-bounds to stay within open intervals (e.g., (0, infinity) instead of [0, infinity)) is something we should definitely do though, and it's not currently implemented. In fact, I will add this as an issue.

[kudkudak]

(or the Python 3 comment)

Yup, the Python3 comment wasn't really accurate. It just didn't happen for that seed under Python2.

I'm not sure I understand the instability comment though

Ups, sorry. I had a slight difference in my script. Changing the seed to 12 in your code makes it diverge (in Python3, not Python2).

[dustinvtran]

Okay gotcha; with set_seed(12) it also diverges in Python 2.7. :)

[dustinvtran]

Okay I was able to fix the internal copy of the layers object across classes in 09bf835. (see also related stackoverflow)

from edward.models import Variational
from edward.models import Normal

print(Variational().layers)
Variational().add(Normal(2))
print(Variational().layers)

prints

[]
[]

as expected.

1. variational re-using a mutable object (fixed here;
2. set_seed not working within the same TensorFlow session (the technical details with TensorFlow op seeds, although it currently works as intended when re-running Python scripts from scratch;
3. other internals that are mysteriously sharing things when running the code block and thus causing the domain error I showed above.

So this fixes 1. The latter two are still open.

[kudkudak]

Sorry, I had a really busy week! I will do my best to look at this during the weekend.

Btw, this PR already introduced fix for 1 :)

[dustinvtran]

No problem. Also yes, the commit above was based on yours. :)