Edward Restructuring

[bhargavvader]

Hello, when I try and redo the getting-started tutorial using the latest github version of the code, it seems as if ed.MFVI behaves very differently. Is the current release of edward on PIP and the bleeding-edge version different for this?

The model learned now doesn't learn the relationship. I'm sorry if I missed out on something - changes happen very fast on Edward!

Edit: Noticed that MVFI doesn't exist in the inference.py file. Where is it right now? And is it different in it's implementation from the older code?

Second edit: Noticed the massive restructuring - but still don't know where MVFI is.

[bhargavvader]

@dustinvtran
Are all the tutorials updated to the latest version of Edward?
Or are they all examples for the PIP version?

Edit: My motivation for these questions is to better organise the future python notebooks and to redo the existing tutorials and example code. If there is going to be a big change from the current version in terms of organisation, this should start being reflected in the examples - unless there are going to be many more changes until the next release!

[bhargavvader]

I've changed the name of the issue to better represent my doubt.

My main questions are:

1. I know Edward has been undergoing lots of changes in it's directory structure - is there a CHANGELOG or an issue raised which documents all of this? Especially in the differences between the version out on PIP and the bleeding edge version.

2. Are the tutorials and example code going to be changed accordingly? It seems to me that using the latest Edward for the getting_started code, for example, is giving me completely different results. Is this because of MFVI or something else? I am not making any changes in the notebook, and running it with different Edward installations gives me very different results.

Thanks a lot for the patience to hear this out - it'll help me further contribute, and am sure will help others as well.

Edit: Found #249, but am not very sure why just running MVFI (as it is a synonym) is not giving me the same results as before.

[dustinvtran]

The website is updated according to the latest development version of Edward and not the latest stable version.

This is because during development we update scripts in the examples/ directory (and now also the notebooks). This in turn breaks compatibility if users follow a link to it from the website, and the rest of the website displays the stable version. (One solution would be to have multiple branches to display for the website, analogous to TensorFlow. Although that requires infrastructure.)

1. There is not a changelog; the commit and pull request histories suffice.
2. For now, I recommend ensuring that the notebooks work on the development version and not worrying about what happens in the stable version.

Note this is all bad practice. We have no guarantees that any code from the latest stable version also works on the latest development version. Edward's infrastructure is in flux, as we're solving open research challenges in the design and implementation of probabilistic programming languages. (But we're slowly but surely converging!)

[bhargavvader]

@dustinvtran, thanks for the reply!
I'm still confused about the behaviour of MVFI though. Any thoughts as to why it does not work post PR #249 ?

[dustinvtran]

Sure. I tried it on a few machines (both Linux and Mac, and on GPU vs CPU) and it does work. Can you provide more details?

[bhargavvader]

This is what the plotting looks like after I've run the notebook.

[bhargavvader]

I also occasionally get the ValueError: Incompatible shapes for broadcasting: (50,) and (1, 2) error while running the notebook with the latest version.

[dustinvtran]

Looks like inference wasn't run. Have you tried to restart the kernel and run each block sequentially? Alternatively, you can look at the output from the inference run. In the last several hundred iterations the loss function should be close to zero.

re:ValueError. Those two problems seem correlated, and related to the mysteriousness of #248. Would be good to have a complete minimal example.

[bhargavvader]

Strange things:

1. I tried restarting and running each cell individually 3 times (as in, 3 cycles of restarting and running) before inference.initialize() didn't give me a value error. It seems very random, when it fails and when it doesn't.

2. As you can see, the loss function stops converging after about 50-100 iterations. I've not altered the notebook at all, and running it on an older Edward version it seems fine.

[bhargavvader]

Would also like to note that the plotting of before running inference and after running it in each case is different. So I am running the inference each time, it's just not working.

Also, if you see the current version on the getting-started notebook in the code-base, the last few cells are not run and you cannot see the final plot. I was trying to fix that when I noticed that MVFI was not working on my machine...

[dustinvtran]

Some steps at reproducibility might be helpful. Could you report a seed via ed.set_seed(...), where (from a restarted kernel) running each code block sequentially ends in this behavior?

In general, I can only say that repeated application of certain code blocks will certainly produce bugs. Unlike NumPy code, calling TensorFlow lines multiple times adds more tensors to the computational graph, which can break subsequent code.

[bhargavvader]

Btw, I have made no changes to the getting-started notebook currently in the repo - this sets the seed in row 5, right before the start of the real code.

I've run this notebook multiple times (what I mean by multiple times is that I have restarted the kernel each time and run it).

Each time, I get either of 2 results - either one where row 8 does inference.initialize() and fails with a ValueError, or that when I run inference in row 10 the loss does not go down to zero.

I end up getting the same images I posted before in this thread.
Could this be verified by someone else as well? Because right now the getting-started notebook on the repo is incomplete. The last few cells need to be run.

Never mind, works now :D

[bhargavvader]

Ok, figured where I was going wrong - my installations of Edward were all over the place, and the tensorflow graphs were messing with the results.

Sorry for the trouble!

[dustinvtran]

oh, perfect. glad it was sorted out!