stable gaussian process classification with global kernel #316
Conversation
|
cool! could we have a version of this that uses the modeling language? currently, the |
|
i added a |
|
@dustinvtran : yes this will be helpful. Thx! I was having "cannot copy z" errors with Edward language...So I will submit the initial code I have that do not use Edward language as tf_gpc.py and add yours as gpc.py into one PR. Will get back to you this weekend. |
|
@dustinvtran : this is ready to merge... |
edward/util/tensorflow.py
Outdated
| @@ -310,6 +310,44 @@ def multivariate_rbf(x, y=0.0, sigma=1.0, l=1.0): | |||
| tf.exp(-1.0 / (2.0 * tf.pow(l, 2.0)) * tf.reduce_sum(tf.pow(x - y, 2.0))) | |||
|
|
|||
|
|
|||
| def multivariate_rbf_kernel(x, sigma=1.0, l=1.0): | |||
There was a problem hiding this comment.
when a function is added to the codebase in util, it needs a unit test.
There was a problem hiding this comment.
also, in general, do we plan on supporting kernel functions in edward.util as a long term thing? or do you think it makes sense to leave them as part of the example scripts?
There was a problem hiding this comment.
multivariate_rbf_kernel() might be needed for any model with a RBF kernel such as GP, Cox.
in the long term i think we might want to implement other kernels and not just RBF. Probably not in util but in a kernel.py file as is done in GPflow. Then in the example scripts we can simply do:
from ed.kernel import rbf, linear
K = rbf(x)
K_lin = linear(x)
There was a problem hiding this comment.
i will move it to the example scripts for now.
edward/util/tensorflow.py
Outdated
| def multivariate_rbf_kernel(x, sigma=1.0, l=1.0): | ||
| """ | ||
| computes the rbf kernel for the whole data x | ||
| Args: |
There was a problem hiding this comment.
following our convention, can we convert this docstring style to be NumPy?
edward/util/tensorflow.py
Outdated
| """ | ||
| N = x.get_shape()[0] | ||
| mat = [] | ||
| for i in range(N): |
There was a problem hiding this comment.
is there any way we can convert the multivariate rbf to be vectorized in some way, and not have to loop over every entry in the matrix? i've found this to be the computational bottleneck in our GP experiments.
looking at GPflow's kernels.py, i couldn't find out how (or if) they vectorize this operation.
There was a problem hiding this comment.
yes i agree. i think this is what is slowing it down. GPflow seems to be using a custom _slice() method on their input. I will have to look into that. they don't use a loop.
examples/gpc.py
Outdated
| X_train = df[:, 1:][permutation] | ||
| y_train = df[:, 0][permutation] | ||
|
|
||
| print("pre-computing the kernel matrix...") |
There was a problem hiding this comment.
Can you elaborate on what you mean by "pre-computing the kernel matrix"?
There was a problem hiding this comment.
i meant compute K externally and not inside the model definition...
There was a problem hiding this comment.
got it. i guess that makes sense for the model wrapper. everything is already "pre-computed" in the native language.
examples/tf_gpc.py
Outdated
| y = df[:, 0][permutation] | ||
|
|
||
| print("computing the kernel matrix...") | ||
| K = multivariate_rbf_kernel( |
There was a problem hiding this comment.
if we're building this as part of the model, it would be nice to prevent global scoping for classes and have K be an argument to GaussianProcess. self.N and self.n_vars for example could also be inferred from the shape of K.
There was a problem hiding this comment.
yes passing K as an argument is also an option. will add that.
|
Thanks for the update! It looks great. Comments above. |
|
@dustinvtran : i am not sure why the python3.4 check is failing. this pr is ready once that is sorted out. |
|
That's a legacy issue which has already been fixed in #324. |
|
@dawenl : that's great! thanks |
|
cool! since there's already a |
dustinvtran
force-pushed
the
master
branch
2 times, most recently
from
a5ac146
to
de5c24c
Compare
|
Improved via #596. |
This provides a more stable GPC code...The changes are the following:
added a kernel() method to the model wrapper that uses the existing multivariate.rbf() method to compute the kernel matrix for the whole data
the kernel has been stabilized by adding a negligible number in its diagonal...
Note: this uses the benchmark crabs dataset from UCI. The data is shuffled...