-
Notifications
You must be signed in to change notification settings - Fork 760
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
stable gaussian process classification with global kernel #316
Conversation
cool! could we have a version of this that uses the modeling language? currently, the |
i added a |
@dustinvtran : yes this will be helpful. Thx! I was having "cannot copy z" errors with Edward language...So I will submit the initial code I have that do not use Edward language as tf_gpc.py and add yours as gpc.py into one PR. Will get back to you this weekend. |
@dustinvtran : this is ready to merge... |
edward/util/tensorflow.py
Outdated
@@ -310,6 +310,44 @@ def multivariate_rbf(x, y=0.0, sigma=1.0, l=1.0): | |||
tf.exp(-1.0 / (2.0 * tf.pow(l, 2.0)) * tf.reduce_sum(tf.pow(x - y, 2.0))) | |||
|
|||
|
|||
def multivariate_rbf_kernel(x, sigma=1.0, l=1.0): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
when a function is added to the codebase in util
, it needs a unit test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also, in general, do we plan on supporting kernel functions in edward.util
as a long term thing? or do you think it makes sense to leave them as part of the example scripts?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
multivariate_rbf_kernel() might be needed for any model with a RBF kernel such as GP, Cox.
in the long term i think we might want to implement other kernels and not just RBF. Probably not in util but in a kernel.py file as is done in GPflow. Then in the example scripts we can simply do:
from ed.kernel import rbf, linear
K = rbf(x)
K_lin = linear(x)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i will move it to the example scripts for now.
edward/util/tensorflow.py
Outdated
def multivariate_rbf_kernel(x, sigma=1.0, l=1.0): | ||
""" | ||
computes the rbf kernel for the whole data x | ||
Args: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
following our convention, can we convert this docstring style to be NumPy?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sure.
edward/util/tensorflow.py
Outdated
""" | ||
N = x.get_shape()[0] | ||
mat = [] | ||
for i in range(N): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is there any way we can convert the multivariate rbf to be vectorized in some way, and not have to loop over every entry in the matrix? i've found this to be the computational bottleneck in our GP experiments.
looking at GPflow's kernels.py
, i couldn't find out how (or if) they vectorize this operation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes i agree. i think this is what is slowing it down. GPflow seems to be using a custom _slice() method on their input. I will have to look into that. they don't use a loop.
examples/gpc.py
Outdated
X_train = df[:, 1:][permutation] | ||
y_train = df[:, 0][permutation] | ||
|
||
print("pre-computing the kernel matrix...") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you elaborate on what you mean by "pre-computing the kernel matrix"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i meant compute K externally and not inside the model definition...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
got it. i guess that makes sense for the model wrapper. everything is already "pre-computed" in the native language.
examples/tf_gpc.py
Outdated
y = df[:, 0][permutation] | ||
|
||
print("computing the kernel matrix...") | ||
K = multivariate_rbf_kernel( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if we're building this as part of the model, it would be nice to prevent global scoping for classes and have K
be an argument to GaussianProcess
. self.N
and self.n_vars
for example could also be inferred from the shape of K
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes passing K as an argument is also an option. will add that.
Thanks for the update! It looks great. Comments above. |
@dustinvtran : i am not sure why the python3.4 check is failing. this pr is ready once that is sorted out. |
That's a legacy issue which has already been fixed in #324. |
@dawenl : that's great! thanks |
cool! since there's already a |
a5ac146
to
de5c24c
Compare
Improved via #596. |
This provides a more stable GPC code...The changes are the following:
added a kernel() method to the model wrapper that uses the existing multivariate.rbf() method to compute the kernel matrix for the whole data
the kernel has been stabilized by adding a negligible number in its diagonal...
Note: this uses the benchmark crabs dataset from UCI. The data is shuffled...