Release 2.0.2 (#1459)

* add long_description and project_urls to setup.py (#1438)
* add type hints for probability distributions (#1421)
* remove comment (#1441)
* Use a type alias in the fnction signature of leading_transpose (#1442)
* refactor natgrads to be more efficient (#1443)
* Fix dimensions of kernel evaluation of changepoint kernel (#1446)
* Removed unusued imports. (#1450)
* Improve representation of GPflow objects in IPython/Jupyter notebook (#1453)
* includes the repr() string in IPython/Jupyter notebook representation as well (i.e. fully-qualified class name and object hash (memory address), which helps distinguish objects from each other)
* only displays the parameter table when it is not empty
* makes use of default_summary_fmt() for IPython shell
* Convert data structures to tensor in model init method (#1452)
* Use a boolean for full covariance in sample_mvn. (#1448)
* #1452 for GPMC model (#1458)
* release candidate v2.0.2 (#1457)

Co-authored-by: st-- <st--@users.noreply.github.com>
Co-authored-by: joelberkeley-pio <joel.berkeley@prowler.io>
Co-authored-by: John Mcleod <43960404+johnamcleod@users.noreply.github.com>
Co-authored-by: Mark van der Wilk <markvanderw@gmail.com>
Co-authored-by: Artem Artemev <art.art.v@gmail.com>

.gitignore

@@ -70,3 +70,6 @@ target/
 
 # OSX
 .DS_Store
+
+# mypy artifacts
+.mypy_cache

RELEASE.md

@@ -1,138 +1 @@
-# Release 1.2.0
-
-- Added `SoftMax` likelihood (#799)
-- Added likelihoods where expectations are evaluated with Monte Carlo, `MonteCarloLikelihood` (#799)
-- GPflow monitor refactoring, check `monitor-tensorboard.ipynb` for details (#792)
-- Speedup testing on Travis using utility functions for configuration in notebooks (#789)
-- Support Python 3.5.2 in typing checks (Ubuntu 16.04 default python3) (#787)
-- Corrected scaling in Students-t likelihood variance (#777)
-- Removed jitter before taking the cholesky of the covariance in NatGrad optimizer (#768)
-- Added GPflow logger. Created option for setting logger level in `gpflowrc` (#764)
-- Fixed bug at `params_as_tensors_for` (#751)
-- Fixed GPflow SciPy optimizer to pass options to _actual_ scipy optimizer correctly (#738)
-- Improved quadrature for likelihoods. Unified quadrature method introduced - `ndiagquad` (#736), (#747)
-- Added support for multi-output GPs, check `multioutput.ipynb` for details (#724)
-    * Multi-output features
-    * Multi-output kernels
-    * Multi-dispatch for conditional
-    * Multi-dispatch for Kuu and Kuf
-- Support Exponential distribution as prior (#717)
-- Added notebook to demonstrate advanced usage of GPflow, such as combining GP with Neural Network (#712)
-- Minibatch shape is `None` by default to allow dynamic change of data size (#704)
-- Epsilon parameter of the Robustmax likelihood is trainable now (#635)
-- GPflow model saver (#660)
-    * Supports native GPflow models and provides an interface for defining custom savers for user's models
-    * Saver stores GPflow structures and pythonic types as numpy structured arrays and serializes them using HDF5
-
-# Release 1.1
- - Added inter-domain inducing features. Inducing points are used by default and are now set with `model.feature.Z`.
-
-# Release 1.0
-* Clear and aligned with tree-like structure of GPflow models design.
-* GPflow trainable parameters are no longer packed into one TensorFlow variable.
-* Integration of bare TensorFlow and Keras models with GPflow became very simple.
-* GPflow parameter wraps multiple tensors: unconstained variable, constrained tensor and prior tensor.
-* Instantaneous parameter's building into the TensorFlow graph. Once you created an instance of parameter, it creates necessary tensors at default graph immediately.
-* New implementation for AutoFlow. `autoflow` decorator is a replacement.
-* GPflow optimizers match TensorFlow optimizer names. For e.g. `gpflow.train.GradientDescentOptimizer` mimics `tf.train.GradientDescentOptimizer`. They even has the same instantialization signature.
-* GPflow has native support for Scipy optimizers - `gpflow.train.ScipyOptimizer`.
-* GPflow has advanced HMC implementation - `gpflow.train.HMC`. It works only within TensorFlow memory scope.
-* Tensor conversion decorator and context manager designed for cases when user needs to implicitly convert parameters to TensorFlow tensors: `gpflow.params_as_tensors` and `gpflow.params_as_tensors_for`.
-* GPflow parameters and parameterized objects provide convenient methods and properties for building, intializing their tensors. Check `initializables`, `initializable_feeds`, `feeds` and other properties and methods.
-* Floating shapes of parameters and dataholders without re-building TensorFlow graph.
-
-# Release 0.5
- - bugfix for log_jacobian in transforms
-
-# Release 0.4.1
- - Different variants of `gauss_kl_*` are now deprecated in favour of a unified `gauss_kl` implementation
-
-# Release 0.4.0
- - Rename python package name to `gpflow`.
- - Compile function has external session and graph arguments.
- - Tests use Tensorflow TestCase class for proper session managing.
-
-# Release 0.3.8
- - Change to LowerTriangular transform interface.
- - LowerTriangular transform now used by default in VGP and SVGP
- - LowerTriangular transform now used native TensorFlow
- - No longer use bespoke GPflow user ops.
-
-# Release 0.3.7
- - Improvements to VGP class allow more straightforward optimization
-
-# Release 0.3.6
- - Changed ordering of parameters to be alphabetical, to ensure consistency
-
-# Release 0.3.5
- - Update to work with TensorFlow 0.12.1.
-
-# Release 0.3.4
- - Changes to stop computations all being done on the default graph.
- - Update list of GPflow contributors and other small changes to front page.
- - Better deduction of `input_dim` for `kernels.Combination`
- - Some kernels did not properly respect active dims, now fixed.
- - Make sure log jacobian is computed even for fixed variables
-
-# Release 0.3.3
- - House keeping changes for paper submission.
-
-# Release 0.3.2
- - updated to work with tensorflow 0.11 (release candidate 1 available at time of writing)
- - bugfixes in vgp._compile
-
-# Release 0.3.1
- - Added configuration file, which controls verbosity and level of numerical jitter
- - tf_hacks is deprecated, became tf_wraps (tf_hacks will raise visible deprecation warnings)
- - Documentation now at gpflow.readthedocs.io
- - Many functions are now contained in tensorflow scopes for easier tensorboad visualisation and profiling
-
-# Release 0.3
- - Improvements to the way that parameters for triangular matrices are stored and optimised.
- - Automatically generated Apache license headers.
- - Ability to track log probabilities.
-
-# Release 0.2
- - Significant improvements to the way that data and fixed parameters are handled.
-
-Previously, data and fixed parameters were treated as tensorflow constants.
-Now, a new mechanism called `get_feed_dict()` can gather up data and and fixed
-parameters and pass them into the graph as placeholders.
-
- - To enable the above, data are now stored in objects called `DataHolder`. To
-   access values of the data, use the same syntax as parameters:
-   `print(m.X.value)`
- - Models do not need to be recompiled when the data changes.
- - Two models, VGP and GPMC, do need to be recompiled if the *shape* of the data changes
-
- - A multi-class likelihood is implemented
-
-
-
-# Release 0.1.4
- - Updated to work with tensorflow 0.9
- - Added a Logistic transform to enable contraining a parameter between two bounds
- - Added a Laplace distribution to use as a prior
- - Added a periodic kernel
- - Several improvements to the AutoFlow mechanism
- - added FITC approximation (see comparison notebook)
- - improved readability of code according to pep8
- - significantly improved the speed of the test suite
- - allowed passing of the 'tol' argument to scipy.minimize routine
- - added ability to add and multiply MeanFunction objects
- - Several new contributors (see README.md)
-
-# Release 0.1.3
- - Removed the need for a fork of TensorFlow. Some of our bespoke ops are replaced by equivalent versions.
-
-# Release 0.1.2
- - Included the ability to compute the full covaraince matrix at predict time. See `GPModel.predict_f`
- - Included the ability to sample from the posterior function values. See `GPModel.predict_f_samples`
- - Unified code in conditionals.py: see deprecations in `gp_predict`, etc.
- - Added SGPR method (Sparse GP Regression)
-
-# Release 0.1.1
- -  included the ability to use tensorflow's optimizers as well as the scipy ones
-
-# Release 0.1.0
-The initial release of GPflow.
+The release notes have been moved to the ['Releases' section](https://github.com/GPflow/GPflow/releases) of the GPflow GitHub Repo

VERSION

@@ -1 +1 @@
-2.0.1
+2.0.2

doc/source/conf.py

@@ -92,9 +92,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = "2.0.1"
+version = "2.0.2"
 # The full version, including alpha/beta/rc tags.
-release = "2.0.1"
+release = "2.0.2"
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

doc/source/notebooks/advanced/ordinal_regression.pct.py

@@ -83,7 +83,7 @@ def generate_data(num_data):
 # %%
 # here we'll plot the expected value of Y +- 2 std deviations, as if the distribution were Gaussian
 plt.figure(figsize=(11, 6))
-X_data, Y_data = m.data
+X_data, Y_data = (m.data[0].numpy(), m.data[1].numpy())
 Xtest = np.linspace(X_data.min(), X_data.max(), 100).reshape(-1, 1)
 mu, var = m.predict_y(Xtest)
 (line,) = plt.plot(Xtest, mu, lw=2)

gpflow/base.py

@@ -6,14 +6,12 @@
 import tensorflow as tf
 import tensorflow_probability as tfp
 from tensorflow.python.ops import array_ops
+from typing_extensions import Final
 
-from .config import default_float
+from .config import default_float, default_summary_fmt
 
 DType = Union[np.dtype, tf.DType]
 VariableData = Union[List, Tuple, np.ndarray, int, float]
-TensorLike = (
-    object  # Union[tf.Tensor, tf.Variable, np.ndarray], but doesn't work with multipledispatch
-)
 Transform = Union[tfp.bijectors.Bijector]
 Prior = Union[tfp.distributions.Distribution]
 
@@ -35,15 +33,24 @@ def parameters(self):
     def trainable_parameters(self):
         return tuple(self._flatten(predicate=_IS_TRAINABLE_PARAMETER))
 
+    def _representation_table(self, object_name, tablefmt):
+        from .utilities import leaf_components, tabulate_module_summary
+
+        repr_components = [object_name]
+        if leaf_components(self):
+            repr_components.append(tabulate_module_summary(self, tablefmt=tablefmt))
+        return "\n".join(repr_components)
+
     def _repr_html_(self):
-        from .utilities import tabulate_module_summary
+        """ Nice representation of GPflow objects in IPython/Jupyter notebooks """
+        from html import escape
 
-        return tabulate_module_summary(self, tablefmt="html")
+        return self._representation_table(escape(repr(self)), "html")
 
     def _repr_pretty_(self, p, cycle):
-        from .utilities import tabulate_module_summary
-
-        p.text(tabulate_module_summary(self, tablefmt=""))
+        """ Nice representation of GPflow objects in the IPython shell """
+        repr_str = self._representation_table(repr(self), default_summary_fmt())
+        p.text(repr_str)
 
 
 class PriorOn(Enum):
@@ -307,6 +314,22 @@ def _run_op(a, *args, **kwargs):
 Parameter._OverloadAllOperators()
 tf.register_tensor_conversion_function(Parameter, lambda x, *args, **kwds: x.read_value())
 
+TensorType = Union[np.ndarray, tf.Tensor, tf.Variable, Parameter]
+"""
+Type alias for tensor-like types that are supported by most TensorFlow, NumPy and GPflow operations.
+
+NOTE: Union types like this do not work with the `register` method of multipledispatch's
+`Dispatcher` class. Instead use `TensorLike` for dispatching on tensor-like types.
+"""
+
+# We've left this as object until we've tested the performance consequences of using the full set
+# (np.ndarray, tf.Tensor, tf.Variable, Parameter), see https://github.com/GPflow/GPflow/issues/1434
+TensorLike: Final[Tuple[type, ...]] = (object,)
+"""
+:var TensorLike: Collection of tensor-like types for registering implementations with
+    `multipledispatch` dispatchers.
+"""
+
 
 def _cast_to_dtype(value: VariableData, dtype: Optional[DType] = None) -> tf.Tensor:
     if dtype is None:

gpflow/conditionals/multioutput/sample_conditionals.py

@@ -43,7 +43,7 @@ def _sample_conditional(
     g_mu, g_var = ind_conditional(
         Xnew, inducing_variable, kernel, f, white=white, q_sqrt=q_sqrt
     )  # [..., N, L], [..., N, L]
-    g_sample = sample_mvn(g_mu, g_var, "diag", num_samples=num_samples)  # [..., (S), N, L]
+    g_sample = sample_mvn(g_mu, g_var, full_cov, num_samples=num_samples)  # [..., (S), N, L]
     f_mu, f_var = mix_latent_gp(kernel.W, g_mu, g_var, full_cov, full_output_cov)
     f_sample = tf.tensordot(g_sample, kernel.W, [[-1], [-1]])  # [..., N, P]
     return f_sample, f_mu, f_var

gpflow/conditionals/sample_conditionals.py

@@ -50,13 +50,14 @@ def _sample_conditional(
         # cov: [..., P, N, N]
         mean_for_sample = tf.linalg.adjoint(mean)  # [..., P, N]
         samples = sample_mvn(
-            mean_for_sample, cov, "full", num_samples=num_samples
+            mean_for_sample, cov, full_cov=True, num_samples=num_samples
         )  # [..., (S), P, N]
         samples = tf.linalg.adjoint(samples)  # [..., (S), N, P]
     else:
         # mean: [..., N, P]
         # cov: [..., N, P] or [..., N, P, P]
-        cov_structure = "full" if full_output_cov else "diag"
-        samples = sample_mvn(mean, cov, cov_structure, num_samples=num_samples)  # [..., (S), N, P]
+        samples = sample_mvn(
+            mean, cov, full_cov=full_output_cov, num_samples=num_samples
+        )  # [..., (S), N, P]
 
     return samples, mean, cov

gpflow/conditionals/util.py

@@ -134,22 +134,19 @@ def base_conditional(
     return fmean, fvar
 
 
-def sample_mvn(mean, cov, cov_structure=None, num_samples=None):
+def sample_mvn(mean, cov, full_cov, num_samples=None):
     """
     Returns a sample from a D-dimensional Multivariate Normal distribution
     :param mean: [..., N, D]
     :param cov: [..., N, D] or [..., N, D, D]
-    :param cov_structure: "diag" or "full"
-    - "diag": cov holds the diagonal elements of the covariance matrix
+    :param full_cov: if `True` return a "full" covariance matrix, otherwise a "diag":
     - "full": cov holds the full covariance matrix (without jitter)
+    - "diag": cov holds the diagonal elements of the covariance matrix
     :return: sample from the MVN of shape [..., (S), N, D], S = num_samples
     """
-    if cov_structure not in ("diag", "full"):
-        raise ValueError("cov_structure must be 'diag' or 'full'")
-
     shape_constraints = [
         (mean, [..., "N", "D"]),
-        (cov, [..., "N", "D"] if cov_structure == "diag" else [..., "N", "D", "D"]),
+        (cov, [..., "N", "D", "D"] if full_cov else [..., "N", "D"]),
     ]
     tf.debugging.assert_shapes(shape_constraints, message="sample_mvn() arguments")
 
@@ -158,13 +155,13 @@ def sample_mvn(mean, cov, cov_structure=None, num_samples=None):
     D = mean_shape[-1]
     leading_dims = mean_shape[:-2]
 
-    if cov_structure == "diag":
+    if not full_cov:
         # mean: [..., N, D] and cov [..., N, D]
         eps_shape = tf.concat([leading_dims, [S], mean_shape[-2:]], 0)
         eps = tf.random.normal(eps_shape, dtype=default_float())  # [..., S, N, D]
         samples = mean[..., None, :, :] + tf.sqrt(cov)[..., None, :, :] * eps  # [..., S, N, D]
 
-    elif cov_structure == "full":
+    else:
         # mean: [..., N, D] and cov [..., N, D, D]
         jittermat = (
             tf.eye(D, batch_shape=mean_shape[:-1], dtype=default_float()) * default_jitter()

gpflow/covariances/kufs.py

@@ -1,4 +1,5 @@
 import tensorflow as tf
+
 from ..base import TensorLike
 from ..inducing_variables import InducingPoints, Multiscale, InducingPatches
 from ..kernels import Kernel, SquaredExponential, Convolutional

gpflow/kernels/changepoints.py

@@ -70,18 +70,19 @@ def _set_kernels(self, kernels: List[Kernel]):
         self.kernels = kernels
 
     def K(self, X: tf.Tensor, X2: Optional[tf.Tensor] = None) -> tf.Tensor:
-        sig_X = self._sigmoids(X)  # N x 1 x Ncp
-        sig_X2 = self._sigmoids(X2) if X2 is not None else sig_X
+        sig_X = self._sigmoids(X)  # N1 x 1 x Ncp
+        sig_X2 = self._sigmoids(X2) if X2 is not None else sig_X  # N2 x 1 x Ncp
 
         # `starters` are the sigmoids going from 0 -> 1, whilst `stoppers` go
-        # from 1 -> 0, dimensions are N x N x Ncp
+        # from 1 -> 0, dimensions are N1 x N2 x Ncp
         starters = sig_X * tf.transpose(sig_X2, perm=(1, 0, 2))
         stoppers = (1 - sig_X) * tf.transpose((1 - sig_X2), perm=(1, 0, 2))
 
         # prepend `starters` with ones and append ones to `stoppers` since the
         # first kernel has no start and the last kernel has no end
-        N = tf.shape(X)[0]
-        ones = tf.ones((N, N, 1), dtype=X.dtype)
+        N1 = tf.shape(X)[0]
+        N2 = tf.shape(X2)[0] if X2 is not None else N1
+        ones = tf.ones((N1, N2, 1), dtype=X.dtype)
         starters = tf.concat([ones, starters], axis=2)
         stoppers = tf.concat([stoppers, ones], axis=2)
 

gpflow/models/gplvm.py

@@ -28,9 +28,9 @@
 from ..utilities import positive, to_default_float
 from ..utilities.ops import pca_reduce
 from .gpr import GPR
-from .model import InputData, OutputData, GPModel, MeanAndVariance
-from .training_mixins import InternalDataTrainingLossMixin
-from .util import inducingpoint_wrapper
+from .model import GPModel, MeanAndVariance
+from .training_mixins import InputData, InternalDataTrainingLossMixin, OutputData
+from .util import data_input_to_tensor, inducingpoint_wrapper
 
 
 class GPLVM(GPR):
@@ -72,7 +72,7 @@ def __init__(
         if data.shape[1] < num_latent_gps:
             raise ValueError("More latent dimensions than observed.")
 
-        gpr_data = (Parameter(X_data_mean), data)
+        gpr_data = (Parameter(X_data_mean), data_input_to_tensor(data))
         super().__init__(gpr_data, kernel, mean_function=mean_function)
 
 
@@ -103,18 +103,18 @@ def __init__(
         """
         num_data, num_latent_gps = X_data_mean.shape
         super().__init__(kernel, likelihoods.Gaussian(), num_latent_gps=num_latent_gps)
-        self.data = data
+        self.data = data_input_to_tensor(data)
         assert X_data_var.ndim == 2
 
         self.X_data_mean = Parameter(X_data_mean)
         self.X_data_var = Parameter(X_data_var, transform=positive())
 
         self.num_data = num_data
-        self.output_dim = data.shape[-1]
+        self.output_dim = self.data.shape[-1]
 
         assert np.all(X_data_mean.shape == X_data_var.shape)
-        assert X_data_mean.shape[0] == data.shape[0], "X mean and Y must be same size."
-        assert X_data_var.shape[0] == data.shape[0], "X var and Y must be same size."
+        assert X_data_mean.shape[0] == self.data.shape[0], "X mean and Y must be same size."
+        assert X_data_var.shape[0] == self.data.shape[0], "X var and Y must be same size."
 
         if (inducing_variable is None) == (num_inducing_variables is None):
             raise ValueError(