1.0.0

The objective of this release is to stabilize Pyro's interface and thereby make it safer to build high level components on top of Pyro.

Stability statement

• Behavior of documented APIs will remain stable across minor releases, except for bug fixes and features marked EXPERIMENTAL or DEPRECATED.
• Serialization formats will remain stable across patch releases, but may change across minor releases (e.g. if you save a model in 1.0.0, it will be safe to load it in 1.0.1, but not in 1.1.0).
• Undocumented APIs, features marked EXPERIMENTAL or DEPRECATED, and anything inpyro.contrib may change at any time (though we aim for stability).
• All deprecated features throw a FutureWarning and specify possible work-arounds. Features marked as deprecated will not be maintained, and are likely to be removed in a future release.
• If you want more stability for a particular feature, contribute a unit test.

New features

• pyro.infer.Predictive is a new utility for serving models, supporting jit tracing and serialization.
• pyro.distributions.transforms has many new transforms, and includes helper functions to easily create a variety of normalizing flows. The transforms library has also been reorganized.
• pyro.contrib.timeseries is an experimental new module with fast Gaussian Process inference for univariate and multivariate time series and state space models.
• pyro.nn.PyroModule is an experimental new interface that adds Pyro effects to an nn.Module. PyroModule is already used internally by AutoGuide, EasyGuide pyro.contrib.gp, pyro.contrib.timeseries, and elsewhere.
• FoldedDistribution is a new distribution factory, essentially equivalent to TransformedDistribution(-, AbsTransform()) but providing a .log_prob() method.
• A new tutorial illustrates the usage of pyro.contrib.oed in the context of adaptive election polling.

Breaking changes

• Autoguides have slightly changed interfaces:
  • AutoGuide and EasyGuide are now nn.Modules and can be serialized separately from the param store. This enables serving via torch.jit.trace_module.
  • The Auto*Normal family of autoguides now have init_scale arguments, and init_loc_fn has better support. Autoguides no longer support initialization by writing directly to the param store.
• Many transforms have been renamed to enforce a consistent interface, such as the renaming of InverseAutoregressiveFlow to AffineAutoregressive.
• pyro.generic has been moved to a separate project pyroapi.
• poutine.do has slightly changed semantics to follow Single World Intervention Graph semantics.
• pyro.contrib.glmm has been moved to pyro.contrib.oed.glmm and will eventually be replaced by BRMP.
• Existing DeprecationWarnings have been promoted to FutureWarnings.

Deprecated features

• pyro.random_module: The pyro.random_module primitive has been deprecated in favor of PyroModule which can be used to create Bayesian modules from torch.nn.Module instances.
• SVI.run: The SVI.run method is deprecated and users are encouraged to use the .step method directly to run inference. For drawing samples from the posterior distribution, we recommend using the Predictive utility class, or directly by using the trace and replay effect handlers.
• TracePredictive: The TracePredictive class is deprecated in favor of Predictive, that can be used to gather samples from the posterior and predictive distributions in SVI and MCMC.
• mcmc.predictive: This utility function has been absorbed into the more general Predictive class.

0.5.1

Patches 0.5.0 with the following bug fixes:

• Removes f-string which is only supported in Python 3.6+, so that Python 3.5 is supported.
• Fix incompatibility with recent tqdm releases which make multiple bars not work in the notebook environment (for MCMC with multiple chains).

0.5.0

New features

• pyro.factor to add arbitrary log probability factor to a probabilistic model.
• Conditional MADE Autoregressive Network available in pyro.nn.
• Tutorial on adaptive experiment design for studying working memory.
• KL divergence for Delta and Independent distributions.
• A fast n log(n) implementation of the Continuous Ranked Probability Score (CRPS) for sample sets: pyro.ops.stats.crps_empirical

Code changes and bug fixes

• Moved pyro.generic to a separate pyro-api package.
• Minor changes to ensure compatibility with pyro-api, a generic modeling and inference API for dispatch to different Pyro backends.
• Improve numerical stability of MixtureOfDiagonals distribution using logsumexp operation.
• Improved U-Turn check condition in NUTS for better sampling efficiency.
• Reorganized constraints and transforms module to match torch.distributions.
• Fixed AutoGuide intitialization stragtegies, resolving a bug in init_to_median.

0.4.1

New Features:

• *HMM.filter() methods for forecasting.
• Support for Independent(Normal) observations in GaussianHMM.

Fixes:

• Fix for HMC / NUTS to handle errors arising from numerical issues when computing Cholesky decomposition.

0.4.0

This release drops support for Python 2. Additionally, it includes a few fixes to enable Pyro to use the latest PyTorch release, version 1.2.

Some other additions / minor changes:

• Add option for sequential predictions for MCMC predictive.
• Move pyro.contrib.autoguide to the core Pyro repo.
• Additional inference algorithms
  • SMCFilter for filtering via Sequential Monte Carlo
  • Stein Variational Gradient Descent (SVGD).
• Add a GaussianHMM distribution for fast tuning of Gaussian state space models / Kalman filters

0.3.4

New features

• A more flexible easyguide module. Refer to the tutorial for usage instructions.
• Different initialization methods for autoguides.
• More normalizing flows - Block Neural Autoregressive Flow, Sum of Squares, Sylvester Flow, DeepELUFlow, Householder Flow, RealNVP.
• Support ReduceLROnPlateau scheduler.
• New interface for MCMC inference:
  • Ability to specify a potential function directly instead of Pyro model in HMC/NUTS kernels.
  • MCMC.summary() method that provides site level summary and diagnostic information.
  • Utility function for predictive that replaces the TracePredictive class.
  • Add divergence information to MCMC.diagnostics().
• A DiscreteHMM distribution for fast parallel training of discrete-state Hidden Markov Models with arbitrary observation distributions. See examples/hmm.py for example usage in a neural HMM.

Code changes and bug fixes

• Addresses pickling issue with Pyro handlers that makes it possible to pickle a much larger class of models.
• Multiple fixes for multiprocessing bugs with MCMC. With the new interface, the memory consumption is low thereby allowing for collecting many more samples.
• Performance enhancements for models with many sample sites.

0.3.3

Updates code for compatibility with PyTorch's latest release of version 1.1.0 - mostly function renaming, and using alternate tensor constructors in JIT to avoid DeprecationWarning.

0.3.2

New features

• A capture-recapture example using stochastic variational inference.
• ELBO with trace adaptive f-divergence - TraceTailAdaptive_ELBO.
• New distribution classes - LKJCorrCholesky, SpanningTree.
• Distribution transforms - RadialFlow, DeepSigmoidalFlow, BatchNormTransform.
• Pareto-smoothed Importance Sampling (PSIS) diagnostic for Variational Inference.
• Vectorized indexing with Vindex. Refer to the enumeration tutorial for more details on usage.
• pyro.contrib.minipyro now supports constrained parameters.
• pyro.generic module to support an API for backend-agnostic Pyro models. This makes it easier to switch between full Pyro and Minipyro. New backends like funsor and numpyro are under active development.
• pyro.contrib.conjugate that provides utilities for exact inference on a small subset of conjugate models.

Code changes and bug fixes

• Optimized Categorical.log_prob so that evaluation on the distribution's support is much faster leading to almost 2X faster inference on models with enumerated discrete random variables.
• pyro.module ignores params with requires_grad=False.
• Correct shape inference in MaskedDistribution when run under torch.jit.trace.
• Fixed infer_discrete to support plates of size 1 and variable dependencies across plate contexts.

0.3.1

Dependency changes

• Removes dependency on networkx.
• Uses opt_einsum version 2.3.2.

New features

• Conjugate probability distributions BetaBinomial, DirichletMultinomial, GammaPoisson.
• The VonMises distribution now supports sampling (thanks to @ahmadsalim)
• A second order HMM example.
• A mixed-HMM example with seals data.

Minor changes and bug fixes

• Renamed ubersum(..., batch_dims=...) (deprecated) to einsum(..., plates=...).
• HMC - fix bug in initial trace setter and diagnostics, resolve slowness issues during adaptation, expose target_accept_prob and max_tree_depth as arguments to the constructor to allow finer grained control over hyper-parameters.
• Many small fixes to the tutorials.

0.3.0

New features

Parallel sampling

Building on poutine.broadcast, Pyro's SVI and HMC inference algorithms now support parallel sampling. For example to use parallel sampling in SVI, create an ELBO instance and configure two particles options, e.g.

elbo = Trace_ELBO(num_particles=100,
                  vectorize_particles=True)

Dependent enumeration

TraceEnum_ELBO, HMC, NUTS, and infer_discrete can now exactly marginalize-out discrete latent variables. For dependency structures with narrow treewidth, Pyro performs cheap marginalization via message-passing algorithms, enabling classic learning algorithms such as Baum-Welch for HMMs, DBNs, and CRFs. See our enumeration tutorial for details.

HMC/NUTS enhancements

• Mass matrix can be learned along with step size during the warmup phase. This feature significantly improves the performance of HMC/NUTS.
• Multiple parallel chain is supported (on the CPU), together with various chain diagnostics such as R-hat and effective sample size.
• Models with discrete latent variables will be enumerated over in parallel.
• In NUTS, there are two ways of choosing a candidate from a trajectory: multinomial sampling and slice sampling. We have implemented and used multinomial sampling by default.

New distributions

• MaskedMixture interleaves two distributions element-wise, as a succinct alternative to multiple sample statements inside multiple poutine.mask contexts.
• RelaxedBernoulliStraightThrough and RelaxedOneHotCategoricalStraightThrough
  These are discrete distributions that have been relaxed to continuous space and thus are equipped with pathwise gradients. Thus these distributions can be useful in the context of variational inference, where they can provide lower variance (but biased) gradient estimates. Note that these distributions may be numerically finicky so please let us know if you run into any problems.
• VonMises and VonMises3D are likelihood-only distributions that are useful for observing 2D or 3D angle data.
• AVFMultivariateNormal is a multivariate normal distribution that comes equipped with an adaptive gradient estimator that can lead to reduce gradient variance.
• MixtureOfDiagNormals, MixtureOfDiagNormalsSharedCovariance and GaussianScaleMixture are three families of mixture distributions that come equipped with pathwise gradient estimators (which tend to yield low variance gradients).
• PlanarFlow and PermutationFlow are two transforms useful for constructing normalizing flows.
• InverseAutoregressiveFlow improvements such as an explicit inversion operator.

GPyTorch integration

This isn't really a feature of Pyro, but we'd like to point out a new feature of the excellent GPyTorch library: GPyTorch can now use Pyro for variational inference, and GPyTorch models can now be used in Pyro models. We recommend the new TraceMeanField_ELBO loss for GPyTorch models.

Analytic KL in ELBO

TraceMeanField_ELBO can take advantage of analytic KL divergence expressions in ELBO computations, when available. This ELBO implementation makes some restriction on variable dependency structure. This is especially useful for GPyTorch models.

IWELBO

An implementation of the Importance Weighted ELBO objective (pyro.infer.RenyiELBO) is now included. This implementation also includes the generalization of IWELBO to the alpha-divergence (or Rényi divergence of order α) case.

Automatic max_plate_nesting

Pyro's ELBO implementations can now automatically determine max_plate_nesting (formerly know as max_iarange_nesting) in models with static plate nesting structure.

Autoguide

Some new autoguides are implemented: AutoIAFNormal and AutoLaplaceApproximation.

Support for the PyTorch JIT

The PyTorch JIT compiler currently has only partial support for ops used in Pyro programs. If your model has static structure and you're lucky enough to use ops supported by the JIT, you can sometimes get an order-of-magnitude speedup. To enable the JIT in SVI, simply replace Trace_ELBO, TraceGraph_ELBO, or TraceEnum_ELBO classes with their JIT wrappers JitTrace_ELBO, JitTraceGraph_ELBO, or JitTraceEnum_ELBO. To enable the JIT in HMC or NUTS pass the jit_compile kwarg. See our JIT tutorial for details.

Stats

pyro.ops.stats contains many popular statistics functions such as resample, quantile, pi (percentile interval), hpdi (highest posterior density interval), autocorrelation, autocovariance, etc

Better error messages

Pyro now provides more validation checks and better error messages, including shape logging using the Trace.format_shapes() method. This is very useful for debugging shape errors. See the tensor shapes tutorial for help in reading the shapes table.

New Tutorials

Language

• Custom Variational Objectives
• Enumeration
• Mini-Pyro
• Effect Handlers
• Using the PyTorch jit compiler with Pyro

Examples

• Rational Speech Acts hyperbole and implicature
• Compiled Sequential Importance Sampling
• Object tracking
• (Extended) Kalman Filter

and additional examples in the examples directory.

Breaking changes

• pyro.plate replaces all of pyro.irange, pyro.iarange, and poutine.broadcast. You should no longer need to use poutine.broadcast manually.
• independent() is now renamed to_event()
• poutine.mask was separated from poutine.scale. Now you should use poutine.mask with ByteTensors and poutine.scale for positive tensors (usually just scalars).
• .enumerate_support(expand=False)
• Some distributions interfaces changed, e.g. LowRankMultivariateNormal and HalfNormal
• Pyro Gaussian Process module has been revised:
  • Variational inference now works with PyTorch parameters directly instead of interacting with Pyro ParamStoreDict as before.
  • Methods .get_param(name) and .fix_param(name, value) are removed.
  • Auto guide is supported through the method .autoguide(name, ...). And we have implemented Bayesian GPLVM model to illustrate autoguide functionality.
  • Kernel methods .sum(), .product() are removed. Instead, we encourage users using a better paradigm: Sum(kern0, kern1), Product(kern0, kern1).

Note also that Pyro 0.3 now uses PyTorch 1.0, which makes a number of breaking changes.

Experimental features

We are releasing the following features early, intended for experimental use only. Pyro provides no backward-compatibility guarantees for these features.

Tracking and data association

pyro.contrib.tracking is provides some experimental components for data association and multiple-object tracking. See the object tracking and Kalman Filter tutorials for examples of using the library.

Optimal experimental design

pyro.contrib.oed This package provides some support for doing Bayesian Optimal Experimental Design (OED) in Pyro. In particular it provides support for estimating the Estimated Information Gain, which is one of the key quantities required for Bayesian OED. This package is in active development and is expected to undergo significant changes. See the docs for more details.

Automatic naming and scoping

pyro.contrib.autoname provides some automatic naming utilities that can ease the burden of subscripting like "x_{}".format(t).