+```
+
+#### Args:
+
+
+* `method`: The method to wrap.
+
+
+#### Returns:
+
+The original method wrapped such that it enters the module's name scope.
+
+
+
+
diff --git a/docs/api_docs/python/tfa/metrics/cohens_kappa.md b/docs/api_docs/python/tfa/metrics/cohens_kappa.md
new file mode 100644
index 0000000000..7c01111f0a
--- /dev/null
+++ b/docs/api_docs/python/tfa/metrics/cohens_kappa.md
@@ -0,0 +1,20 @@
+
+
+
+
+
+# Module: tfa.metrics.cohens_kappa
+
+Implements Cohen's Kappa.
+
+
+
+Defined in [`metrics/cohens_kappa.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/metrics/cohens_kappa.py).
+
+
+
+
+## Classes
+
+[`class CohenKappa`](../../tfa/metrics/CohenKappa.md): Computes Kappa score between two raters.
+
diff --git a/docs/api_docs/python/tfa/optimizers.md b/docs/api_docs/python/tfa/optimizers.md
new file mode 100644
index 0000000000..3dd4e6e411
--- /dev/null
+++ b/docs/api_docs/python/tfa/optimizers.md
@@ -0,0 +1,38 @@
+
+
+
+
+
+# Module: tfa.optimizers
+
+Additional optimizers that conform to Keras API.
+
+
+
+Defined in [`optimizers/__init__.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/optimizers/__init__.py).
+
+
+
+
+## Modules
+
+[`lazy_adam`](../tfa/optimizers/lazy_adam.md) module: Variant of the Adam optimizer that handles sparse updates more efficiently.
+
+[`moving_average`](../tfa/optimizers/moving_average.md) module
+
+[`weight_decay_optimizers`](../tfa/optimizers/weight_decay_optimizers.md) module: Base class to make optimizers weight decay ready.
+
+## Classes
+
+[`class AdamW`](../tfa/optimizers/AdamW.md): Optimizer that implements the Adam algorithm with weight decay.
+
+[`class LazyAdam`](../tfa/optimizers/LazyAdam.md): Variant of the Adam optimizer that handles sparse updates more
+
+[`class MovingAverage`](../tfa/optimizers/MovingAverage.md): Optimizer that computes a moving average of the variables.
+
+[`class SGDW`](../tfa/optimizers/SGDW.md): Optimizer that implements the Momentum algorithm with weight_decay.
+
+## Functions
+
+[`extend_with_decoupled_weight_decay(...)`](../tfa/optimizers/extend_with_decoupled_weight_decay.md): Factory function returning an optimizer class with decoupled weight
+
diff --git a/docs/api_docs/python/tfa/optimizers/AdamW.md b/docs/api_docs/python/tfa/optimizers/AdamW.md
new file mode 100644
index 0000000000..92265900d5
--- /dev/null
+++ b/docs/api_docs/python/tfa/optimizers/AdamW.md
@@ -0,0 +1,384 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+# tfa.optimizers.AdamW
+
+## Class `AdamW`
+
+Optimizer that implements the Adam algorithm with weight decay.
+
+Inherits From: [`DecoupledWeightDecayExtension`](../../tfa/optimizers/weight_decay_optimizers/DecoupledWeightDecayExtension.md)
+
+### Aliases:
+
+* Class `tfa.optimizers.AdamW`
+* Class `tfa.optimizers.weight_decay_optimizers.AdamW`
+
+
+
+Defined in [`optimizers/weight_decay_optimizers.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/optimizers/weight_decay_optimizers.py).
+
+
+
+This is an implementation of the AdamW optimizer described in "Decoupled
+Weight Decay Regularization" by Loshchilov & Hutter
+(https://arxiv.org/abs/1711.05101)
+([pdf])(https://arxiv.org/pdf/1711.05101.pdf).
+
+It computes the update step of `tf.keras.optimizers.Adam` and additionally
+decays the variable. Note that this is different from adding L2
+regularization on the variables to the loss: it regularizes variables with
+large gradients more than L2 regularization would, which was shown to yield
+better training loss and generalization error in the paper above.
+
+For further information see the documentation of the Adam Optimizer.
+
+This optimizer can also be instantiated as
+```python
+extend_with_decoupled_weight_decay(tf.keras.optimizers.Adam,
+ weight_decay=weight_decay)
+```
+
+Note: when applying a decay to the learning rate, be sure to manually apply
+the decay to the `weight_decay` as well. For example:
+
+```python
+step = tf.Variable(0, trainable=False)
+schedule = tf.optimizers.schedules.PiecewiseConstantDecay(
+ [10000, 15000], [1e-0, 1e-1, 1e-2])
+# lr and wd can be a function or a tensor
+lr = 1e-1 * schedule(step)
+wd = lambda: 1e-4 * schedule(step)
+
+# ...
+
+optimizer = tfa.optimizers.AdamW(learning_rate=lr, weight_decay=wd)
+```
+
+__init__
+
+``` python
+__init__(
+ weight_decay,
+ learning_rate=0.001,
+ beta_1=0.9,
+ beta_2=0.999,
+ epsilon=1e-07,
+ amsgrad=False,
+ name='AdamW',
+ **kwargs
+)
+```
+
+Construct a new AdamW optimizer.
+
+For further information see the documentation of the Adam Optimizer.
+
+#### Args:
+
+
+* `weight_decay`: A Tensor or a floating point value. The weight decay.
+* `learning_rate`: A Tensor or a floating point value. The learning
+ rate.
+* `beta_1`: A float value or a constant float tensor. The exponential
+ decay rate for the 1st moment estimates.
+* `beta_2`: A float value or a constant float tensor. The exponential
+ decay rate for the 2nd moment estimates.
+* `epsilon`: A small constant for numerical stability. This epsilon is
+ "epsilon hat" in the Kingma and Ba paper (in the formula just
+ before Section 2.1), not the epsilon in Algorithm 1 of the
+ paper.
+* `amsgrad`: boolean. Whether to apply AMSGrad variant of this
+ algorithm from the paper "On the Convergence of Adam and
+ beyond".
+* `name`: Optional name for the operations created when applying
+ gradients. Defaults to "AdamW".
+* `**kwargs`: keyword arguments. Allowed to be {`clipnorm`,
+ `clipvalue`, `lr`, `decay`}. `clipnorm` is clip gradients by
+ norm; `clipvalue` is clip gradients by value, `decay` is
+ included for backward compatibility to allow time inverse decay
+ of learning rate. `lr` is included for backward compatibility,
+ recommended to use `learning_rate` instead.
+
+
+
+## Properties
+
+iterations
+
+Variable. The number of training steps this Optimizer has run.
+
+
+weights
+
+Returns variables of this Optimizer based on the order created.
+
+
+
+
+## Methods
+
+add_slot
+
+``` python
+add_slot(
+ var,
+ slot_name,
+ initializer='zeros'
+)
+```
+
+Add a new slot variable for `var`.
+
+
+add_weight
+
+``` python
+add_weight(
+ name,
+ shape,
+ dtype=None,
+ initializer='zeros',
+ trainable=None,
+ synchronization=tf_variables.VariableSynchronization.AUTO,
+ aggregation=tf_variables.VariableAggregation.NONE
+)
+```
+
+
+
+
+apply_gradients
+
+``` python
+apply_gradients(
+ grads_and_vars,
+ name=None,
+ decay_var_list=None
+)
+```
+
+Apply gradients to variables.
+
+This is the second part of `minimize()`. It returns an `Operation` that
+applies gradients.
+
+#### Args:
+
+
+* `grads_and_vars`: List of (gradient, variable) pairs.
+* `name`: Optional name for the returned operation. Default to the
+ name passed to the `Optimizer` constructor.
+* `decay_var_list`: Optional list of variables to be decayed. Defaults
+ to all variables in var_list.
+
+#### Returns:
+
+An `Operation` that applies the specified gradients. If
+`global_step` was not None, that operation also increments
+`global_step`.
+
+
+#### Raises:
+
+
+* `TypeError`: If `grads_and_vars` is malformed.
+* `ValueError`: If none of the variables have gradients.
+
+from_config
+
+``` python
+from_config(
+ cls,
+ config,
+ custom_objects=None
+)
+```
+
+Creates an optimizer from its config.
+
+This method is the reverse of `get_config`,
+capable of instantiating the same optimizer from the config
+dictionary.
+
+#### Arguments:
+
+
+* `config`: A Python dictionary, typically the output of get_config.
+* `custom_objects`: A Python dictionary mapping names to additional Python
+ objects used to create this optimizer, such as a function used for a
+ hyperparameter.
+
+
+#### Returns:
+
+An optimizer instance.
+
+
+get_config
+
+``` python
+get_config()
+```
+
+
+
+
+get_gradients
+
+``` python
+get_gradients(
+ loss,
+ params
+)
+```
+
+Returns gradients of `loss` with respect to `params`.
+
+
+#### Arguments:
+
+
+* `loss`: Loss tensor.
+* `params`: List of variables.
+
+
+#### Returns:
+
+List of gradient tensors.
+
+
+
+#### Raises:
+
+
+* `ValueError`: In case any gradient cannot be computed (e.g. if gradient
+ function not implemented).
+
+get_slot
+
+``` python
+get_slot(
+ var,
+ slot_name
+)
+```
+
+
+
+
+get_slot_names
+
+``` python
+get_slot_names()
+```
+
+A list of names for this optimizer's slots.
+
+
+get_updates
+
+``` python
+get_updates(
+ loss,
+ params
+)
+```
+
+
+
+
+get_weights
+
+``` python
+get_weights()
+```
+
+
+
+
+minimize
+
+``` python
+minimize(
+ loss,
+ var_list,
+ grad_loss=None,
+ name=None,
+ decay_var_list=None
+)
+```
+
+Minimize `loss` by updating `var_list`.
+
+This method simply computes gradient using `tf.GradientTape` and calls
+`apply_gradients()`. If you want to process the gradient before
+applying then call `tf.GradientTape` and `apply_gradients()` explicitly
+instead of using this function.
+
+#### Args:
+
+
+* `loss`: A callable taking no arguments which returns the value to
+ minimize.
+* `var_list`: list or tuple of `Variable` objects to update to
+ minimize `loss`, or a callable returning the list or tuple of
+ `Variable` objects. Use callable when the variable list would
+ otherwise be incomplete before `minimize` since the variables
+ are created at the first time `loss` is called.
+* `grad_loss`: Optional. A `Tensor` holding the gradient computed for
+ `loss`.
+* `decay_var_list`: Optional list of variables to be decayed. Defaults
+ to all variables in var_list.
+* `name`: Optional name for the returned operation.
+
+#### Returns:
+
+An Operation that updates the variables in `var_list`. If
+`global_step` was not `None`, that operation also increments
+`global_step`.
+
+
+#### Raises:
+
+
+* `ValueError`: If some of the variables are not `Variable` objects.
+
+set_weights
+
+``` python
+set_weights(weights)
+```
+
+
+
+
+variables
+
+``` python
+variables()
+```
+
+Returns variables of this Optimizer based on the order created.
+
+
+
+
diff --git a/docs/api_docs/python/tfa/optimizers/LazyAdam.md b/docs/api_docs/python/tfa/optimizers/LazyAdam.md
new file mode 100644
index 0000000000..68439b093f
--- /dev/null
+++ b/docs/api_docs/python/tfa/optimizers/LazyAdam.md
@@ -0,0 +1,329 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+# tfa.optimizers.LazyAdam
+
+## Class `LazyAdam`
+
+Variant of the Adam optimizer that handles sparse updates more
+
+
+
+### Aliases:
+
+* Class `tfa.optimizers.LazyAdam`
+* Class `tfa.optimizers.lazy_adam.LazyAdam`
+
+
+
+Defined in [`optimizers/lazy_adam.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/optimizers/lazy_adam.py).
+
+
+efficiently.
+
+The original Adam algorithm maintains two moving-average accumulators for
+each trainable variable; the accumulators are updated at every step.
+This class provides lazier handling of gradient updates for sparse
+variables. It only updates moving-average accumulators for sparse variable
+indices that appear in the current batch, rather than updating the
+accumulators for all indices. Compared with the original Adam optimizer,
+it can provide large improvements in model training throughput for some
+applications. However, it provides slightly different semantics than the
+original Adam algorithm, and may lead to different empirical results.
+
+Note, amsgrad is currently not supported and the argument can only be
+False.
+
+__init__
+
+``` python
+__init__(
+ learning_rate=0.001,
+ beta_1=0.9,
+ beta_2=0.999,
+ epsilon=1e-07,
+ amsgrad=False,
+ name='LazyAdam',
+ **kwargs
+)
+```
+
+
+
+
+
+
+## Properties
+
+iterations
+
+Variable. The number of training steps this Optimizer has run.
+
+
+weights
+
+Returns variables of this Optimizer based on the order created.
+
+
+
+
+## Methods
+
+add_slot
+
+``` python
+add_slot(
+ var,
+ slot_name,
+ initializer='zeros'
+)
+```
+
+Add a new slot variable for `var`.
+
+
+add_weight
+
+``` python
+add_weight(
+ name,
+ shape,
+ dtype=None,
+ initializer='zeros',
+ trainable=None,
+ synchronization=tf_variables.VariableSynchronization.AUTO,
+ aggregation=tf_variables.VariableAggregation.NONE
+)
+```
+
+
+
+
+apply_gradients
+
+``` python
+apply_gradients(
+ grads_and_vars,
+ name=None
+)
+```
+
+Apply gradients to variables.
+
+This is the second part of `minimize()`. It returns an `Operation` that
+applies gradients.
+
+#### Args:
+
+
+* `grads_and_vars`: List of (gradient, variable) pairs.
+* `name`: Optional name for the returned operation. Default to the name
+ passed to the `Optimizer` constructor.
+
+
+#### Returns:
+
+An `Operation` that applies the specified gradients. If `global_step`
+was not None, that operation also increments `global_step`.
+
+
+
+#### Raises:
+
+
+* `TypeError`: If `grads_and_vars` is malformed.
+* `ValueError`: If none of the variables have gradients.
+
+from_config
+
+``` python
+from_config(
+ cls,
+ config,
+ custom_objects=None
+)
+```
+
+Creates an optimizer from its config.
+
+This method is the reverse of `get_config`,
+capable of instantiating the same optimizer from the config
+dictionary.
+
+#### Arguments:
+
+
+* `config`: A Python dictionary, typically the output of get_config.
+* `custom_objects`: A Python dictionary mapping names to additional Python
+ objects used to create this optimizer, such as a function used for a
+ hyperparameter.
+
+
+#### Returns:
+
+An optimizer instance.
+
+
+get_config
+
+``` python
+get_config()
+```
+
+
+
+
+get_gradients
+
+``` python
+get_gradients(
+ loss,
+ params
+)
+```
+
+Returns gradients of `loss` with respect to `params`.
+
+
+#### Arguments:
+
+
+* `loss`: Loss tensor.
+* `params`: List of variables.
+
+
+#### Returns:
+
+List of gradient tensors.
+
+
+
+#### Raises:
+
+
+* `ValueError`: In case any gradient cannot be computed (e.g. if gradient
+ function not implemented).
+
+get_slot
+
+``` python
+get_slot(
+ var,
+ slot_name
+)
+```
+
+
+
+
+get_slot_names
+
+``` python
+get_slot_names()
+```
+
+A list of names for this optimizer's slots.
+
+
+get_updates
+
+``` python
+get_updates(
+ loss,
+ params
+)
+```
+
+
+
+
+get_weights
+
+``` python
+get_weights()
+```
+
+
+
+
+minimize
+
+``` python
+minimize(
+ loss,
+ var_list,
+ grad_loss=None,
+ name=None
+)
+```
+
+Minimize `loss` by updating `var_list`.
+
+This method simply computes gradient using `tf.GradientTape` and calls
+`apply_gradients()`. If you want to process the gradient before applying
+then call `tf.GradientTape` and `apply_gradients()` explicitly instead
+of using this function.
+
+#### Args:
+
+
+* `loss`: A callable taking no arguments which returns the value to minimize.
+* `var_list`: list or tuple of `Variable` objects to update to minimize
+ `loss`, or a callable returning the list or tuple of `Variable` objects.
+ Use callable when the variable list would otherwise be incomplete before
+ `minimize` since the variables are created at the first time `loss` is
+ called.
+* `grad_loss`: Optional. A `Tensor` holding the gradient computed for `loss`.
+* `name`: Optional name for the returned operation.
+
+
+#### Returns:
+
+An Operation that updates the variables in `var_list`. If `global_step`
+was not `None`, that operation also increments `global_step`.
+
+
+
+#### Raises:
+
+
+* `ValueError`: If some of the variables are not `Variable` objects.
+
+set_weights
+
+``` python
+set_weights(weights)
+```
+
+
+
+
+variables
+
+``` python
+variables()
+```
+
+Returns variables of this Optimizer based on the order created.
+
+
+
+
diff --git a/docs/api_docs/python/tfa/optimizers/MovingAverage.md b/docs/api_docs/python/tfa/optimizers/MovingAverage.md
new file mode 100644
index 0000000000..acd7601ee0
--- /dev/null
+++ b/docs/api_docs/python/tfa/optimizers/MovingAverage.md
@@ -0,0 +1,334 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+# tfa.optimizers.MovingAverage
+
+## Class `MovingAverage`
+
+Optimizer that computes a moving average of the variables.
+
+
+
+### Aliases:
+
+* Class `tfa.optimizers.MovingAverage`
+* Class `tfa.optimizers.moving_average.MovingAverage`
+
+
+
+Defined in [`optimizers/moving_average.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/optimizers/moving_average.py).
+
+
+
+Empirically it has been found that using the moving average of the trained
+parameters of a deep network is better than using its trained parameters
+directly. This optimizer allows you to compute this moving average and swap
+the variables at save time so that any code outside of the training loop
+will use by default the average values instead of the original ones.
+
+#### Example of usage:
+
+
+
+```python
+opt = tf.keras.optimizers.SGD(learning_rate)
+opt = tfa.optimizers.MovingAverage(opt)
+
+```
+
+__init__
+
+``` python
+__init__(
+ optimizer,
+ average_decay=0.1,
+ num_updates=None,
+ sequential_update=True,
+ name='MovingAverage',
+ **kwargs
+)
+```
+
+
+
+
+
+
+## Properties
+
+iterations
+
+Variable. The number of training steps this Optimizer has run.
+
+
+weights
+
+
+
+
+
+
+## Methods
+
+add_slot
+
+``` python
+add_slot(
+ var,
+ slot_name,
+ initializer='zeros'
+)
+```
+
+Add a new slot variable for `var`.
+
+
+add_weight
+
+``` python
+add_weight(
+ name,
+ shape,
+ dtype=None,
+ initializer='zeros',
+ trainable=None,
+ synchronization=tf_variables.VariableSynchronization.AUTO,
+ aggregation=tf_variables.VariableAggregation.NONE
+)
+```
+
+
+
+
+apply_gradients
+
+``` python
+apply_gradients(
+ grads_and_vars,
+ name=None
+)
+```
+
+
+
+
+assign_average_vars
+
+``` python
+assign_average_vars(var_list)
+```
+
+Update variables in var_list with the running mean of the variables.
+
+
+#### Example:
+
+
+```python
+model = tf.Sequential([...])
+opt = tfa.optimizers.MovingAverage(
+ tf.keras.optimizers.SGD(lr=2.0), 0.5)
+
+model.compile(opt, ...)
+model.fit(x, y, ...)
+
+# Update the weights to their mean before saving
+opt.assign_average_vars(model.variables)
+
+model.save('model.h5')
+```
+
+from_config
+
+``` python
+from_config(
+ cls,
+ config,
+ custom_objects=None
+)
+```
+
+Creates an optimizer from its config.
+
+This method is the reverse of `get_config`,
+capable of instantiating the same optimizer from the config
+dictionary.
+
+#### Arguments:
+
+
+* `config`: A Python dictionary, typically the output of get_config.
+* `custom_objects`: A Python dictionary mapping names to additional Python
+ objects used to create this optimizer, such as a function used for a
+ hyperparameter.
+
+
+#### Returns:
+
+An optimizer instance.
+
+
+get_config
+
+``` python
+get_config()
+```
+
+
+
+
+get_gradients
+
+``` python
+get_gradients(
+ loss,
+ params
+)
+```
+
+Returns gradients of `loss` with respect to `params`.
+
+
+#### Arguments:
+
+
+* `loss`: Loss tensor.
+* `params`: List of variables.
+
+
+#### Returns:
+
+List of gradient tensors.
+
+
+
+#### Raises:
+
+
+* `ValueError`: In case any gradient cannot be computed (e.g. if gradient
+ function not implemented).
+
+get_slot
+
+``` python
+get_slot(
+ var,
+ slot_name
+)
+```
+
+
+
+
+get_slot_names
+
+``` python
+get_slot_names()
+```
+
+A list of names for this optimizer's slots.
+
+
+get_updates
+
+``` python
+get_updates(
+ loss,
+ params
+)
+```
+
+
+
+
+get_weights
+
+``` python
+get_weights()
+```
+
+
+
+
+minimize
+
+``` python
+minimize(
+ loss,
+ var_list,
+ grad_loss=None,
+ name=None
+)
+```
+
+Minimize `loss` by updating `var_list`.
+
+This method simply computes gradient using `tf.GradientTape` and calls
+`apply_gradients()`. If you want to process the gradient before applying
+then call `tf.GradientTape` and `apply_gradients()` explicitly instead
+of using this function.
+
+#### Args:
+
+
+* `loss`: A callable taking no arguments which returns the value to minimize.
+* `var_list`: list or tuple of `Variable` objects to update to minimize
+ `loss`, or a callable returning the list or tuple of `Variable` objects.
+ Use callable when the variable list would otherwise be incomplete before
+ `minimize` since the variables are created at the first time `loss` is
+ called.
+* `grad_loss`: Optional. A `Tensor` holding the gradient computed for `loss`.
+* `name`: Optional name for the returned operation.
+
+
+#### Returns:
+
+An Operation that updates the variables in `var_list`. If `global_step`
+was not `None`, that operation also increments `global_step`.
+
+
+
+#### Raises:
+
+
+* `ValueError`: If some of the variables are not `Variable` objects.
+
+set_weights
+
+``` python
+set_weights(weights)
+```
+
+
+
+
+variables
+
+``` python
+variables()
+```
+
+Returns variables of this Optimizer based on the order created.
+
+
+
+
diff --git a/docs/api_docs/python/tfa/optimizers/SGDW.md b/docs/api_docs/python/tfa/optimizers/SGDW.md
new file mode 100644
index 0000000000..a564362933
--- /dev/null
+++ b/docs/api_docs/python/tfa/optimizers/SGDW.md
@@ -0,0 +1,372 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+# tfa.optimizers.SGDW
+
+## Class `SGDW`
+
+Optimizer that implements the Momentum algorithm with weight_decay.
+
+Inherits From: [`DecoupledWeightDecayExtension`](../../tfa/optimizers/weight_decay_optimizers/DecoupledWeightDecayExtension.md)
+
+### Aliases:
+
+* Class `tfa.optimizers.SGDW`
+* Class `tfa.optimizers.weight_decay_optimizers.SGDW`
+
+
+
+Defined in [`optimizers/weight_decay_optimizers.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/optimizers/weight_decay_optimizers.py).
+
+
+
+This is an implementation of the SGDW optimizer described in "Decoupled
+Weight Decay Regularization" by Loshchilov & Hutter
+(https://arxiv.org/abs/1711.05101)
+([pdf])(https://arxiv.org/pdf/1711.05101.pdf).
+It computes the update step of `tf.keras.optimizers.SGD` and additionally
+decays the variable. Note that this is different from adding
+L2 regularization on the variables to the loss. Decoupling the weight decay
+from other hyperparameters (in particular the learning rate) simplifies
+hyperparameter search.
+
+For further information see the documentation of the SGD Optimizer.
+
+This optimizer can also be instantiated as
+```python
+extend_with_decoupled_weight_decay(tf.keras.optimizers.SGD,
+ weight_decay=weight_decay)
+```
+
+Note: when applying a decay to the learning rate, be sure to manually apply
+the decay to the `weight_decay` as well. For example:
+
+```python
+step = tf.Variable(0, trainable=False)
+schedule = tf.optimizers.schedules.PiecewiseConstantDecay(
+ [10000, 15000], [1e-0, 1e-1, 1e-2])
+# lr and wd can be a function or a tensor
+lr = 1e-1 * schedule(step)
+wd = lambda: 1e-4 * schedule(step)
+
+# ...
+
+optimizer = tfa.optimizers.SGDW(
+ learning_rate=lr, weight_decay=wd, momentum=0.9)
+```
+
+__init__
+
+``` python
+__init__(
+ weight_decay,
+ learning_rate=0.001,
+ momentum=0.0,
+ nesterov=False,
+ name='SGDW',
+ **kwargs
+)
+```
+
+Construct a new SGDW optimizer.
+
+For further information see the documentation of the SGD Optimizer.
+
+#### Args:
+
+
+* `learning_rate`: float hyperparameter >= 0. Learning rate.
+* `momentum`: float hyperparameter >= 0 that accelerates SGD in the
+ relevant direction and dampens oscillations.
+* `nesterov`: boolean. Whether to apply Nesterov momentum.
+* `name`: Optional name prefix for the operations created when applying
+ gradients. Defaults to 'SGD'.
+* `**kwargs`: keyword arguments. Allowed to be {`clipnorm`,
+ `clipvalue`, `lr`, `decay`}. `clipnorm` is clip gradients by
+ norm; `clipvalue` is clip gradients by value, `decay` is
+ included for backward compatibility to allow time inverse decay
+ of learning rate. `lr` is included for backward compatibility,
+ recommended to use `learning_rate` instead.
+
+
+
+## Properties
+
+iterations
+
+Variable. The number of training steps this Optimizer has run.
+
+
+weights
+
+Returns variables of this Optimizer based on the order created.
+
+
+
+
+## Methods
+
+add_slot
+
+``` python
+add_slot(
+ var,
+ slot_name,
+ initializer='zeros'
+)
+```
+
+Add a new slot variable for `var`.
+
+
+add_weight
+
+``` python
+add_weight(
+ name,
+ shape,
+ dtype=None,
+ initializer='zeros',
+ trainable=None,
+ synchronization=tf_variables.VariableSynchronization.AUTO,
+ aggregation=tf_variables.VariableAggregation.NONE
+)
+```
+
+
+
+
+apply_gradients
+
+``` python
+apply_gradients(
+ grads_and_vars,
+ name=None,
+ decay_var_list=None
+)
+```
+
+Apply gradients to variables.
+
+This is the second part of `minimize()`. It returns an `Operation` that
+applies gradients.
+
+#### Args:
+
+
+* `grads_and_vars`: List of (gradient, variable) pairs.
+* `name`: Optional name for the returned operation. Default to the
+ name passed to the `Optimizer` constructor.
+* `decay_var_list`: Optional list of variables to be decayed. Defaults
+ to all variables in var_list.
+
+#### Returns:
+
+An `Operation` that applies the specified gradients. If
+`global_step` was not None, that operation also increments
+`global_step`.
+
+
+#### Raises:
+
+
+* `TypeError`: If `grads_and_vars` is malformed.
+* `ValueError`: If none of the variables have gradients.
+
+from_config
+
+``` python
+from_config(
+ cls,
+ config,
+ custom_objects=None
+)
+```
+
+Creates an optimizer from its config.
+
+This method is the reverse of `get_config`,
+capable of instantiating the same optimizer from the config
+dictionary.
+
+#### Arguments:
+
+
+* `config`: A Python dictionary, typically the output of get_config.
+* `custom_objects`: A Python dictionary mapping names to additional Python
+ objects used to create this optimizer, such as a function used for a
+ hyperparameter.
+
+
+#### Returns:
+
+An optimizer instance.
+
+
+get_config
+
+``` python
+get_config()
+```
+
+
+
+
+get_gradients
+
+``` python
+get_gradients(
+ loss,
+ params
+)
+```
+
+Returns gradients of `loss` with respect to `params`.
+
+
+#### Arguments:
+
+
+* `loss`: Loss tensor.
+* `params`: List of variables.
+
+
+#### Returns:
+
+List of gradient tensors.
+
+
+
+#### Raises:
+
+
+* `ValueError`: In case any gradient cannot be computed (e.g. if gradient
+ function not implemented).
+
+get_slot
+
+``` python
+get_slot(
+ var,
+ slot_name
+)
+```
+
+
+
+
+get_slot_names
+
+``` python
+get_slot_names()
+```
+
+A list of names for this optimizer's slots.
+
+
+get_updates
+
+``` python
+get_updates(
+ loss,
+ params
+)
+```
+
+
+
+
+get_weights
+
+``` python
+get_weights()
+```
+
+
+
+
+minimize
+
+``` python
+minimize(
+ loss,
+ var_list,
+ grad_loss=None,
+ name=None,
+ decay_var_list=None
+)
+```
+
+Minimize `loss` by updating `var_list`.
+
+This method simply computes gradient using `tf.GradientTape` and calls
+`apply_gradients()`. If you want to process the gradient before
+applying then call `tf.GradientTape` and `apply_gradients()` explicitly
+instead of using this function.
+
+#### Args:
+
+
+* `loss`: A callable taking no arguments which returns the value to
+ minimize.
+* `var_list`: list or tuple of `Variable` objects to update to
+ minimize `loss`, or a callable returning the list or tuple of
+ `Variable` objects. Use callable when the variable list would
+ otherwise be incomplete before `minimize` since the variables
+ are created at the first time `loss` is called.
+* `grad_loss`: Optional. A `Tensor` holding the gradient computed for
+ `loss`.
+* `decay_var_list`: Optional list of variables to be decayed. Defaults
+ to all variables in var_list.
+* `name`: Optional name for the returned operation.
+
+#### Returns:
+
+An Operation that updates the variables in `var_list`. If
+`global_step` was not `None`, that operation also increments
+`global_step`.
+
+
+#### Raises:
+
+
+* `ValueError`: If some of the variables are not `Variable` objects.
+
+set_weights
+
+``` python
+set_weights(weights)
+```
+
+
+
+
+variables
+
+``` python
+variables()
+```
+
+Returns variables of this Optimizer based on the order created.
+
+
+
+
diff --git a/docs/api_docs/python/tfa/optimizers/extend_with_decoupled_weight_decay.md b/docs/api_docs/python/tfa/optimizers/extend_with_decoupled_weight_decay.md
new file mode 100644
index 0000000000..833f2a8169
--- /dev/null
+++ b/docs/api_docs/python/tfa/optimizers/extend_with_decoupled_weight_decay.md
@@ -0,0 +1,83 @@
+
+
+
+
+
+# tfa.optimizers.extend_with_decoupled_weight_decay
+
+Factory function returning an optimizer class with decoupled weight
+
+### Aliases:
+
+* `tfa.optimizers.extend_with_decoupled_weight_decay`
+* `tfa.optimizers.weight_decay_optimizers.extend_with_decoupled_weight_decay`
+
+``` python
+tfa.optimizers.extend_with_decoupled_weight_decay(base_optimizer)
+```
+
+
+
+Defined in [`optimizers/weight_decay_optimizers.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/optimizers/weight_decay_optimizers.py).
+
+
+decay.
+
+Returns an optimizer class. An instance of the returned class computes the
+update step of `base_optimizer` and additionally decays the weights.
+E.g., the class returned by
+`extend_with_decoupled_weight_decay(tf.keras.optimizers.Adam)` is
+equivalent to tfa.optimizers.AdamW.
+
+The API of the new optimizer class slightly differs from the API of the
+base optimizer:
+- The first argument to the constructor is the weight decay rate.
+- `minimize` and `apply_gradients` accept the optional keyword argument
+ `decay_var_list`, which specifies the variables that should be decayed.
+ If `None`, all variables that are optimized are decayed.
+
+#### Usage example:
+
+
+```python
+# MyAdamW is a new class
+MyAdamW = extend_with_decoupled_weight_decay(tf.keras.optimizers.Adam)
+# Create a MyAdamW object
+optimizer = MyAdamW(weight_decay=0.001, learning_rate=0.001)
+# update var1, var2 but only decay var1
+optimizer.minimize(loss, var_list=[var1, var2], decay_variables=[var1])
+
+Note: this extension decays weights BEFORE applying the update based
+on the gradient, i.e. this extension only has the desired behaviour for
+optimizers which do not depend on the value of 'var' in the update step!
+
+Note: when applying a decay to the learning rate, be sure to manually apply
+the decay to the `weight_decay` as well. For example:
+
+```python
+step = tf.Variable(0, trainable=False)
+schedule = tf.optimizers.schedules.PiecewiseConstantDecay(
+ [10000, 15000], [1e-0, 1e-1, 1e-2])
+# lr and wd can be a function or a tensor
+lr = 1e-1 * schedule(step)
+wd = lambda: 1e-4 * schedule(step)
+
+# ...
+
+optimizer = tfa.optimizers.AdamW(learning_rate=lr, weight_decay=wd)
+```
+
+Note: you might want to register your own custom optimizer using
+`tf.keras.utils.get_custom_objects()`.
+
+#### Args:
+
+
+* `base_optimizer`: An optimizer class that inherits from
+ tf.optimizers.Optimizer.
+
+
+#### Returns:
+
+A new optimizer class that inherits from DecoupledWeightDecayExtension
+and base_optimizer.
diff --git a/docs/api_docs/python/tfa/optimizers/lazy_adam.md b/docs/api_docs/python/tfa/optimizers/lazy_adam.md
new file mode 100644
index 0000000000..d4c7a6e1c4
--- /dev/null
+++ b/docs/api_docs/python/tfa/optimizers/lazy_adam.md
@@ -0,0 +1,24 @@
+
+
+
+
+
+# Module: tfa.optimizers.lazy_adam
+
+Variant of the Adam optimizer that handles sparse updates more efficiently.
+
+
+
+Defined in [`optimizers/lazy_adam.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/optimizers/lazy_adam.py).
+
+
+
+Compared with the original Adam optimizer, the one in this file can
+provide a large improvement in model training throughput for some
+applications. However, it provides slightly different semantics than the
+original Adam algorithm, and may lead to different empirical results.
+
+## Classes
+
+[`class LazyAdam`](../../tfa/optimizers/LazyAdam.md): Variant of the Adam optimizer that handles sparse updates more
+
diff --git a/docs/api_docs/python/tfa/optimizers/moving_average.md b/docs/api_docs/python/tfa/optimizers/moving_average.md
new file mode 100644
index 0000000000..e069a84e92
--- /dev/null
+++ b/docs/api_docs/python/tfa/optimizers/moving_average.md
@@ -0,0 +1,20 @@
+
+
+
+
+
+# Module: tfa.optimizers.moving_average
+
+
+
+
+
+Defined in [`optimizers/moving_average.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/optimizers/moving_average.py).
+
+
+
+
+## Classes
+
+[`class MovingAverage`](../../tfa/optimizers/MovingAverage.md): Optimizer that computes a moving average of the variables.
+
diff --git a/docs/api_docs/python/tfa/optimizers/weight_decay_optimizers.md b/docs/api_docs/python/tfa/optimizers/weight_decay_optimizers.md
new file mode 100644
index 0000000000..3029d1bb54
--- /dev/null
+++ b/docs/api_docs/python/tfa/optimizers/weight_decay_optimizers.md
@@ -0,0 +1,28 @@
+
+
+
+
+
+# Module: tfa.optimizers.weight_decay_optimizers
+
+Base class to make optimizers weight decay ready.
+
+
+
+Defined in [`optimizers/weight_decay_optimizers.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/optimizers/weight_decay_optimizers.py).
+
+
+
+
+## Classes
+
+[`class AdamW`](../../tfa/optimizers/AdamW.md): Optimizer that implements the Adam algorithm with weight decay.
+
+[`class DecoupledWeightDecayExtension`](../../tfa/optimizers/weight_decay_optimizers/DecoupledWeightDecayExtension.md): This class allows to extend optimizers with decoupled weight decay.
+
+[`class SGDW`](../../tfa/optimizers/SGDW.md): Optimizer that implements the Momentum algorithm with weight_decay.
+
+## Functions
+
+[`extend_with_decoupled_weight_decay(...)`](../../tfa/optimizers/extend_with_decoupled_weight_decay.md): Factory function returning an optimizer class with decoupled weight
+
diff --git a/docs/api_docs/python/tfa/optimizers/weight_decay_optimizers/DecoupledWeightDecayExtension.md b/docs/api_docs/python/tfa/optimizers/weight_decay_optimizers/DecoupledWeightDecayExtension.md
new file mode 100644
index 0000000000..e8ff936bde
--- /dev/null
+++ b/docs/api_docs/python/tfa/optimizers/weight_decay_optimizers/DecoupledWeightDecayExtension.md
@@ -0,0 +1,186 @@
+
+
+
+
+
+
+
+
+
+# tfa.optimizers.weight_decay_optimizers.DecoupledWeightDecayExtension
+
+## Class `DecoupledWeightDecayExtension`
+
+This class allows to extend optimizers with decoupled weight decay.
+
+
+
+
+
+Defined in [`optimizers/weight_decay_optimizers.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/optimizers/weight_decay_optimizers.py).
+
+
+
+It implements the decoupled weight decay described by Loshchilov & Hutter
+(https://arxiv.org/pdf/1711.05101.pdf), in which the weight decay is
+decoupled from the optimization steps w.r.t. to the loss function.
+For SGD variants, this simplifies hyperparameter search since it decouples
+the settings of weight decay and learning rate.
+For adaptive gradient algorithms, it regularizes variables with large
+gradients more than L2 regularization would, which was shown to yield
+better training loss and generalization error in the paper above.
+
+This class alone is not an optimizer but rather extends existing
+optimizers with decoupled weight decay. We explicitly define the two
+examples used in the above paper (SGDW and AdamW), but in general this
+can extend any OptimizerX by using
+`extend_with_decoupled_weight_decay(
+ OptimizerX, weight_decay=weight_decay)`.
+In order for it to work, it must be the first class the Optimizer with
+weight decay inherits from, e.g.
+
+```python
+class AdamW(DecoupledWeightDecayExtension, tf.keras.optimizers.Adam):
+ def __init__(self, weight_decay, *args, **kwargs):
+ super(AdamW, self).__init__(weight_decay, *args, **kwargs).
+```
+
+Note: this extension decays weights BEFORE applying the update based
+on the gradient, i.e. this extension only has the desired behaviour for
+optimizers which do not depend on the value of'var' in the update step!
+
+Note: when applying a decay to the learning rate, be sure to manually apply
+the decay to the `weight_decay` as well. For example:
+
+```python
+step = tf.Variable(0, trainable=False)
+schedule = tf.optimizers.schedules.PiecewiseConstantDecay(
+ [10000, 15000], [1e-0, 1e-1, 1e-2])
+# lr and wd can be a function or a tensor
+lr = 1e-1 * schedule(step)
+wd = lambda: 1e-4 * schedule(step)
+
+# ...
+
+optimizer = tfa.optimizers.AdamW(learning_rate=lr, weight_decay=wd)
+```
+
+__init__
+
+``` python
+__init__(
+ weight_decay,
+ **kwargs
+)
+```
+
+Extension class that adds weight decay to an optimizer.
+
+
+#### Args:
+
+
+* `weight_decay`: A `Tensor` or a floating point value, the factor by
+ which a variable is decayed in the update step.
+* `**kwargs`: Optional list or tuple or set of `Variable` objects to
+ decay.
+
+
+
+## Methods
+
+apply_gradients
+
+``` python
+apply_gradients(
+ grads_and_vars,
+ name=None,
+ decay_var_list=None
+)
+```
+
+Apply gradients to variables.
+
+This is the second part of `minimize()`. It returns an `Operation` that
+applies gradients.
+
+#### Args:
+
+
+* `grads_and_vars`: List of (gradient, variable) pairs.
+* `name`: Optional name for the returned operation. Default to the
+ name passed to the `Optimizer` constructor.
+* `decay_var_list`: Optional list of variables to be decayed. Defaults
+ to all variables in var_list.
+
+#### Returns:
+
+An `Operation` that applies the specified gradients. If
+`global_step` was not None, that operation also increments
+`global_step`.
+
+
+#### Raises:
+
+
+* `TypeError`: If `grads_and_vars` is malformed.
+* `ValueError`: If none of the variables have gradients.
+
+get_config
+
+``` python
+get_config()
+```
+
+
+
+
+minimize
+
+``` python
+minimize(
+ loss,
+ var_list,
+ grad_loss=None,
+ name=None,
+ decay_var_list=None
+)
+```
+
+Minimize `loss` by updating `var_list`.
+
+This method simply computes gradient using `tf.GradientTape` and calls
+`apply_gradients()`. If you want to process the gradient before
+applying then call `tf.GradientTape` and `apply_gradients()` explicitly
+instead of using this function.
+
+#### Args:
+
+
+* `loss`: A callable taking no arguments which returns the value to
+ minimize.
+* `var_list`: list or tuple of `Variable` objects to update to
+ minimize `loss`, or a callable returning the list or tuple of
+ `Variable` objects. Use callable when the variable list would
+ otherwise be incomplete before `minimize` since the variables
+ are created at the first time `loss` is called.
+* `grad_loss`: Optional. A `Tensor` holding the gradient computed for
+ `loss`.
+* `decay_var_list`: Optional list of variables to be decayed. Defaults
+ to all variables in var_list.
+* `name`: Optional name for the returned operation.
+
+#### Returns:
+
+An Operation that updates the variables in `var_list`. If
+`global_step` was not `None`, that operation also increments
+`global_step`.
+
+
+#### Raises:
+
+
+* `ValueError`: If some of the variables are not `Variable` objects.
+
+
+
diff --git a/docs/api_docs/python/tfa/rnn.md b/docs/api_docs/python/tfa/rnn.md
new file mode 100644
index 0000000000..07eac378ce
--- /dev/null
+++ b/docs/api_docs/python/tfa/rnn.md
@@ -0,0 +1,26 @@
+
+
+
+
+
+# Module: tfa.rnn
+
+Customized RNN cells.
+
+
+
+Defined in [`rnn/__init__.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/rnn/__init__.py).
+
+
+
+
+## Modules
+
+[`cell`](../tfa/rnn/cell.md) module: Module for RNN Cells.
+
+## Classes
+
+[`class LayerNormLSTMCell`](../tfa/rnn/LayerNormLSTMCell.md): LSTM cell with layer normalization and recurrent dropout.
+
+[`class NASCell`](../tfa/rnn/NASCell.md): Neural Architecture Search (NAS) recurrent network cell.
+
diff --git a/docs/api_docs/python/tfa/rnn/LayerNormLSTMCell.md b/docs/api_docs/python/tfa/rnn/LayerNormLSTMCell.md
new file mode 100644
index 0000000000..4d13c7dd0a
--- /dev/null
+++ b/docs/api_docs/python/tfa/rnn/LayerNormLSTMCell.md
@@ -0,0 +1,995 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+# tfa.rnn.LayerNormLSTMCell
+
+## Class `LayerNormLSTMCell`
+
+LSTM cell with layer normalization and recurrent dropout.
+
+
+
+### Aliases:
+
+* Class `tfa.rnn.LayerNormLSTMCell`
+* Class `tfa.rnn.cell.LayerNormLSTMCell`
+
+
+
+Defined in [`rnn/cell.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/rnn/cell.py).
+
+
+
+This class adds layer normalization and recurrent dropout to a LSTM unit.
+Layer normalization implementation is based on:
+
+ https://arxiv.org/abs/1607.06450.
+
+"Layer Normalization" Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton
+
+and is applied before the internal nonlinearities.
+Recurrent dropout is based on:
+
+ https://arxiv.org/abs/1603.05118
+
+"Recurrent Dropout without Memory Loss"
+Stanislau Semeniuta, Aliaksei Severyn, Erhardt Barth.
+
+__init__
+
+``` python
+__init__(
+ units,
+ activation='tanh',
+ recurrent_activation='sigmoid',
+ use_bias=True,
+ kernel_initializer='glorot_uniform',
+ recurrent_initializer='orthogonal',
+ bias_initializer='zeros',
+ unit_forget_bias=True,
+ kernel_regularizer=None,
+ recurrent_regularizer=None,
+ bias_regularizer=None,
+ kernel_constraint=None,
+ recurrent_constraint=None,
+ bias_constraint=None,
+ dropout=0.0,
+ recurrent_dropout=0.0,
+ norm_gamma_initializer='ones',
+ norm_beta_initializer='zeros',
+ norm_epsilon=0.001,
+ **kwargs
+)
+```
+
+Initializes the LSTM cell.
+
+
+#### Args:
+
+
+* `units`: Positive integer, dimensionality of the output space.
+* `activation`: Activation function to use. Default: hyperbolic tangent
+ (`tanh`). If you pass `None`, no activation is applied (ie.
+ "linear" activation: `a(x) = x`).
+* `recurrent_activation`: Activation function to use for the recurrent
+ step. Default: sigmoid (`sigmoid`). If you pass `None`, no
+ activation is applied (ie. "linear" activation: `a(x) = x`).
+* `use_bias`: Boolean, whether the layer uses a bias vector.
+* `kernel_initializer`: Initializer for the `kernel` weights matrix, used
+ for the linear transformation of the inputs.
+* `recurrent_initializer`: Initializer for the `recurrent_kernel` weights
+ matrix, used for the linear transformation of the recurrent state.
+* `bias_initializer`: Initializer for the bias vector.
+* `unit_forget_bias`: Boolean. If True, add 1 to the bias of the forget
+ gate at initialization. Setting it to true will also force
+ `bias_initializer="zeros"`. This is recommended in [Jozefowicz et
+ al.](http://www.jmlr.org/proceedings/papers/v37/jozefowicz15.pdf)
+* `kernel_regularizer`: Regularizer function applied to the `kernel`
+ weights matrix.
+* `recurrent_regularizer`: Regularizer function applied to
+ the `recurrent_kernel` weights matrix.
+* `bias_regularizer`: Regularizer function applied to the bias vector.
+* `kernel_constraint`: Constraint function applied to the `kernel`
+ weights matrix.
+* `recurrent_constraint`: Constraint function applied to the
+ `recurrent_kernel` weights matrix.
+* `bias_constraint`: Constraint function applied to the bias vector.
+* `dropout`: Float between 0 and 1. Fraction of the units to drop for the
+ linear transformation of the inputs.
+* `recurrent_dropout`: Float between 0 and 1. Fraction of the units to
+ drop for the linear transformation of the recurrent state.
+* `norm_gamma_initializer`: Initializer for the layer normalization gain
+ initial value.
+* `norm_beta_initializer`: Initializer for the layer normalization shift
+ initial value.
+* `norm_epsilon`: Float, the epsilon value for normalization layers.
+* `**kwargs`: Dict, the other keyword arguments for layer creation.
+
+
+
+## Properties
+
+activity_regularizer
+
+Optional regularizer function for the output of this layer.
+
+
+dtype
+
+
+
+
+dynamic
+
+
+
+
+
+
+Retrieves the input tensor(s) of a layer.
+
+Only applicable if the layer has exactly one input,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Input tensor or list of input tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+* `AttributeError`: If no inbound nodes are found.
+
+
+
+Retrieves the input mask tensor(s) of a layer.
+
+Only applicable if the layer has exactly one inbound node,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Input mask tensor (potentially None) or list of input
+mask tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+
+
+Retrieves the input shape(s) of a layer.
+
+Only applicable if the layer has exactly one input,
+i.e. if it is connected to one incoming layer, or if all inputs
+have the same shape.
+
+#### Returns:
+
+Input shape, as an integer shape tuple
+(or list of shape tuples, one tuple per input tensor).
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer has no defined input_shape.
+* `RuntimeError`: if called in Eager mode.
+
+
+
+
+
+
+losses
+
+Losses which are associated with this `Layer`.
+
+Variable regularization tensors are created when this property is accessed,
+so it is eager safe: accessing `losses` under a `tf.GradientTape` will
+propagate gradients back to the corresponding variables.
+
+#### Returns:
+
+A list of tensors.
+
+
+metrics
+
+
+
+
+name
+
+
+
+
+name_scope
+
+Returns a `tf.name_scope` instance for this class.
+
+
+non_trainable_variables
+
+
+
+
+non_trainable_weights
+
+
+
+
+output
+
+Retrieves the output tensor(s) of a layer.
+
+Only applicable if the layer has exactly one output,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Output tensor or list of output tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to more than one incoming
+ layers.
+* `RuntimeError`: if called in Eager mode.
+
+output_mask
+
+Retrieves the output mask tensor(s) of a layer.
+
+Only applicable if the layer has exactly one inbound node,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Output mask tensor (potentially None) or list of output
+mask tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+output_shape
+
+Retrieves the output shape(s) of a layer.
+
+Only applicable if the layer has one output,
+or if all outputs have the same shape.
+
+#### Returns:
+
+Output shape, as an integer shape tuple
+(or list of shape tuples, one tuple per output tensor).
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer has no defined output shape.
+* `RuntimeError`: if called in Eager mode.
+
+submodules
+
+Sequence of all sub-modules.
+
+Submodules are modules which are properties of this module, or found as
+properties of modules which are properties of this module (and so on).
+
+```
+a = tf.Module()
+b = tf.Module()
+c = tf.Module()
+a.b = b
+b.c = c
+assert list(a.submodules) == [b, c]
+assert list(b.submodules) == [c]
+assert list(c.submodules) == []
+```
+
+#### Returns:
+
+A sequence of all submodules.
+
+
+trainable
+
+
+
+
+trainable_variables
+
+
+
+
+trainable_weights
+
+
+
+
+updates
+
+
+
+
+variables
+
+Returns the list of all layer variables/weights.
+
+Alias of `self.weights`.
+
+#### Returns:
+
+A list of variables.
+
+
+weights
+
+Returns the list of all layer variables/weights.
+
+
+#### Returns:
+
+A list of variables.
+
+
+
+
+## Methods
+
+__call__
+
+``` python
+__call__(
+ inputs,
+ *args,
+ **kwargs
+)
+```
+
+Wraps `call`, applying pre- and post-processing steps.
+
+
+#### Arguments:
+
+
+* `inputs`: input tensor(s).
+* `*args`: additional positional arguments to be passed to `self.call`.
+* `**kwargs`: additional keyword arguments to be passed to `self.call`.
+
+
+#### Returns:
+
+Output tensor(s).
+
+
+
+#### Note:
+
+- The following optional keyword arguments are reserved for specific uses:
+ * `training`: Boolean scalar tensor of Python boolean indicating
+ whether the `call` is meant for training or inference.
+ * `mask`: Boolean input mask.
+- If the layer's `call` method takes a `mask` argument (as some Keras
+ layers do), its default value will be set to the mask generated
+ for `inputs` by the previous layer (if `input` did come from
+ a layer that generated a corresponding mask, i.e. if it came from
+ a Keras layer with masking support.
+
+
+
+#### Raises:
+
+
+* `ValueError`: if the layer's `call` method returns None (an invalid value).
+
+apply
+
+``` python
+apply(
+ inputs,
+ *args,
+ **kwargs
+)
+```
+
+Apply the layer on a input.
+
+This is an alias of `self.__call__`.
+
+#### Arguments:
+
+
+* `inputs`: Input tensor(s).
+* `*args`: additional positional arguments to be passed to `self.call`.
+* `**kwargs`: additional keyword arguments to be passed to `self.call`.
+
+
+#### Returns:
+
+Output tensor(s).
+
+
+build
+
+``` python
+build(input_shape)
+```
+
+
+
+
+compute_mask
+
+``` python
+compute_mask(
+ inputs,
+ mask=None
+)
+```
+
+Computes an output mask tensor.
+
+
+#### Arguments:
+
+
+* `inputs`: Tensor or list of tensors.
+* `mask`: Tensor or list of tensors.
+
+
+#### Returns:
+
+None or a tensor (or list of tensors,
+ one per output tensor of the layer).
+
+
+compute_output_shape
+
+``` python
+compute_output_shape(input_shape)
+```
+
+Computes the output shape of the layer.
+
+Assumes that the layer will be built
+to match that input shape provided.
+
+#### Arguments:
+
+
+* `input_shape`: Shape tuple (tuple of integers)
+ or list of shape tuples (one per output tensor of the layer).
+ Shape tuples can include None for free dimensions,
+ instead of an integer.
+
+
+#### Returns:
+
+An input shape tuple.
+
+
+count_params
+
+``` python
+count_params()
+```
+
+Count the total number of scalars composing the weights.
+
+
+#### Returns:
+
+An integer count.
+
+
+
+#### Raises:
+
+
+* `ValueError`: if the layer isn't yet built
+ (in which case its weights aren't yet defined).
+
+from_config
+
+``` python
+from_config(
+ cls,
+ config
+)
+```
+
+Creates a layer from its config.
+
+This method is the reverse of `get_config`,
+capable of instantiating the same layer from the config
+dictionary. It does not handle layer connectivity
+(handled by Network), nor weights (handled by `set_weights`).
+
+#### Arguments:
+
+
+* `config`: A Python dictionary, typically the
+ output of get_config.
+
+
+#### Returns:
+
+A layer instance.
+
+
+get_config
+
+``` python
+get_config()
+```
+
+
+
+
+get_dropout_mask_for_cell
+
+``` python
+get_dropout_mask_for_cell(
+ inputs,
+ training,
+ count=1
+)
+```
+
+Get the dropout mask for RNN cell's input.
+
+It will create mask based on context if there isn't any existing cached
+mask. If a new mask is generated, it will update the cache in the cell.
+
+#### Args:
+
+
+* `inputs`: the input tensor whose shape will be used to generate dropout
+ mask.
+* `training`: boolean tensor, whether its in training mode, dropout will be
+ ignored in non-training mode.
+* `count`: int, how many dropout mask will be generated. It is useful for cell
+ that has internal weights fused together.
+
+#### Returns:
+
+List of mask tensor, generated or cached mask based on context.
+
+
+get_initial_state
+
+``` python
+get_initial_state(
+ inputs=None,
+ batch_size=None,
+ dtype=None
+)
+```
+
+
+
+
+
+
+``` python
+get_input_at(node_index)
+```
+
+Retrieves the input tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A tensor (or list of tensors if the layer has multiple inputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+
+
+``` python
+get_input_mask_at(node_index)
+```
+
+Retrieves the input mask tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A mask tensor
+(or list of tensors if the layer has multiple inputs).
+
+
+
+
+``` python
+get_input_shape_at(node_index)
+```
+
+Retrieves the input shape(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A shape tuple
+(or list of shape tuples if the layer has multiple inputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_losses_for
+
+``` python
+get_losses_for(inputs)
+```
+
+Retrieves losses relevant to a specific set of inputs.
+
+
+#### Arguments:
+
+
+* `inputs`: Input tensor or list/tuple of input tensors.
+
+
+#### Returns:
+
+List of loss tensors of the layer that depend on `inputs`.
+
+
+get_output_at
+
+``` python
+get_output_at(node_index)
+```
+
+Retrieves the output tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A tensor (or list of tensors if the layer has multiple outputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_output_mask_at
+
+``` python
+get_output_mask_at(node_index)
+```
+
+Retrieves the output mask tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A mask tensor
+(or list of tensors if the layer has multiple outputs).
+
+
+get_output_shape_at
+
+``` python
+get_output_shape_at(node_index)
+```
+
+Retrieves the output shape(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A shape tuple
+(or list of shape tuples if the layer has multiple outputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_recurrent_dropout_mask_for_cell
+
+``` python
+get_recurrent_dropout_mask_for_cell(
+ inputs,
+ training,
+ count=1
+)
+```
+
+Get the recurrent dropout mask for RNN cell.
+
+It will create mask based on context if there isn't any existing cached
+mask. If a new mask is generated, it will update the cache in the cell.
+
+#### Args:
+
+
+* `inputs`: the input tensor whose shape will be used to generate dropout
+ mask.
+* `training`: boolean tensor, whether its in training mode, dropout will be
+ ignored in non-training mode.
+* `count`: int, how many dropout mask will be generated. It is useful for cell
+ that has internal weights fused together.
+
+#### Returns:
+
+List of mask tensor, generated or cached mask based on context.
+
+
+get_updates_for
+
+``` python
+get_updates_for(inputs)
+```
+
+Retrieves updates relevant to a specific set of inputs.
+
+
+#### Arguments:
+
+
+* `inputs`: Input tensor or list/tuple of input tensors.
+
+
+#### Returns:
+
+List of update ops of the layer that depend on `inputs`.
+
+
+get_weights
+
+``` python
+get_weights()
+```
+
+Returns the current weights of the layer.
+
+
+#### Returns:
+
+Weights values as a list of numpy arrays.
+
+
+reset_dropout_mask
+
+``` python
+reset_dropout_mask()
+```
+
+Reset the cached dropout masks if any.
+
+This is important for the RNN layer to invoke this in it call() method so
+that the cached mask is cleared before calling the cell.call(). The mask
+should be cached across the timestep within the same batch, but shouldn't
+be cached between batches. Otherwise it will introduce unreasonable bias
+against certain index of data within the batch.
+
+reset_recurrent_dropout_mask
+
+``` python
+reset_recurrent_dropout_mask()
+```
+
+Reset the cached recurrent dropout masks if any.
+
+This is important for the RNN layer to invoke this in it call() method so
+that the cached mask is cleared before calling the cell.call(). The mask
+should be cached across the timestep within the same batch, but shouldn't
+be cached between batches. Otherwise it will introduce unreasonable bias
+against certain index of data within the batch.
+
+set_weights
+
+``` python
+set_weights(weights)
+```
+
+Sets the weights of the layer, from Numpy arrays.
+
+
+#### Arguments:
+
+
+* `weights`: a list of Numpy arrays. The number
+ of arrays and their shape must match
+ number of the dimensions of the weights
+ of the layer (i.e. it should match the
+ output of `get_weights`).
+
+
+#### Raises:
+
+
+* `ValueError`: If the provided weights list does not match the
+ layer's specifications.
+
+with_name_scope
+
+``` python
+with_name_scope(
+ cls,
+ method
+)
+```
+
+Decorator to automatically enter the module name scope.
+
+```
+class MyModule(tf.Module):
+ @tf.Module.with_name_scope
+ def __call__(self, x):
+ if not hasattr(self, 'w'):
+ self.w = tf.Variable(tf.random.normal([x.shape[1], 64]))
+ return tf.matmul(x, self.w)
+```
+
+Using the above module would produce `tf.Variable`s and `tf.Tensor`s whose
+names included the module name:
+
+```
+mod = MyModule()
+mod(tf.ones([8, 32]))
+# ==>
+mod.w
+# ==>
+```
+
+#### Args:
+
+
+* `method`: The method to wrap.
+
+
+#### Returns:
+
+The original method wrapped such that it enters the module's name scope.
+
+
+
+
diff --git a/docs/api_docs/python/tfa/rnn/NASCell.md b/docs/api_docs/python/tfa/rnn/NASCell.md
new file mode 100644
index 0000000000..eb101fabd8
--- /dev/null
+++ b/docs/api_docs/python/tfa/rnn/NASCell.md
@@ -0,0 +1,871 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+# tfa.rnn.NASCell
+
+## Class `NASCell`
+
+Neural Architecture Search (NAS) recurrent network cell.
+
+
+
+### Aliases:
+
+* Class `tfa.rnn.NASCell`
+* Class `tfa.rnn.cell.NASCell`
+
+
+
+Defined in [`rnn/cell.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/rnn/cell.py).
+
+
+
+This implements the recurrent cell from the paper:
+
+ https://arxiv.org/abs/1611.01578
+
+Barret Zoph and Quoc V. Le.
+"Neural Architecture Search with Reinforcement Learning" Proc. ICLR 2017.
+
+The class uses an optional projection layer.
+
+__init__
+
+``` python
+__init__(
+ units,
+ projection=None,
+ use_bias=False,
+ kernel_initializer='glorot_uniform',
+ recurrent_initializer='glorot_uniform',
+ projection_initializer='glorot_uniform',
+ bias_initializer='zeros',
+ **kwargs
+)
+```
+
+Initialize the parameters for a NAS cell.
+
+
+#### Args:
+
+
+* `units`: int, The number of units in the NAS cell.
+* `projection`: (optional) int, The output dimensionality for the
+ projection matrices. If None, no projection is performed.
+* `use_bias`: (optional) bool, If True then use biases within the cell.
+ This is False by default.
+* `kernel_initializer`: Initializer for kernel weight.
+* `recurrent_initializer`: Initializer for recurrent kernel weight.
+* `projection_initializer`: Initializer for projection weight, used when
+ projection is not None.
+* `bias_initializer`: Initializer for bias, used when use_bias is True.
+* `**kwargs`: Additional keyword arguments.
+
+
+
+## Properties
+
+activity_regularizer
+
+Optional regularizer function for the output of this layer.
+
+
+dtype
+
+
+
+
+dynamic
+
+
+
+
+
+
+Retrieves the input tensor(s) of a layer.
+
+Only applicable if the layer has exactly one input,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Input tensor or list of input tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+* `AttributeError`: If no inbound nodes are found.
+
+
+
+Retrieves the input mask tensor(s) of a layer.
+
+Only applicable if the layer has exactly one inbound node,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Input mask tensor (potentially None) or list of input
+mask tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+
+
+Retrieves the input shape(s) of a layer.
+
+Only applicable if the layer has exactly one input,
+i.e. if it is connected to one incoming layer, or if all inputs
+have the same shape.
+
+#### Returns:
+
+Input shape, as an integer shape tuple
+(or list of shape tuples, one tuple per input tensor).
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer has no defined input_shape.
+* `RuntimeError`: if called in Eager mode.
+
+
+
+
+
+
+losses
+
+Losses which are associated with this `Layer`.
+
+Variable regularization tensors are created when this property is accessed,
+so it is eager safe: accessing `losses` under a `tf.GradientTape` will
+propagate gradients back to the corresponding variables.
+
+#### Returns:
+
+A list of tensors.
+
+
+metrics
+
+
+
+
+name
+
+
+
+
+name_scope
+
+Returns a `tf.name_scope` instance for this class.
+
+
+non_trainable_variables
+
+
+
+
+non_trainable_weights
+
+
+
+
+output
+
+Retrieves the output tensor(s) of a layer.
+
+Only applicable if the layer has exactly one output,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Output tensor or list of output tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to more than one incoming
+ layers.
+* `RuntimeError`: if called in Eager mode.
+
+output_mask
+
+Retrieves the output mask tensor(s) of a layer.
+
+Only applicable if the layer has exactly one inbound node,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Output mask tensor (potentially None) or list of output
+mask tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+output_shape
+
+Retrieves the output shape(s) of a layer.
+
+Only applicable if the layer has one output,
+or if all outputs have the same shape.
+
+#### Returns:
+
+Output shape, as an integer shape tuple
+(or list of shape tuples, one tuple per output tensor).
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer has no defined output shape.
+* `RuntimeError`: if called in Eager mode.
+
+output_size
+
+
+
+
+state_size
+
+
+
+
+submodules
+
+Sequence of all sub-modules.
+
+Submodules are modules which are properties of this module, or found as
+properties of modules which are properties of this module (and so on).
+
+```
+a = tf.Module()
+b = tf.Module()
+c = tf.Module()
+a.b = b
+b.c = c
+assert list(a.submodules) == [b, c]
+assert list(b.submodules) == [c]
+assert list(c.submodules) == []
+```
+
+#### Returns:
+
+A sequence of all submodules.
+
+
+trainable
+
+
+
+
+trainable_variables
+
+
+
+
+trainable_weights
+
+
+
+
+updates
+
+
+
+
+variables
+
+Returns the list of all layer variables/weights.
+
+Alias of `self.weights`.
+
+#### Returns:
+
+A list of variables.
+
+
+weights
+
+Returns the list of all layer variables/weights.
+
+
+#### Returns:
+
+A list of variables.
+
+
+
+
+## Methods
+
+__call__
+
+``` python
+__call__(
+ inputs,
+ *args,
+ **kwargs
+)
+```
+
+Wraps `call`, applying pre- and post-processing steps.
+
+
+#### Arguments:
+
+
+* `inputs`: input tensor(s).
+* `*args`: additional positional arguments to be passed to `self.call`.
+* `**kwargs`: additional keyword arguments to be passed to `self.call`.
+
+
+#### Returns:
+
+Output tensor(s).
+
+
+
+#### Note:
+
+- The following optional keyword arguments are reserved for specific uses:
+ * `training`: Boolean scalar tensor of Python boolean indicating
+ whether the `call` is meant for training or inference.
+ * `mask`: Boolean input mask.
+- If the layer's `call` method takes a `mask` argument (as some Keras
+ layers do), its default value will be set to the mask generated
+ for `inputs` by the previous layer (if `input` did come from
+ a layer that generated a corresponding mask, i.e. if it came from
+ a Keras layer with masking support.
+
+
+
+#### Raises:
+
+
+* `ValueError`: if the layer's `call` method returns None (an invalid value).
+
+apply
+
+``` python
+apply(
+ inputs,
+ *args,
+ **kwargs
+)
+```
+
+Apply the layer on a input.
+
+This is an alias of `self.__call__`.
+
+#### Arguments:
+
+
+* `inputs`: Input tensor(s).
+* `*args`: additional positional arguments to be passed to `self.call`.
+* `**kwargs`: additional keyword arguments to be passed to `self.call`.
+
+
+#### Returns:
+
+Output tensor(s).
+
+
+build
+
+``` python
+build(inputs_shape)
+```
+
+
+
+
+compute_mask
+
+``` python
+compute_mask(
+ inputs,
+ mask=None
+)
+```
+
+Computes an output mask tensor.
+
+
+#### Arguments:
+
+
+* `inputs`: Tensor or list of tensors.
+* `mask`: Tensor or list of tensors.
+
+
+#### Returns:
+
+None or a tensor (or list of tensors,
+ one per output tensor of the layer).
+
+
+compute_output_shape
+
+``` python
+compute_output_shape(input_shape)
+```
+
+Computes the output shape of the layer.
+
+Assumes that the layer will be built
+to match that input shape provided.
+
+#### Arguments:
+
+
+* `input_shape`: Shape tuple (tuple of integers)
+ or list of shape tuples (one per output tensor of the layer).
+ Shape tuples can include None for free dimensions,
+ instead of an integer.
+
+
+#### Returns:
+
+An input shape tuple.
+
+
+count_params
+
+``` python
+count_params()
+```
+
+Count the total number of scalars composing the weights.
+
+
+#### Returns:
+
+An integer count.
+
+
+
+#### Raises:
+
+
+* `ValueError`: if the layer isn't yet built
+ (in which case its weights aren't yet defined).
+
+from_config
+
+``` python
+from_config(
+ cls,
+ config
+)
+```
+
+Creates a layer from its config.
+
+This method is the reverse of `get_config`,
+capable of instantiating the same layer from the config
+dictionary. It does not handle layer connectivity
+(handled by Network), nor weights (handled by `set_weights`).
+
+#### Arguments:
+
+
+* `config`: A Python dictionary, typically the
+ output of get_config.
+
+
+#### Returns:
+
+A layer instance.
+
+
+get_config
+
+``` python
+get_config()
+```
+
+
+
+
+get_initial_state
+
+``` python
+get_initial_state(
+ inputs=None,
+ batch_size=None,
+ dtype=None
+)
+```
+
+
+
+
+
+
+``` python
+get_input_at(node_index)
+```
+
+Retrieves the input tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A tensor (or list of tensors if the layer has multiple inputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+
+
+``` python
+get_input_mask_at(node_index)
+```
+
+Retrieves the input mask tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A mask tensor
+(or list of tensors if the layer has multiple inputs).
+
+
+
+
+``` python
+get_input_shape_at(node_index)
+```
+
+Retrieves the input shape(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A shape tuple
+(or list of shape tuples if the layer has multiple inputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_losses_for
+
+``` python
+get_losses_for(inputs)
+```
+
+Retrieves losses relevant to a specific set of inputs.
+
+
+#### Arguments:
+
+
+* `inputs`: Input tensor or list/tuple of input tensors.
+
+
+#### Returns:
+
+List of loss tensors of the layer that depend on `inputs`.
+
+
+get_output_at
+
+``` python
+get_output_at(node_index)
+```
+
+Retrieves the output tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A tensor (or list of tensors if the layer has multiple outputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_output_mask_at
+
+``` python
+get_output_mask_at(node_index)
+```
+
+Retrieves the output mask tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A mask tensor
+(or list of tensors if the layer has multiple outputs).
+
+
+get_output_shape_at
+
+``` python
+get_output_shape_at(node_index)
+```
+
+Retrieves the output shape(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A shape tuple
+(or list of shape tuples if the layer has multiple outputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_updates_for
+
+``` python
+get_updates_for(inputs)
+```
+
+Retrieves updates relevant to a specific set of inputs.
+
+
+#### Arguments:
+
+
+* `inputs`: Input tensor or list/tuple of input tensors.
+
+
+#### Returns:
+
+List of update ops of the layer that depend on `inputs`.
+
+
+get_weights
+
+``` python
+get_weights()
+```
+
+Returns the current weights of the layer.
+
+
+#### Returns:
+
+Weights values as a list of numpy arrays.
+
+
+set_weights
+
+``` python
+set_weights(weights)
+```
+
+Sets the weights of the layer, from Numpy arrays.
+
+
+#### Arguments:
+
+
+* `weights`: a list of Numpy arrays. The number
+ of arrays and their shape must match
+ number of the dimensions of the weights
+ of the layer (i.e. it should match the
+ output of `get_weights`).
+
+
+#### Raises:
+
+
+* `ValueError`: If the provided weights list does not match the
+ layer's specifications.
+
+with_name_scope
+
+``` python
+with_name_scope(
+ cls,
+ method
+)
+```
+
+Decorator to automatically enter the module name scope.
+
+```
+class MyModule(tf.Module):
+ @tf.Module.with_name_scope
+ def __call__(self, x):
+ if not hasattr(self, 'w'):
+ self.w = tf.Variable(tf.random.normal([x.shape[1], 64]))
+ return tf.matmul(x, self.w)
+```
+
+Using the above module would produce `tf.Variable`s and `tf.Tensor`s whose
+names included the module name:
+
+```
+mod = MyModule()
+mod(tf.ones([8, 32]))
+# ==>
+mod.w
+# ==>
+```
+
+#### Args:
+
+
+* `method`: The method to wrap.
+
+
+#### Returns:
+
+The original method wrapped such that it enters the module's name scope.
+
+
+
+
diff --git a/docs/api_docs/python/tfa/rnn/cell.md b/docs/api_docs/python/tfa/rnn/cell.md
new file mode 100644
index 0000000000..d271c28ee1
--- /dev/null
+++ b/docs/api_docs/python/tfa/rnn/cell.md
@@ -0,0 +1,22 @@
+
+
+
+
+
+# Module: tfa.rnn.cell
+
+Module for RNN Cells.
+
+
+
+Defined in [`rnn/cell.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/rnn/cell.py).
+
+
+
+
+## Classes
+
+[`class LayerNormLSTMCell`](../../tfa/rnn/LayerNormLSTMCell.md): LSTM cell with layer normalization and recurrent dropout.
+
+[`class NASCell`](../../tfa/rnn/NASCell.md): Neural Architecture Search (NAS) recurrent network cell.
+
diff --git a/docs/api_docs/python/tfa/seq2seq.md b/docs/api_docs/python/tfa/seq2seq.md
new file mode 100644
index 0000000000..6b734a1491
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq.md
@@ -0,0 +1,96 @@
+
+
+
+
+
+# Module: tfa.seq2seq
+
+Ops for building neural network sequence to sequence decoders and losses.
+
+
+
+Defined in [`seq2seq/__init__.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/__init__.py).
+
+
+
+
+## Modules
+
+[`attention_wrapper`](../tfa/seq2seq/attention_wrapper.md) module: A powerful dynamic attention wrapper object.
+
+[`basic_decoder`](../tfa/seq2seq/basic_decoder.md) module: A class of Decoders that may sample to generate the next input.
+
+[`beam_search_decoder`](../tfa/seq2seq/beam_search_decoder.md) module: A decoder that performs beam search.
+
+[`decoder`](../tfa/seq2seq/decoder.md) module: Seq2seq layer operations for use in neural networks.
+
+[`loss`](../tfa/seq2seq/loss.md) module: Seq2seq loss operations for use in sequence models.
+
+[`sampler`](../tfa/seq2seq/sampler.md) module: A library of sampler for use with SamplingDecoders.
+
+## Classes
+
+[`class AttentionMechanism`](../tfa/seq2seq/AttentionMechanism.md)
+
+[`class AttentionWrapper`](../tfa/seq2seq/AttentionWrapper.md): Wraps another `RNNCell` with attention.
+
+[`class AttentionWrapperState`](../tfa/seq2seq/AttentionWrapperState.md): `namedtuple` storing the state of a `AttentionWrapper`.
+
+[`class BahdanauAttention`](../tfa/seq2seq/BahdanauAttention.md): Implements Bahdanau-style (additive) attention.
+
+[`class BahdanauMonotonicAttention`](../tfa/seq2seq/BahdanauMonotonicAttention.md): Monotonic attention mechanism with Bahadanau-style energy function.
+
+[`class BaseDecoder`](../tfa/seq2seq/BaseDecoder.md): An RNN Decoder that is based on a Keras layer.
+
+[`class BasicDecoder`](../tfa/seq2seq/BasicDecoder.md): Basic sampling decoder.
+
+[`class BasicDecoderOutput`](../tfa/seq2seq/BasicDecoderOutput.md)
+
+[`class BeamSearchDecoder`](../tfa/seq2seq/BeamSearchDecoder.md): BeamSearch sampling decoder.
+
+[`class BeamSearchDecoderOutput`](../tfa/seq2seq/BeamSearchDecoderOutput.md)
+
+[`class BeamSearchDecoderState`](../tfa/seq2seq/BeamSearchDecoderState.md)
+
+[`class CustomSampler`](../tfa/seq2seq/CustomSampler.md): Base abstract class that allows the user to customize sampling.
+
+[`class Decoder`](../tfa/seq2seq/Decoder.md): An RNN Decoder abstract interface object.
+
+[`class FinalBeamSearchDecoderOutput`](../tfa/seq2seq/FinalBeamSearchDecoderOutput.md): Final outputs returned by the beam search after all decoding is
+
+[`class GreedyEmbeddingSampler`](../tfa/seq2seq/GreedyEmbeddingSampler.md): A sampler for use during inference.
+
+[`class InferenceSampler`](../tfa/seq2seq/InferenceSampler.md): A helper to use during inference with a custom sampling function.
+
+[`class LuongAttention`](../tfa/seq2seq/LuongAttention.md): Implements Luong-style (multiplicative) attention scoring.
+
+[`class LuongMonotonicAttention`](../tfa/seq2seq/LuongMonotonicAttention.md): Monotonic attention mechanism with Luong-style energy function.
+
+[`class SampleEmbeddingSampler`](../tfa/seq2seq/SampleEmbeddingSampler.md): A sampler for use during inference.
+
+[`class Sampler`](../tfa/seq2seq/Sampler.md): Interface for implementing sampling in seq2seq decoders.
+
+[`class ScheduledEmbeddingTrainingSampler`](../tfa/seq2seq/ScheduledEmbeddingTrainingSampler.md): A training sampler that adds scheduled sampling.
+
+[`class ScheduledOutputTrainingSampler`](../tfa/seq2seq/ScheduledOutputTrainingSampler.md): A training sampler that adds scheduled sampling directly to outputs.
+
+[`class SequenceLoss`](../tfa/seq2seq/SequenceLoss.md): Weighted cross-entropy loss for a sequence of logits.
+
+[`class TrainingSampler`](../tfa/seq2seq/TrainingSampler.md): A Sampler for use during training.
+
+## Functions
+
+[`dynamic_decode(...)`](../tfa/seq2seq/dynamic_decode.md): Perform dynamic decoding with `decoder`.
+
+[`gather_tree_from_array(...)`](../tfa/seq2seq/gather_tree_from_array.md): Calculates the full beams for `TensorArray`s.
+
+[`hardmax(...)`](../tfa/seq2seq/hardmax.md): Returns batched one-hot vectors.
+
+[`monotonic_attention(...)`](../tfa/seq2seq/monotonic_attention.md): Compute monotonic attention distribution from choosing probabilities.
+
+[`safe_cumprod(...)`](../tfa/seq2seq/safe_cumprod.md): Computes cumprod of x in logspace using cumsum to avoid underflow.
+
+[`sequence_loss(...)`](../tfa/seq2seq/sequence_loss.md): Weighted cross-entropy loss for a sequence of logits.
+
+[`tile_batch(...)`](../tfa/seq2seq/tile_batch.md): Tile the batch dimension of a (possibly nested structure of) tensor(s)
+
diff --git a/docs/api_docs/python/tfa/seq2seq/AttentionMechanism.md b/docs/api_docs/python/tfa/seq2seq/AttentionMechanism.md
new file mode 100644
index 0000000000..ef8158bd10
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/AttentionMechanism.md
@@ -0,0 +1,41 @@
+
+
+
+
+
+
+
+# tfa.seq2seq.AttentionMechanism
+
+## Class `AttentionMechanism`
+
+
+
+
+
+### Aliases:
+
+* Class `tfa.seq2seq.AttentionMechanism`
+* Class `tfa.seq2seq.attention_wrapper.AttentionMechanism`
+
+
+
+Defined in [`seq2seq/attention_wrapper.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/attention_wrapper.py).
+
+
+
+
+## Properties
+
+alignments_size
+
+
+
+
+state_size
+
+
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/AttentionWrapper.md b/docs/api_docs/python/tfa/seq2seq/AttentionWrapper.md
new file mode 100644
index 0000000000..f45fe89c7b
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/AttentionWrapper.md
@@ -0,0 +1,1000 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+# tfa.seq2seq.AttentionWrapper
+
+## Class `AttentionWrapper`
+
+Wraps another `RNNCell` with attention.
+
+
+
+### Aliases:
+
+* Class `tfa.seq2seq.AttentionWrapper`
+* Class `tfa.seq2seq.attention_wrapper.AttentionWrapper`
+
+
+
+Defined in [`seq2seq/attention_wrapper.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/attention_wrapper.py).
+
+
+
+
+__init__
+
+``` python
+__init__(
+ cell,
+ attention_mechanism,
+ attention_layer_size=None,
+ alignment_history=False,
+ cell_input_fn=None,
+ output_attention=True,
+ initial_cell_state=None,
+ name=None,
+ attention_layer=None,
+ attention_fn=None
+)
+```
+
+Construct the `AttentionWrapper`.
+
+**NOTE** If you are using the `BeamSearchDecoder` with a cell wrapped
+in `AttentionWrapper`, then you must ensure that:
+
+- The encoder output has been tiled to `beam_width` via
+ `tf.contrib.seq2seq.tile_batch` (NOT `tf.tile`).
+- The `batch_size` argument passed to the `get_initial_state` method of
+ this wrapper is equal to `true_batch_size * beam_width`.
+- The initial state created with `get_initial_state` above contains a
+ `cell_state` value containing properly tiled final state from the
+ encoder.
+
+#### An example:
+
+
+
+```
+tiled_encoder_outputs = tf.contrib.seq2seq.tile_batch(
+ encoder_outputs, multiplier=beam_width)
+tiled_encoder_final_state = tf.conrib.seq2seq.tile_batch(
+ encoder_final_state, multiplier=beam_width)
+tiled_sequence_length = tf.contrib.seq2seq.tile_batch(
+ sequence_length, multiplier=beam_width)
+attention_mechanism = MyFavoriteAttentionMechanism(
+ num_units=attention_depth,
+ memory=tiled_inputs,
+ memory_sequence_length=tiled_sequence_length)
+attention_cell = AttentionWrapper(cell, attention_mechanism, ...)
+decoder_initial_state = attention_cell.get_initial_state(
+ batch_size=true_batch_size * beam_width, dtype=dtype)
+decoder_initial_state = decoder_initial_state.clone(
+ cell_state=tiled_encoder_final_state)
+```
+
+#### Args:
+
+
+* `cell`: An instance of `RNNCell`.
+* `attention_mechanism`: A list of `AttentionMechanism` instances or a
+ single instance.
+* `attention_layer_size`: A list of Python integers or a single Python
+ integer, the depth of the attention (output) layer(s). If None
+ (default), use the context as attention at each time step.
+ Otherwise, feed the context and cell output into the attention
+ layer to generate attention at each time step. If
+ attention_mechanism is a list, attention_layer_size must be a list
+ of the same length. If attention_layer is set, this must be None.
+ If attention_fn is set, it must guaranteed that the outputs of
+ attention_fn also meet the above requirements.
+* `alignment_history`: Python boolean, whether to store alignment history
+ from all time steps in the final output state (currently stored as
+ a time major `TensorArray` on which you must call `stack()`).
+* `cell_input_fn`: (optional) A `callable`. The default is:
+ `lambda inputs, attention:
+ tf.concat([inputs, attention], -1)`.
+* `output_attention`: Python bool. If `True` (default), the output at
+ each time step is the attention value. This is the behavior of
+ Luong-style attention mechanisms. If `False`, the output at each
+ time step is the output of `cell`. This is the behavior of
+ Bhadanau-style attention mechanisms. In both cases, the
+ `attention` tensor is propagated to the next time step via the
+ state and is used there. This flag only controls whether the
+ attention mechanism is propagated up to the next cell in an RNN
+ stack or to the top RNN output.
+* `initial_cell_state`: The initial state value to use for the cell when
+ the user calls `get_initial_state()`. Note that if this value is
+ provided now, and the user uses a `batch_size` argument of
+ `get_initial_state` which does not match the batch size of
+ `initial_cell_state`, proper behavior is not guaranteed.
+* `name`: Name to use when creating ops.
+* `attention_layer`: A list of `tf.tf.keras.layers.Layer` instances or a
+ single `tf.tf.keras.layers.Layer` instance taking the context
+ and cell output as inputs to generate attention at each time step.
+ If None (default), use the context as attention at each time step.
+ If attention_mechanism is a list, attention_layer must be a list of
+ the same length. If attention_layers_size is set, this must be
+ None.
+* `attention_fn`: An optional callable function that allows users to
+ provide their own customized attention function, which takes input
+ (attention_mechanism, cell_output, attention_state,
+ attention_layer) and outputs (attention, alignments,
+ next_attention_state). If provided, the attention_layer_size should
+ be the size of the outputs of attention_fn.
+
+
+#### Raises:
+
+
+* `TypeError`: `attention_layer_size` is not None and
+ (`attention_mechanism` is a list but `attention_layer_size` is not;
+ or vice versa).
+* `ValueError`: if `attention_layer_size` is not None,
+ `attention_mechanism` is a list, and its length does not match that
+ of `attention_layer_size`; if `attention_layer_size` and
+ `attention_layer` are set simultaneously.
+
+
+
+## Properties
+
+activity_regularizer
+
+Optional regularizer function for the output of this layer.
+
+
+dtype
+
+
+
+
+dynamic
+
+
+
+
+
+
+Retrieves the input tensor(s) of a layer.
+
+Only applicable if the layer has exactly one input,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Input tensor or list of input tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+* `AttributeError`: If no inbound nodes are found.
+
+
+
+Retrieves the input mask tensor(s) of a layer.
+
+Only applicable if the layer has exactly one inbound node,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Input mask tensor (potentially None) or list of input
+mask tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+
+
+Retrieves the input shape(s) of a layer.
+
+Only applicable if the layer has exactly one input,
+i.e. if it is connected to one incoming layer, or if all inputs
+have the same shape.
+
+#### Returns:
+
+Input shape, as an integer shape tuple
+(or list of shape tuples, one tuple per input tensor).
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer has no defined input_shape.
+* `RuntimeError`: if called in Eager mode.
+
+
+
+
+
+
+losses
+
+Losses which are associated with this `Layer`.
+
+Variable regularization tensors are created when this property is accessed,
+so it is eager safe: accessing `losses` under a `tf.GradientTape` will
+propagate gradients back to the corresponding variables.
+
+#### Returns:
+
+A list of tensors.
+
+
+metrics
+
+
+
+
+name
+
+
+
+
+name_scope
+
+Returns a `tf.name_scope` instance for this class.
+
+
+non_trainable_variables
+
+
+
+
+non_trainable_weights
+
+
+
+
+output
+
+Retrieves the output tensor(s) of a layer.
+
+Only applicable if the layer has exactly one output,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Output tensor or list of output tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to more than one incoming
+ layers.
+* `RuntimeError`: if called in Eager mode.
+
+output_mask
+
+Retrieves the output mask tensor(s) of a layer.
+
+Only applicable if the layer has exactly one inbound node,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Output mask tensor (potentially None) or list of output
+mask tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+output_shape
+
+Retrieves the output shape(s) of a layer.
+
+Only applicable if the layer has one output,
+or if all outputs have the same shape.
+
+#### Returns:
+
+Output shape, as an integer shape tuple
+(or list of shape tuples, one tuple per output tensor).
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer has no defined output shape.
+* `RuntimeError`: if called in Eager mode.
+
+output_size
+
+
+
+
+state_size
+
+The `state_size` property of `AttentionWrapper`.
+
+
+#### Returns:
+
+An `AttentionWrapperState` tuple containing shapes used
+by this object.
+
+
+submodules
+
+Sequence of all sub-modules.
+
+Submodules are modules which are properties of this module, or found as
+properties of modules which are properties of this module (and so on).
+
+```
+a = tf.Module()
+b = tf.Module()
+c = tf.Module()
+a.b = b
+b.c = c
+assert list(a.submodules) == [b, c]
+assert list(b.submodules) == [c]
+assert list(c.submodules) == []
+```
+
+#### Returns:
+
+A sequence of all submodules.
+
+
+trainable
+
+
+
+
+trainable_variables
+
+
+
+
+trainable_weights
+
+
+
+
+updates
+
+
+
+
+variables
+
+Returns the list of all layer variables/weights.
+
+Alias of `self.weights`.
+
+#### Returns:
+
+A list of variables.
+
+
+weights
+
+Returns the list of all layer variables/weights.
+
+
+#### Returns:
+
+A list of variables.
+
+
+
+
+## Methods
+
+__call__
+
+``` python
+__call__(
+ inputs,
+ *args,
+ **kwargs
+)
+```
+
+Wraps `call`, applying pre- and post-processing steps.
+
+
+#### Arguments:
+
+
+* `inputs`: input tensor(s).
+* `*args`: additional positional arguments to be passed to `self.call`.
+* `**kwargs`: additional keyword arguments to be passed to `self.call`.
+
+
+#### Returns:
+
+Output tensor(s).
+
+
+
+#### Note:
+
+- The following optional keyword arguments are reserved for specific uses:
+ * `training`: Boolean scalar tensor of Python boolean indicating
+ whether the `call` is meant for training or inference.
+ * `mask`: Boolean input mask.
+- If the layer's `call` method takes a `mask` argument (as some Keras
+ layers do), its default value will be set to the mask generated
+ for `inputs` by the previous layer (if `input` did come from
+ a layer that generated a corresponding mask, i.e. if it came from
+ a Keras layer with masking support.
+
+
+
+#### Raises:
+
+
+* `ValueError`: if the layer's `call` method returns None (an invalid value).
+
+apply
+
+``` python
+apply(
+ inputs,
+ *args,
+ **kwargs
+)
+```
+
+Apply the layer on a input.
+
+This is an alias of `self.__call__`.
+
+#### Arguments:
+
+
+* `inputs`: Input tensor(s).
+* `*args`: additional positional arguments to be passed to `self.call`.
+* `**kwargs`: additional keyword arguments to be passed to `self.call`.
+
+
+#### Returns:
+
+Output tensor(s).
+
+
+build
+
+``` python
+build(input_shape)
+```
+
+Creates the variables of the layer (optional, for subclass implementers).
+
+This is a method that implementers of subclasses of `Layer` or `Model`
+can override if they need a state-creation step in-between
+layer instantiation and layer call.
+
+This is typically used to create the weights of `Layer` subclasses.
+
+#### Arguments:
+
+
+* `input_shape`: Instance of `TensorShape`, or list of instances of
+ `TensorShape` if the layer expects a list of inputs
+ (one instance per input).
+
+compute_mask
+
+``` python
+compute_mask(
+ inputs,
+ mask=None
+)
+```
+
+Computes an output mask tensor.
+
+
+#### Arguments:
+
+
+* `inputs`: Tensor or list of tensors.
+* `mask`: Tensor or list of tensors.
+
+
+#### Returns:
+
+None or a tensor (or list of tensors,
+ one per output tensor of the layer).
+
+
+compute_output_shape
+
+``` python
+compute_output_shape(input_shape)
+```
+
+Computes the output shape of the layer.
+
+Assumes that the layer will be built
+to match that input shape provided.
+
+#### Arguments:
+
+
+* `input_shape`: Shape tuple (tuple of integers)
+ or list of shape tuples (one per output tensor of the layer).
+ Shape tuples can include None for free dimensions,
+ instead of an integer.
+
+
+#### Returns:
+
+An input shape tuple.
+
+
+count_params
+
+``` python
+count_params()
+```
+
+Count the total number of scalars composing the weights.
+
+
+#### Returns:
+
+An integer count.
+
+
+
+#### Raises:
+
+
+* `ValueError`: if the layer isn't yet built
+ (in which case its weights aren't yet defined).
+
+from_config
+
+``` python
+from_config(
+ cls,
+ config
+)
+```
+
+Creates a layer from its config.
+
+This method is the reverse of `get_config`,
+capable of instantiating the same layer from the config
+dictionary. It does not handle layer connectivity
+(handled by Network), nor weights (handled by `set_weights`).
+
+#### Arguments:
+
+
+* `config`: A Python dictionary, typically the
+ output of get_config.
+
+
+#### Returns:
+
+A layer instance.
+
+
+get_config
+
+``` python
+get_config()
+```
+
+Returns the config of the layer.
+
+A layer config is a Python dictionary (serializable)
+containing the configuration of a layer.
+The same layer can be reinstantiated later
+(without its trained weights) from this configuration.
+
+The config of a layer does not include connectivity
+information, nor the layer class name. These are handled
+by `Network` (one layer of abstraction above).
+
+#### Returns:
+
+Python dictionary.
+
+
+get_initial_state
+
+``` python
+get_initial_state(
+ inputs=None,
+ batch_size=None,
+ dtype=None
+)
+```
+
+Return an initial (zero) state tuple for this `AttentionWrapper`.
+
+**NOTE** Please see the initializer documentation for details of how
+to call `get_initial_state` if using an `AttentionWrapper` with a
+`BeamSearchDecoder`.
+
+#### Args:
+
+
+* `inputs`: The inputs that will be fed to this cell.
+* `batch_size`: `0D` integer tensor: the batch size.
+* `dtype`: The internal state data type.
+
+
+#### Returns:
+
+An `AttentionWrapperState` tuple containing zeroed out tensors and,
+possibly, empty `TensorArray` objects.
+
+
+
+#### Raises:
+
+
+* `ValueError`: (or, possibly at runtime, InvalidArgument), if
+ `batch_size` does not match the output size of the encoder passed
+ to the wrapper object at initialization time.
+
+
+
+``` python
+get_input_at(node_index)
+```
+
+Retrieves the input tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A tensor (or list of tensors if the layer has multiple inputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+
+
+``` python
+get_input_mask_at(node_index)
+```
+
+Retrieves the input mask tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A mask tensor
+(or list of tensors if the layer has multiple inputs).
+
+
+
+
+``` python
+get_input_shape_at(node_index)
+```
+
+Retrieves the input shape(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A shape tuple
+(or list of shape tuples if the layer has multiple inputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_losses_for
+
+``` python
+get_losses_for(inputs)
+```
+
+Retrieves losses relevant to a specific set of inputs.
+
+
+#### Arguments:
+
+
+* `inputs`: Input tensor or list/tuple of input tensors.
+
+
+#### Returns:
+
+List of loss tensors of the layer that depend on `inputs`.
+
+
+get_output_at
+
+``` python
+get_output_at(node_index)
+```
+
+Retrieves the output tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A tensor (or list of tensors if the layer has multiple outputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_output_mask_at
+
+``` python
+get_output_mask_at(node_index)
+```
+
+Retrieves the output mask tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A mask tensor
+(or list of tensors if the layer has multiple outputs).
+
+
+get_output_shape_at
+
+``` python
+get_output_shape_at(node_index)
+```
+
+Retrieves the output shape(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A shape tuple
+(or list of shape tuples if the layer has multiple outputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_updates_for
+
+``` python
+get_updates_for(inputs)
+```
+
+Retrieves updates relevant to a specific set of inputs.
+
+
+#### Arguments:
+
+
+* `inputs`: Input tensor or list/tuple of input tensors.
+
+
+#### Returns:
+
+List of update ops of the layer that depend on `inputs`.
+
+
+get_weights
+
+``` python
+get_weights()
+```
+
+Returns the current weights of the layer.
+
+
+#### Returns:
+
+Weights values as a list of numpy arrays.
+
+
+set_weights
+
+``` python
+set_weights(weights)
+```
+
+Sets the weights of the layer, from Numpy arrays.
+
+
+#### Arguments:
+
+
+* `weights`: a list of Numpy arrays. The number
+ of arrays and their shape must match
+ number of the dimensions of the weights
+ of the layer (i.e. it should match the
+ output of `get_weights`).
+
+
+#### Raises:
+
+
+* `ValueError`: If the provided weights list does not match the
+ layer's specifications.
+
+with_name_scope
+
+``` python
+with_name_scope(
+ cls,
+ method
+)
+```
+
+Decorator to automatically enter the module name scope.
+
+```
+class MyModule(tf.Module):
+ @tf.Module.with_name_scope
+ def __call__(self, x):
+ if not hasattr(self, 'w'):
+ self.w = tf.Variable(tf.random.normal([x.shape[1], 64]))
+ return tf.matmul(x, self.w)
+```
+
+Using the above module would produce `tf.Variable`s and `tf.Tensor`s whose
+names included the module name:
+
+```
+mod = MyModule()
+mod(tf.ones([8, 32]))
+# ==>
+mod.w
+# ==>
+```
+
+#### Args:
+
+
+* `method`: The method to wrap.
+
+
+#### Returns:
+
+The original method wrapped such that it enters the module's name scope.
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/AttentionWrapperState.md b/docs/api_docs/python/tfa/seq2seq/AttentionWrapperState.md
new file mode 100644
index 0000000000..c1d6d70c41
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/AttentionWrapperState.md
@@ -0,0 +1,123 @@
+
+
+
+
+
+
+
+
+
+
+
+
+# tfa.seq2seq.AttentionWrapperState
+
+## Class `AttentionWrapperState`
+
+`namedtuple` storing the state of a `AttentionWrapper`.
+
+
+
+### Aliases:
+
+* Class `tfa.seq2seq.AttentionWrapperState`
+* Class `tfa.seq2seq.attention_wrapper.AttentionWrapperState`
+
+
+
+Defined in [`seq2seq/attention_wrapper.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/attention_wrapper.py).
+
+
+
+
+#### Contains:
+
+
+- `cell_state`: The state of the wrapped `RNNCell` at the previous time
+ step.
+- `attention`: The attention emitted at the previous time step.
+- `time`: int32 scalar containing the current time step.
+- `alignments`: A single or tuple of `Tensor`(s) containing the
+ alignments emitted at the previous time step for each attention
+ mechanism.
+- `alignment_history`: (if enabled) a single or tuple of `TensorArray`(s)
+ containing alignment matrices from all time steps for each attention
+ mechanism. Call `stack()` on each to convert to a `Tensor`.
+- `attention_state`: A single or tuple of nested objects
+ containing attention mechanism state for each attention mechanism.
+ The objects may contain Tensors or TensorArrays.
+
+
+## Properties
+
+cell_state
+
+
+
+
+attention
+
+
+
+
+time
+
+
+
+
+alignments
+
+
+
+
+alignment_history
+
+
+
+
+attention_state
+
+
+
+
+
+
+## Methods
+
+clone
+
+``` python
+clone(**kwargs)
+```
+
+Clone this object, overriding components provided by kwargs.
+
+The new state fields' shape must match original state fields' shape.
+This will be validated, and original fields' shape will be propagated
+to new fields.
+
+#### Example:
+
+
+
+```python
+initial_state = attention_wrapper.get_initial_state(
+ batch_size=..., dtype=...)
+initial_state = initial_state.clone(cell_state=encoder_state)
+```
+
+#### Args:
+
+
+* `**kwargs`: Any properties of the state object to replace in the
+ returned `AttentionWrapperState`.
+
+
+#### Returns:
+
+A new `AttentionWrapperState` whose properties are the same as
+this one, except any overridden properties as provided in `kwargs`.
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/BahdanauAttention.md b/docs/api_docs/python/tfa/seq2seq/BahdanauAttention.md
new file mode 100644
index 0000000000..bd20a0647e
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/BahdanauAttention.md
@@ -0,0 +1,927 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+# tfa.seq2seq.BahdanauAttention
+
+## Class `BahdanauAttention`
+
+Implements Bahdanau-style (additive) attention.
+
+
+
+### Aliases:
+
+* Class `tfa.seq2seq.BahdanauAttention`
+* Class `tfa.seq2seq.attention_wrapper.BahdanauAttention`
+
+
+
+Defined in [`seq2seq/attention_wrapper.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/attention_wrapper.py).
+
+
+
+This attention has two forms. The first is Bahdanau attention,
+as described in:
+
+Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio.
+"Neural Machine Translation by Jointly Learning to Align and Translate."
+ICLR 2015. https://arxiv.org/abs/1409.0473
+
+The second is the normalized form. This form is inspired by the
+weight normalization article:
+
+Tim Salimans, Diederik P. Kingma.
+"Weight Normalization: A Simple Reparameterization to Accelerate
+ Training of Deep Neural Networks."
+https://arxiv.org/abs/1602.07868
+
+To enable the second form, construct the object with parameter
+`normalize=True`.
+
+__init__
+
+``` python
+__init__(
+ units,
+ memory,
+ memory_sequence_length=None,
+ normalize=False,
+ probability_fn='softmax',
+ kernel_initializer='glorot_uniform',
+ dtype=None,
+ name='BahdanauAttention',
+ **kwargs
+)
+```
+
+Construct the Attention mechanism.
+
+
+#### Args:
+
+
+* `units`: The depth of the query mechanism.
+* `memory`: The memory to query; usually the output of an RNN encoder.
+ This tensor should be shaped `[batch_size, max_time, ...]`.
+* `memory_sequence_length`: (optional): Sequence lengths for the batch
+ entries in memory. If provided, the memory tensor rows are masked
+ with zeros for values past the respective sequence lengths.
+* `normalize`: Python boolean. Whether to normalize the energy term.
+* `probability_fn`: (optional) string, the name of function to convert
+ the attention score to probabilities. The default is `softmax`
+ which is `tf.nn.softmax`. Other options is `hardmax`, which is
+ hardmax() within this module. Any other value will result into
+ validation error. Default to use `softmax`.
+* `kernel_initializer`: (optional), the name of the initializer for the
+ attention kernel.
+* `dtype`: The data type for the query and memory layers of the attention
+ mechanism.
+* `name`: Name to use when creating ops.
+* `**kwargs`: Dictionary that contains other common arguments for layer
+ creation.
+
+
+
+## Properties
+
+activity_regularizer
+
+Optional regularizer function for the output of this layer.
+
+
+alignments_size
+
+
+
+
+dtype
+
+
+
+
+dynamic
+
+
+
+
+
+
+Retrieves the input tensor(s) of a layer.
+
+Only applicable if the layer has exactly one input,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Input tensor or list of input tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+* `AttributeError`: If no inbound nodes are found.
+
+
+
+Retrieves the input mask tensor(s) of a layer.
+
+Only applicable if the layer has exactly one inbound node,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Input mask tensor (potentially None) or list of input
+mask tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+
+
+Retrieves the input shape(s) of a layer.
+
+Only applicable if the layer has exactly one input,
+i.e. if it is connected to one incoming layer, or if all inputs
+have the same shape.
+
+#### Returns:
+
+Input shape, as an integer shape tuple
+(or list of shape tuples, one tuple per input tensor).
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer has no defined input_shape.
+* `RuntimeError`: if called in Eager mode.
+
+
+
+
+
+
+losses
+
+Losses which are associated with this `Layer`.
+
+Variable regularization tensors are created when this property is accessed,
+so it is eager safe: accessing `losses` under a `tf.GradientTape` will
+propagate gradients back to the corresponding variables.
+
+#### Returns:
+
+A list of tensors.
+
+
+metrics
+
+
+
+
+name
+
+
+
+
+name_scope
+
+Returns a `tf.name_scope` instance for this class.
+
+
+non_trainable_variables
+
+
+
+
+non_trainable_weights
+
+
+
+
+output
+
+Retrieves the output tensor(s) of a layer.
+
+Only applicable if the layer has exactly one output,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Output tensor or list of output tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to more than one incoming
+ layers.
+* `RuntimeError`: if called in Eager mode.
+
+output_mask
+
+Retrieves the output mask tensor(s) of a layer.
+
+Only applicable if the layer has exactly one inbound node,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Output mask tensor (potentially None) or list of output
+mask tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+output_shape
+
+Retrieves the output shape(s) of a layer.
+
+Only applicable if the layer has one output,
+or if all outputs have the same shape.
+
+#### Returns:
+
+Output shape, as an integer shape tuple
+(or list of shape tuples, one tuple per output tensor).
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer has no defined output shape.
+* `RuntimeError`: if called in Eager mode.
+
+state_size
+
+
+
+
+submodules
+
+Sequence of all sub-modules.
+
+Submodules are modules which are properties of this module, or found as
+properties of modules which are properties of this module (and so on).
+
+```
+a = tf.Module()
+b = tf.Module()
+c = tf.Module()
+a.b = b
+b.c = c
+assert list(a.submodules) == [b, c]
+assert list(b.submodules) == [c]
+assert list(c.submodules) == []
+```
+
+#### Returns:
+
+A sequence of all submodules.
+
+
+trainable
+
+
+
+
+trainable_variables
+
+
+
+
+trainable_weights
+
+
+
+
+updates
+
+
+
+
+variables
+
+Returns the list of all layer variables/weights.
+
+Alias of `self.weights`.
+
+#### Returns:
+
+A list of variables.
+
+
+weights
+
+Returns the list of all layer variables/weights.
+
+
+#### Returns:
+
+A list of variables.
+
+
+
+
+## Methods
+
+__call__
+
+``` python
+__call__(
+ inputs,
+ **kwargs
+)
+```
+
+Preprocess the inputs before calling `base_layer.__call__()`.
+
+Note that there are situation here, one for setup memory, and one with
+actual query and state.
+1. When the memory has not been configured, we just pass all the param
+ to base_layer.__call__(), which will then invoke self.call() with
+ proper inputs, which allows this class to setup memory.
+2. When the memory has already been setup, the input should contain
+ query and state, and optionally processed memory. If the processed
+ memory is not included in the input, we will have to append it to
+ the inputs and give it to the base_layer.__call__(). The processed
+ memory is the output of first invocation of self.__call__(). If we
+ don't add it here, then from keras perspective, the graph is
+ disconnected since the output from previous call is never used.
+
+#### Args:
+
+
+* `inputs`: the inputs tensors.
+* `**kwargs`: dict, other keyeword arguments for the `__call__()`
+
+apply
+
+``` python
+apply(
+ inputs,
+ *args,
+ **kwargs
+)
+```
+
+Apply the layer on a input.
+
+This is an alias of `self.__call__`.
+
+#### Arguments:
+
+
+* `inputs`: Input tensor(s).
+* `*args`: additional positional arguments to be passed to `self.call`.
+* `**kwargs`: additional keyword arguments to be passed to `self.call`.
+
+
+#### Returns:
+
+Output tensor(s).
+
+
+build
+
+``` python
+build(input_shape)
+```
+
+
+
+
+compute_mask
+
+``` python
+compute_mask(
+ inputs,
+ mask=None
+)
+```
+
+
+
+
+compute_output_shape
+
+``` python
+compute_output_shape(input_shape)
+```
+
+Computes the output shape of the layer.
+
+Assumes that the layer will be built
+to match that input shape provided.
+
+#### Arguments:
+
+
+* `input_shape`: Shape tuple (tuple of integers)
+ or list of shape tuples (one per output tensor of the layer).
+ Shape tuples can include None for free dimensions,
+ instead of an integer.
+
+
+#### Returns:
+
+An input shape tuple.
+
+
+count_params
+
+``` python
+count_params()
+```
+
+Count the total number of scalars composing the weights.
+
+
+#### Returns:
+
+An integer count.
+
+
+
+#### Raises:
+
+
+* `ValueError`: if the layer isn't yet built
+ (in which case its weights aren't yet defined).
+
+deserialize_inner_layer_from_config
+
+``` python
+deserialize_inner_layer_from_config(
+ cls,
+ config,
+ custom_objects
+)
+```
+
+Helper method that reconstruct the query and memory from the config.
+
+In the get_config() method, the query and memory layer configs are
+serialized into dict for persistence, this method perform the reverse
+action to reconstruct the layer from the config.
+
+#### Args:
+
+
+* `config`: dict, the configs that will be used to reconstruct the
+ object.
+* `custom_objects`: dict mapping class names (or function names) of
+ custom (non-Keras) objects to class/functions.
+
+#### Returns:
+
+
+* `config`: dict, the config with layer instance created, which is ready
+ to be used as init parameters.
+
+from_config
+
+``` python
+@classmethod
+from_config(
+ cls,
+ config,
+ custom_objects=None
+)
+```
+
+
+
+
+get_config
+
+``` python
+get_config()
+```
+
+
+
+
+
+
+``` python
+get_input_at(node_index)
+```
+
+Retrieves the input tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A tensor (or list of tensors if the layer has multiple inputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+
+
+``` python
+get_input_mask_at(node_index)
+```
+
+Retrieves the input mask tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A mask tensor
+(or list of tensors if the layer has multiple inputs).
+
+
+
+
+``` python
+get_input_shape_at(node_index)
+```
+
+Retrieves the input shape(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A shape tuple
+(or list of shape tuples if the layer has multiple inputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_losses_for
+
+``` python
+get_losses_for(inputs)
+```
+
+Retrieves losses relevant to a specific set of inputs.
+
+
+#### Arguments:
+
+
+* `inputs`: Input tensor or list/tuple of input tensors.
+
+
+#### Returns:
+
+List of loss tensors of the layer that depend on `inputs`.
+
+
+get_output_at
+
+``` python
+get_output_at(node_index)
+```
+
+Retrieves the output tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A tensor (or list of tensors if the layer has multiple outputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_output_mask_at
+
+``` python
+get_output_mask_at(node_index)
+```
+
+Retrieves the output mask tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A mask tensor
+(or list of tensors if the layer has multiple outputs).
+
+
+get_output_shape_at
+
+``` python
+get_output_shape_at(node_index)
+```
+
+Retrieves the output shape(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A shape tuple
+(or list of shape tuples if the layer has multiple outputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_updates_for
+
+``` python
+get_updates_for(inputs)
+```
+
+Retrieves updates relevant to a specific set of inputs.
+
+
+#### Arguments:
+
+
+* `inputs`: Input tensor or list/tuple of input tensors.
+
+
+#### Returns:
+
+List of update ops of the layer that depend on `inputs`.
+
+
+get_weights
+
+``` python
+get_weights()
+```
+
+Returns the current weights of the layer.
+
+
+#### Returns:
+
+Weights values as a list of numpy arrays.
+
+
+initial_alignments
+
+``` python
+initial_alignments(
+ batch_size,
+ dtype
+)
+```
+
+Creates the initial alignment values for the `AttentionWrapper`
+class.
+
+This is important for AttentionMechanisms that use the previous
+alignment to calculate the alignment at the next time step
+(e.g. monotonic attention).
+
+The default behavior is to return a tensor of all zeros.
+
+#### Args:
+
+
+* `batch_size`: `int32` scalar, the batch_size.
+* `dtype`: The `dtype`.
+
+
+#### Returns:
+
+A `dtype` tensor shaped `[batch_size, alignments_size]`
+(`alignments_size` is the values' `max_time`).
+
+
+initial_state
+
+``` python
+initial_state(
+ batch_size,
+ dtype
+)
+```
+
+Creates the initial state values for the `AttentionWrapper` class.
+
+This is important for AttentionMechanisms that use the previous
+alignment to calculate the alignment at the next time step
+(e.g. monotonic attention).
+
+The default behavior is to return the same output as
+initial_alignments.
+
+#### Args:
+
+
+* `batch_size`: `int32` scalar, the batch_size.
+* `dtype`: The `dtype`.
+
+
+#### Returns:
+
+A structure of all-zero tensors with shapes as described by
+`state_size`.
+
+
+set_weights
+
+``` python
+set_weights(weights)
+```
+
+Sets the weights of the layer, from Numpy arrays.
+
+
+#### Arguments:
+
+
+* `weights`: a list of Numpy arrays. The number
+ of arrays and their shape must match
+ number of the dimensions of the weights
+ of the layer (i.e. it should match the
+ output of `get_weights`).
+
+
+#### Raises:
+
+
+* `ValueError`: If the provided weights list does not match the
+ layer's specifications.
+
+with_name_scope
+
+``` python
+with_name_scope(
+ cls,
+ method
+)
+```
+
+Decorator to automatically enter the module name scope.
+
+```
+class MyModule(tf.Module):
+ @tf.Module.with_name_scope
+ def __call__(self, x):
+ if not hasattr(self, 'w'):
+ self.w = tf.Variable(tf.random.normal([x.shape[1], 64]))
+ return tf.matmul(x, self.w)
+```
+
+Using the above module would produce `tf.Variable`s and `tf.Tensor`s whose
+names included the module name:
+
+```
+mod = MyModule()
+mod(tf.ones([8, 32]))
+# ==>
+mod.w
+# ==>
+```
+
+#### Args:
+
+
+* `method`: The method to wrap.
+
+
+#### Returns:
+
+The original method wrapped such that it enters the module's name scope.
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/BahdanauMonotonicAttention.md b/docs/api_docs/python/tfa/seq2seq/BahdanauMonotonicAttention.md
new file mode 100644
index 0000000000..58f4e5b883
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/BahdanauMonotonicAttention.md
@@ -0,0 +1,925 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+# tfa.seq2seq.BahdanauMonotonicAttention
+
+## Class `BahdanauMonotonicAttention`
+
+Monotonic attention mechanism with Bahadanau-style energy function.
+
+
+
+### Aliases:
+
+* Class `tfa.seq2seq.BahdanauMonotonicAttention`
+* Class `tfa.seq2seq.attention_wrapper.BahdanauMonotonicAttention`
+
+
+
+Defined in [`seq2seq/attention_wrapper.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/attention_wrapper.py).
+
+
+
+This type of attention enforces a monotonic constraint on the attention
+distributions; that is once the model attends to a given point in the
+memory it can't attend to any prior points at subsequence output timesteps.
+It achieves this by using the _monotonic_probability_fn instead of softmax
+to construct its attention distributions. Since the attention scores are
+passed through a sigmoid, a learnable scalar bias parameter is applied
+after the score function and before the sigmoid. Otherwise, it is
+equivalent to BahdanauAttention. This approach is proposed in
+
+Colin Raffel, Minh-Thang Luong, Peter J. Liu, Ron J. Weiss, Douglas Eck,
+"Online and Linear-Time Attention by Enforcing Monotonic Alignments."
+ICML 2017. https://arxiv.org/abs/1704.00784
+
+__init__
+
+``` python
+__init__(
+ units,
+ memory,
+ memory_sequence_length=None,
+ normalize=False,
+ sigmoid_noise=0.0,
+ sigmoid_noise_seed=None,
+ score_bias_init=0.0,
+ mode='parallel',
+ kernel_initializer='glorot_uniform',
+ dtype=None,
+ name='BahdanauMonotonicAttention',
+ **kwargs
+)
+```
+
+Construct the Attention mechanism.
+
+
+#### Args:
+
+
+* `units`: The depth of the query mechanism.
+* `memory`: The memory to query; usually the output of an RNN encoder.
+ This tensor should be shaped `[batch_size, max_time, ...]`.
+* `memory_sequence_length`: (optional): Sequence lengths for the batch
+ entries in memory. If provided, the memory tensor rows are masked
+ with zeros for values past the respective sequence lengths.
+* `normalize`: Python boolean. Whether to normalize the energy term.
+* `sigmoid_noise`: Standard deviation of pre-sigmoid noise. See the
+ docstring for `_monotonic_probability_fn` for more information.
+* `sigmoid_noise_seed`: (optional) Random seed for pre-sigmoid noise.
+* `score_bias_init`: Initial value for score bias scalar. It's
+ recommended to initialize this to a negative value when the length
+ of the memory is large.
+* `mode`: How to compute the attention distribution. Must be one of
+ 'recursive', 'parallel', or 'hard'. See the docstring for
+ `tf.contrib.seq2seq.monotonic_attention` for more information.
+* `kernel_initializer`: (optional), the name of the initializer for the
+ attention kernel.
+* `dtype`: The data type for the query and memory layers of the attention
+ mechanism.
+* `name`: Name to use when creating ops.
+* `**kwargs`: Dictionary that contains other common arguments for layer
+ creation.
+
+
+
+## Properties
+
+activity_regularizer
+
+Optional regularizer function for the output of this layer.
+
+
+alignments_size
+
+
+
+
+dtype
+
+
+
+
+dynamic
+
+
+
+
+
+
+Retrieves the input tensor(s) of a layer.
+
+Only applicable if the layer has exactly one input,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Input tensor or list of input tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+* `AttributeError`: If no inbound nodes are found.
+
+
+
+Retrieves the input mask tensor(s) of a layer.
+
+Only applicable if the layer has exactly one inbound node,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Input mask tensor (potentially None) or list of input
+mask tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+
+
+Retrieves the input shape(s) of a layer.
+
+Only applicable if the layer has exactly one input,
+i.e. if it is connected to one incoming layer, or if all inputs
+have the same shape.
+
+#### Returns:
+
+Input shape, as an integer shape tuple
+(or list of shape tuples, one tuple per input tensor).
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer has no defined input_shape.
+* `RuntimeError`: if called in Eager mode.
+
+
+
+
+
+
+losses
+
+Losses which are associated with this `Layer`.
+
+Variable regularization tensors are created when this property is accessed,
+so it is eager safe: accessing `losses` under a `tf.GradientTape` will
+propagate gradients back to the corresponding variables.
+
+#### Returns:
+
+A list of tensors.
+
+
+metrics
+
+
+
+
+name
+
+
+
+
+name_scope
+
+Returns a `tf.name_scope` instance for this class.
+
+
+non_trainable_variables
+
+
+
+
+non_trainable_weights
+
+
+
+
+output
+
+Retrieves the output tensor(s) of a layer.
+
+Only applicable if the layer has exactly one output,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Output tensor or list of output tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to more than one incoming
+ layers.
+* `RuntimeError`: if called in Eager mode.
+
+output_mask
+
+Retrieves the output mask tensor(s) of a layer.
+
+Only applicable if the layer has exactly one inbound node,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Output mask tensor (potentially None) or list of output
+mask tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+output_shape
+
+Retrieves the output shape(s) of a layer.
+
+Only applicable if the layer has one output,
+or if all outputs have the same shape.
+
+#### Returns:
+
+Output shape, as an integer shape tuple
+(or list of shape tuples, one tuple per output tensor).
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer has no defined output shape.
+* `RuntimeError`: if called in Eager mode.
+
+state_size
+
+
+
+
+submodules
+
+Sequence of all sub-modules.
+
+Submodules are modules which are properties of this module, or found as
+properties of modules which are properties of this module (and so on).
+
+```
+a = tf.Module()
+b = tf.Module()
+c = tf.Module()
+a.b = b
+b.c = c
+assert list(a.submodules) == [b, c]
+assert list(b.submodules) == [c]
+assert list(c.submodules) == []
+```
+
+#### Returns:
+
+A sequence of all submodules.
+
+
+trainable
+
+
+
+
+trainable_variables
+
+
+
+
+trainable_weights
+
+
+
+
+updates
+
+
+
+
+variables
+
+Returns the list of all layer variables/weights.
+
+Alias of `self.weights`.
+
+#### Returns:
+
+A list of variables.
+
+
+weights
+
+Returns the list of all layer variables/weights.
+
+
+#### Returns:
+
+A list of variables.
+
+
+
+
+## Methods
+
+__call__
+
+``` python
+__call__(
+ inputs,
+ **kwargs
+)
+```
+
+Preprocess the inputs before calling `base_layer.__call__()`.
+
+Note that there are situation here, one for setup memory, and one with
+actual query and state.
+1. When the memory has not been configured, we just pass all the param
+ to base_layer.__call__(), which will then invoke self.call() with
+ proper inputs, which allows this class to setup memory.
+2. When the memory has already been setup, the input should contain
+ query and state, and optionally processed memory. If the processed
+ memory is not included in the input, we will have to append it to
+ the inputs and give it to the base_layer.__call__(). The processed
+ memory is the output of first invocation of self.__call__(). If we
+ don't add it here, then from keras perspective, the graph is
+ disconnected since the output from previous call is never used.
+
+#### Args:
+
+
+* `inputs`: the inputs tensors.
+* `**kwargs`: dict, other keyeword arguments for the `__call__()`
+
+apply
+
+``` python
+apply(
+ inputs,
+ *args,
+ **kwargs
+)
+```
+
+Apply the layer on a input.
+
+This is an alias of `self.__call__`.
+
+#### Arguments:
+
+
+* `inputs`: Input tensor(s).
+* `*args`: additional positional arguments to be passed to `self.call`.
+* `**kwargs`: additional keyword arguments to be passed to `self.call`.
+
+
+#### Returns:
+
+Output tensor(s).
+
+
+build
+
+``` python
+build(input_shape)
+```
+
+
+
+
+compute_mask
+
+``` python
+compute_mask(
+ inputs,
+ mask=None
+)
+```
+
+
+
+
+compute_output_shape
+
+``` python
+compute_output_shape(input_shape)
+```
+
+Computes the output shape of the layer.
+
+Assumes that the layer will be built
+to match that input shape provided.
+
+#### Arguments:
+
+
+* `input_shape`: Shape tuple (tuple of integers)
+ or list of shape tuples (one per output tensor of the layer).
+ Shape tuples can include None for free dimensions,
+ instead of an integer.
+
+
+#### Returns:
+
+An input shape tuple.
+
+
+count_params
+
+``` python
+count_params()
+```
+
+Count the total number of scalars composing the weights.
+
+
+#### Returns:
+
+An integer count.
+
+
+
+#### Raises:
+
+
+* `ValueError`: if the layer isn't yet built
+ (in which case its weights aren't yet defined).
+
+deserialize_inner_layer_from_config
+
+``` python
+deserialize_inner_layer_from_config(
+ cls,
+ config,
+ custom_objects
+)
+```
+
+Helper method that reconstruct the query and memory from the config.
+
+In the get_config() method, the query and memory layer configs are
+serialized into dict for persistence, this method perform the reverse
+action to reconstruct the layer from the config.
+
+#### Args:
+
+
+* `config`: dict, the configs that will be used to reconstruct the
+ object.
+* `custom_objects`: dict mapping class names (or function names) of
+ custom (non-Keras) objects to class/functions.
+
+#### Returns:
+
+
+* `config`: dict, the config with layer instance created, which is ready
+ to be used as init parameters.
+
+from_config
+
+``` python
+@classmethod
+from_config(
+ cls,
+ config,
+ custom_objects=None
+)
+```
+
+
+
+
+get_config
+
+``` python
+get_config()
+```
+
+
+
+
+
+
+``` python
+get_input_at(node_index)
+```
+
+Retrieves the input tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A tensor (or list of tensors if the layer has multiple inputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+
+
+``` python
+get_input_mask_at(node_index)
+```
+
+Retrieves the input mask tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A mask tensor
+(or list of tensors if the layer has multiple inputs).
+
+
+
+
+``` python
+get_input_shape_at(node_index)
+```
+
+Retrieves the input shape(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A shape tuple
+(or list of shape tuples if the layer has multiple inputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_losses_for
+
+``` python
+get_losses_for(inputs)
+```
+
+Retrieves losses relevant to a specific set of inputs.
+
+
+#### Arguments:
+
+
+* `inputs`: Input tensor or list/tuple of input tensors.
+
+
+#### Returns:
+
+List of loss tensors of the layer that depend on `inputs`.
+
+
+get_output_at
+
+``` python
+get_output_at(node_index)
+```
+
+Retrieves the output tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A tensor (or list of tensors if the layer has multiple outputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_output_mask_at
+
+``` python
+get_output_mask_at(node_index)
+```
+
+Retrieves the output mask tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A mask tensor
+(or list of tensors if the layer has multiple outputs).
+
+
+get_output_shape_at
+
+``` python
+get_output_shape_at(node_index)
+```
+
+Retrieves the output shape(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A shape tuple
+(or list of shape tuples if the layer has multiple outputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_updates_for
+
+``` python
+get_updates_for(inputs)
+```
+
+Retrieves updates relevant to a specific set of inputs.
+
+
+#### Arguments:
+
+
+* `inputs`: Input tensor or list/tuple of input tensors.
+
+
+#### Returns:
+
+List of update ops of the layer that depend on `inputs`.
+
+
+get_weights
+
+``` python
+get_weights()
+```
+
+Returns the current weights of the layer.
+
+
+#### Returns:
+
+Weights values as a list of numpy arrays.
+
+
+initial_alignments
+
+``` python
+initial_alignments(
+ batch_size,
+ dtype
+)
+```
+
+Creates the initial alignment values for the monotonic attentions.
+
+Initializes to dirac distributions, i.e.
+[1, 0, 0, ...memory length..., 0] for all entries in the batch.
+
+#### Args:
+
+
+* `batch_size`: `int32` scalar, the batch_size.
+* `dtype`: The `dtype`.
+
+
+#### Returns:
+
+A `dtype` tensor shaped `[batch_size, alignments_size]`
+(`alignments_size` is the values' `max_time`).
+
+
+initial_state
+
+``` python
+initial_state(
+ batch_size,
+ dtype
+)
+```
+
+Creates the initial state values for the `AttentionWrapper` class.
+
+This is important for AttentionMechanisms that use the previous
+alignment to calculate the alignment at the next time step
+(e.g. monotonic attention).
+
+The default behavior is to return the same output as
+initial_alignments.
+
+#### Args:
+
+
+* `batch_size`: `int32` scalar, the batch_size.
+* `dtype`: The `dtype`.
+
+
+#### Returns:
+
+A structure of all-zero tensors with shapes as described by
+`state_size`.
+
+
+set_weights
+
+``` python
+set_weights(weights)
+```
+
+Sets the weights of the layer, from Numpy arrays.
+
+
+#### Arguments:
+
+
+* `weights`: a list of Numpy arrays. The number
+ of arrays and their shape must match
+ number of the dimensions of the weights
+ of the layer (i.e. it should match the
+ output of `get_weights`).
+
+
+#### Raises:
+
+
+* `ValueError`: If the provided weights list does not match the
+ layer's specifications.
+
+with_name_scope
+
+``` python
+with_name_scope(
+ cls,
+ method
+)
+```
+
+Decorator to automatically enter the module name scope.
+
+```
+class MyModule(tf.Module):
+ @tf.Module.with_name_scope
+ def __call__(self, x):
+ if not hasattr(self, 'w'):
+ self.w = tf.Variable(tf.random.normal([x.shape[1], 64]))
+ return tf.matmul(x, self.w)
+```
+
+Using the above module would produce `tf.Variable`s and `tf.Tensor`s whose
+names included the module name:
+
+```
+mod = MyModule()
+mod(tf.ones([8, 32]))
+# ==>
+mod.w
+# ==>
+```
+
+#### Args:
+
+
+* `method`: The method to wrap.
+
+
+#### Returns:
+
+The original method wrapped such that it enters the module's name scope.
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/BaseDecoder.md b/docs/api_docs/python/tfa/seq2seq/BaseDecoder.md
new file mode 100644
index 0000000000..6a56b0c256
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/BaseDecoder.md
@@ -0,0 +1,977 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+# tfa.seq2seq.BaseDecoder
+
+## Class `BaseDecoder`
+
+An RNN Decoder that is based on a Keras layer.
+
+
+
+### Aliases:
+
+* Class `tfa.seq2seq.BaseDecoder`
+* Class `tfa.seq2seq.decoder.BaseDecoder`
+
+
+
+Defined in [`seq2seq/decoder.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/decoder.py).
+
+
+
+Concepts used by this interface:
+- `inputs`: (structure of) tensors and TensorArrays that is passed as input
+ to the RNNCell composing the decoder, at each time step.
+- `state`: (structure of) tensors and TensorArrays that is passed to the
+ RNNCell instance as the state.
+- `memory`: (sturecute of) tensors that is usually the full output of the
+ encoder, which will be used for the attention wrapper for the RNNCell.
+- `finished`: boolean tensor telling whether each sequence in the batch is
+ finished.
+- `outputs`: Instance of BasicDecoderOutput. Result of the decoding, at
+ each time step.
+
+__init__
+
+``` python
+__init__(
+ output_time_major=False,
+ impute_finished=False,
+ maximum_iterations=None,
+ parallel_iterations=32,
+ swap_memory=False,
+ **kwargs
+)
+```
+
+
+
+
+
+
+## Properties
+
+activity_regularizer
+
+Optional regularizer function for the output of this layer.
+
+
+batch_size
+
+The batch size of input values.
+
+
+dtype
+
+
+
+
+dynamic
+
+
+
+
+
+
+Retrieves the input tensor(s) of a layer.
+
+Only applicable if the layer has exactly one input,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Input tensor or list of input tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+* `AttributeError`: If no inbound nodes are found.
+
+
+
+Retrieves the input mask tensor(s) of a layer.
+
+Only applicable if the layer has exactly one inbound node,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Input mask tensor (potentially None) or list of input
+mask tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+
+
+Retrieves the input shape(s) of a layer.
+
+Only applicable if the layer has exactly one input,
+i.e. if it is connected to one incoming layer, or if all inputs
+have the same shape.
+
+#### Returns:
+
+Input shape, as an integer shape tuple
+(or list of shape tuples, one tuple per input tensor).
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer has no defined input_shape.
+* `RuntimeError`: if called in Eager mode.
+
+
+
+
+
+
+losses
+
+Losses which are associated with this `Layer`.
+
+Variable regularization tensors are created when this property is accessed,
+so it is eager safe: accessing `losses` under a `tf.GradientTape` will
+propagate gradients back to the corresponding variables.
+
+#### Returns:
+
+A list of tensors.
+
+
+metrics
+
+
+
+
+name
+
+
+
+
+name_scope
+
+Returns a `tf.name_scope` instance for this class.
+
+
+non_trainable_variables
+
+
+
+
+non_trainable_weights
+
+
+
+
+output
+
+Retrieves the output tensor(s) of a layer.
+
+Only applicable if the layer has exactly one output,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Output tensor or list of output tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to more than one incoming
+ layers.
+* `RuntimeError`: if called in Eager mode.
+
+output_dtype
+
+A (possibly nested tuple of...) dtype[s].
+
+
+output_mask
+
+Retrieves the output mask tensor(s) of a layer.
+
+Only applicable if the layer has exactly one inbound node,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Output mask tensor (potentially None) or list of output
+mask tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+output_shape
+
+Retrieves the output shape(s) of a layer.
+
+Only applicable if the layer has one output,
+or if all outputs have the same shape.
+
+#### Returns:
+
+Output shape, as an integer shape tuple
+(or list of shape tuples, one tuple per output tensor).
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer has no defined output shape.
+* `RuntimeError`: if called in Eager mode.
+
+output_size
+
+A (possibly nested tuple of...) integer[s] or `TensorShape`
+object[s].
+
+submodules
+
+Sequence of all sub-modules.
+
+Submodules are modules which are properties of this module, or found as
+properties of modules which are properties of this module (and so on).
+
+```
+a = tf.Module()
+b = tf.Module()
+c = tf.Module()
+a.b = b
+b.c = c
+assert list(a.submodules) == [b, c]
+assert list(b.submodules) == [c]
+assert list(c.submodules) == []
+```
+
+#### Returns:
+
+A sequence of all submodules.
+
+
+tracks_own_finished
+
+Describes whether the Decoder keeps track of finished states.
+
+Most decoders will emit a true/false `finished` value independently
+at each time step. In this case, the `dynamic_decode` function keeps
+track of which batch entries are already finished, and performs a
+logical OR to insert new batches to the finished set.
+
+Some decoders, however, shuffle batches / beams between time steps and
+`dynamic_decode` will mix up the finished state across these entries
+because it does not track the reshuffle across time steps. In this
+case, it is up to the decoder to declare that it will keep track of its
+own finished state by setting this property to `True`.
+
+#### Returns:
+
+Python bool.
+
+
+trainable
+
+
+
+
+trainable_variables
+
+
+
+
+trainable_weights
+
+
+
+
+updates
+
+
+
+
+variables
+
+Returns the list of all layer variables/weights.
+
+Alias of `self.weights`.
+
+#### Returns:
+
+A list of variables.
+
+
+weights
+
+Returns the list of all layer variables/weights.
+
+
+#### Returns:
+
+A list of variables.
+
+
+
+
+## Methods
+
+__call__
+
+``` python
+__call__(
+ inputs,
+ *args,
+ **kwargs
+)
+```
+
+Wraps `call`, applying pre- and post-processing steps.
+
+
+#### Arguments:
+
+
+* `inputs`: input tensor(s).
+* `*args`: additional positional arguments to be passed to `self.call`.
+* `**kwargs`: additional keyword arguments to be passed to `self.call`.
+
+
+#### Returns:
+
+Output tensor(s).
+
+
+
+#### Note:
+
+- The following optional keyword arguments are reserved for specific uses:
+ * `training`: Boolean scalar tensor of Python boolean indicating
+ whether the `call` is meant for training or inference.
+ * `mask`: Boolean input mask.
+- If the layer's `call` method takes a `mask` argument (as some Keras
+ layers do), its default value will be set to the mask generated
+ for `inputs` by the previous layer (if `input` did come from
+ a layer that generated a corresponding mask, i.e. if it came from
+ a Keras layer with masking support.
+
+
+
+#### Raises:
+
+
+* `ValueError`: if the layer's `call` method returns None (an invalid value).
+
+apply
+
+``` python
+apply(
+ inputs,
+ *args,
+ **kwargs
+)
+```
+
+Apply the layer on a input.
+
+This is an alias of `self.__call__`.
+
+#### Arguments:
+
+
+* `inputs`: Input tensor(s).
+* `*args`: additional positional arguments to be passed to `self.call`.
+* `**kwargs`: additional keyword arguments to be passed to `self.call`.
+
+
+#### Returns:
+
+Output tensor(s).
+
+
+build
+
+``` python
+build(input_shape)
+```
+
+Creates the variables of the layer (optional, for subclass implementers).
+
+This is a method that implementers of subclasses of `Layer` or `Model`
+can override if they need a state-creation step in-between
+layer instantiation and layer call.
+
+This is typically used to create the weights of `Layer` subclasses.
+
+#### Arguments:
+
+
+* `input_shape`: Instance of `TensorShape`, or list of instances of
+ `TensorShape` if the layer expects a list of inputs
+ (one instance per input).
+
+compute_mask
+
+``` python
+compute_mask(
+ inputs,
+ mask=None
+)
+```
+
+Computes an output mask tensor.
+
+
+#### Arguments:
+
+
+* `inputs`: Tensor or list of tensors.
+* `mask`: Tensor or list of tensors.
+
+
+#### Returns:
+
+None or a tensor (or list of tensors,
+ one per output tensor of the layer).
+
+
+compute_output_shape
+
+``` python
+compute_output_shape(input_shape)
+```
+
+Computes the output shape of the layer.
+
+Assumes that the layer will be built
+to match that input shape provided.
+
+#### Arguments:
+
+
+* `input_shape`: Shape tuple (tuple of integers)
+ or list of shape tuples (one per output tensor of the layer).
+ Shape tuples can include None for free dimensions,
+ instead of an integer.
+
+
+#### Returns:
+
+An input shape tuple.
+
+
+count_params
+
+``` python
+count_params()
+```
+
+Count the total number of scalars composing the weights.
+
+
+#### Returns:
+
+An integer count.
+
+
+
+#### Raises:
+
+
+* `ValueError`: if the layer isn't yet built
+ (in which case its weights aren't yet defined).
+
+finalize
+
+``` python
+finalize(
+ outputs,
+ final_state,
+ sequence_lengths
+)
+```
+
+
+
+
+from_config
+
+``` python
+from_config(
+ cls,
+ config
+)
+```
+
+Creates a layer from its config.
+
+This method is the reverse of `get_config`,
+capable of instantiating the same layer from the config
+dictionary. It does not handle layer connectivity
+(handled by Network), nor weights (handled by `set_weights`).
+
+#### Arguments:
+
+
+* `config`: A Python dictionary, typically the
+ output of get_config.
+
+
+#### Returns:
+
+A layer instance.
+
+
+get_config
+
+``` python
+get_config()
+```
+
+Returns the config of the layer.
+
+A layer config is a Python dictionary (serializable)
+containing the configuration of a layer.
+The same layer can be reinstantiated later
+(without its trained weights) from this configuration.
+
+The config of a layer does not include connectivity
+information, nor the layer class name. These are handled
+by `Network` (one layer of abstraction above).
+
+#### Returns:
+
+Python dictionary.
+
+
+
+
+``` python
+get_input_at(node_index)
+```
+
+Retrieves the input tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A tensor (or list of tensors if the layer has multiple inputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+
+
+``` python
+get_input_mask_at(node_index)
+```
+
+Retrieves the input mask tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A mask tensor
+(or list of tensors if the layer has multiple inputs).
+
+
+
+
+``` python
+get_input_shape_at(node_index)
+```
+
+Retrieves the input shape(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A shape tuple
+(or list of shape tuples if the layer has multiple inputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_losses_for
+
+``` python
+get_losses_for(inputs)
+```
+
+Retrieves losses relevant to a specific set of inputs.
+
+
+#### Arguments:
+
+
+* `inputs`: Input tensor or list/tuple of input tensors.
+
+
+#### Returns:
+
+List of loss tensors of the layer that depend on `inputs`.
+
+
+get_output_at
+
+``` python
+get_output_at(node_index)
+```
+
+Retrieves the output tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A tensor (or list of tensors if the layer has multiple outputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_output_mask_at
+
+``` python
+get_output_mask_at(node_index)
+```
+
+Retrieves the output mask tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A mask tensor
+(or list of tensors if the layer has multiple outputs).
+
+
+get_output_shape_at
+
+``` python
+get_output_shape_at(node_index)
+```
+
+Retrieves the output shape(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A shape tuple
+(or list of shape tuples if the layer has multiple outputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_updates_for
+
+``` python
+get_updates_for(inputs)
+```
+
+Retrieves updates relevant to a specific set of inputs.
+
+
+#### Arguments:
+
+
+* `inputs`: Input tensor or list/tuple of input tensors.
+
+
+#### Returns:
+
+List of update ops of the layer that depend on `inputs`.
+
+
+get_weights
+
+``` python
+get_weights()
+```
+
+Returns the current weights of the layer.
+
+
+#### Returns:
+
+Weights values as a list of numpy arrays.
+
+
+initialize
+
+``` python
+initialize(
+ inputs,
+ initial_state=None,
+ **kwargs
+)
+```
+
+Called before any decoding iterations.
+
+This methods must compute initial input values and initial state.
+
+#### Args:
+
+
+* `inputs`: (structure of) tensors that contains the input for the
+ decoder. In the normal case, its a tensor with shape
+ [batch, timestep, embedding].
+* `initial_state`: (structure of) tensors that contains the initial state
+ for the RNNCell.
+* `**kwargs`: Other arguments that are passed in from layer.call()
+ method. It could contains item like input sequence_length, or
+ masking for input.
+
+
+#### Returns:
+
+`(finished, initial_inputs, initial_state)`: initial values of
+'finished' flags, inputs and state.
+
+
+set_weights
+
+``` python
+set_weights(weights)
+```
+
+Sets the weights of the layer, from Numpy arrays.
+
+
+#### Arguments:
+
+
+* `weights`: a list of Numpy arrays. The number
+ of arrays and their shape must match
+ number of the dimensions of the weights
+ of the layer (i.e. it should match the
+ output of `get_weights`).
+
+
+#### Raises:
+
+
+* `ValueError`: If the provided weights list does not match the
+ layer's specifications.
+
+step
+
+``` python
+step(
+ time,
+ inputs,
+ state
+)
+```
+
+Called per step of decoding (but only once for dynamic decoding).
+
+
+#### Args:
+
+
+* `time`: Scalar `int32` tensor. Current step number.
+* `inputs`: RNNCell input (possibly nested tuple of) tensor[s] for this
+ time step.
+* `state`: RNNCell state (possibly nested tuple of) tensor[s] from
+ previous time step.
+
+
+#### Returns:
+
+`(outputs, next_state, next_inputs, finished)`: `outputs` is an
+object containing the decoder output, `next_state` is a
+(structure of) state tensors and TensorArrays, `next_inputs` is the
+tensor that should be used as input for the next step, `finished` is
+a boolean tensor telling whether the sequence is complete, for each
+sequence in the batch.
+
+
+with_name_scope
+
+``` python
+with_name_scope(
+ cls,
+ method
+)
+```
+
+Decorator to automatically enter the module name scope.
+
+```
+class MyModule(tf.Module):
+ @tf.Module.with_name_scope
+ def __call__(self, x):
+ if not hasattr(self, 'w'):
+ self.w = tf.Variable(tf.random.normal([x.shape[1], 64]))
+ return tf.matmul(x, self.w)
+```
+
+Using the above module would produce `tf.Variable`s and `tf.Tensor`s whose
+names included the module name:
+
+```
+mod = MyModule()
+mod(tf.ones([8, 32]))
+# ==>
+mod.w
+# ==>
+```
+
+#### Args:
+
+
+* `method`: The method to wrap.
+
+
+#### Returns:
+
+The original method wrapped such that it enters the module's name scope.
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/BasicDecoder.md b/docs/api_docs/python/tfa/seq2seq/BasicDecoder.md
new file mode 100644
index 0000000000..053bb24011
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/BasicDecoder.md
@@ -0,0 +1,954 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+# tfa.seq2seq.BasicDecoder
+
+## Class `BasicDecoder`
+
+Basic sampling decoder.
+
+Inherits From: [`BaseDecoder`](../../tfa/seq2seq/BaseDecoder.md)
+
+### Aliases:
+
+* Class `tfa.seq2seq.BasicDecoder`
+* Class `tfa.seq2seq.basic_decoder.BasicDecoder`
+
+
+
+Defined in [`seq2seq/basic_decoder.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/basic_decoder.py).
+
+
+
+
+__init__
+
+``` python
+__init__(
+ cell,
+ sampler,
+ output_layer=None,
+ **kwargs
+)
+```
+
+Initialize BasicDecoder.
+
+
+#### Args:
+
+
+* `cell`: An `RNNCell` instance.
+* `sampler`: A `Sampler` instance.
+* `output_layer`: (Optional) An instance of `tf.layers.Layer`, i.e.,
+ `tf.layers.Dense`. Optional layer to apply to the RNN output prior
+ to storing the result or sampling.
+* `**kwargs`: Other keyward arguments for layer creation.
+
+
+#### Raises:
+
+
+* `TypeError`: if `cell`, `helper` or `output_layer` have an incorrect
+type.
+
+
+
+## Properties
+
+activity_regularizer
+
+Optional regularizer function for the output of this layer.
+
+
+batch_size
+
+
+
+
+dtype
+
+
+
+
+dynamic
+
+
+
+
+
+
+Retrieves the input tensor(s) of a layer.
+
+Only applicable if the layer has exactly one input,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Input tensor or list of input tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+* `AttributeError`: If no inbound nodes are found.
+
+
+
+Retrieves the input mask tensor(s) of a layer.
+
+Only applicable if the layer has exactly one inbound node,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Input mask tensor (potentially None) or list of input
+mask tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+
+
+Retrieves the input shape(s) of a layer.
+
+Only applicable if the layer has exactly one input,
+i.e. if it is connected to one incoming layer, or if all inputs
+have the same shape.
+
+#### Returns:
+
+Input shape, as an integer shape tuple
+(or list of shape tuples, one tuple per input tensor).
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer has no defined input_shape.
+* `RuntimeError`: if called in Eager mode.
+
+
+
+
+
+
+losses
+
+Losses which are associated with this `Layer`.
+
+Variable regularization tensors are created when this property is accessed,
+so it is eager safe: accessing `losses` under a `tf.GradientTape` will
+propagate gradients back to the corresponding variables.
+
+#### Returns:
+
+A list of tensors.
+
+
+metrics
+
+
+
+
+name
+
+
+
+
+name_scope
+
+Returns a `tf.name_scope` instance for this class.
+
+
+non_trainable_variables
+
+
+
+
+non_trainable_weights
+
+
+
+
+output
+
+Retrieves the output tensor(s) of a layer.
+
+Only applicable if the layer has exactly one output,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Output tensor or list of output tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to more than one incoming
+ layers.
+* `RuntimeError`: if called in Eager mode.
+
+output_dtype
+
+
+
+
+output_mask
+
+Retrieves the output mask tensor(s) of a layer.
+
+Only applicable if the layer has exactly one inbound node,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Output mask tensor (potentially None) or list of output
+mask tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+output_shape
+
+Retrieves the output shape(s) of a layer.
+
+Only applicable if the layer has one output,
+or if all outputs have the same shape.
+
+#### Returns:
+
+Output shape, as an integer shape tuple
+(or list of shape tuples, one tuple per output tensor).
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer has no defined output shape.
+* `RuntimeError`: if called in Eager mode.
+
+output_size
+
+
+
+
+submodules
+
+Sequence of all sub-modules.
+
+Submodules are modules which are properties of this module, or found as
+properties of modules which are properties of this module (and so on).
+
+```
+a = tf.Module()
+b = tf.Module()
+c = tf.Module()
+a.b = b
+b.c = c
+assert list(a.submodules) == [b, c]
+assert list(b.submodules) == [c]
+assert list(c.submodules) == []
+```
+
+#### Returns:
+
+A sequence of all submodules.
+
+
+tracks_own_finished
+
+Describes whether the Decoder keeps track of finished states.
+
+Most decoders will emit a true/false `finished` value independently
+at each time step. In this case, the `dynamic_decode` function keeps
+track of which batch entries are already finished, and performs a
+logical OR to insert new batches to the finished set.
+
+Some decoders, however, shuffle batches / beams between time steps and
+`dynamic_decode` will mix up the finished state across these entries
+because it does not track the reshuffle across time steps. In this
+case, it is up to the decoder to declare that it will keep track of its
+own finished state by setting this property to `True`.
+
+#### Returns:
+
+Python bool.
+
+
+trainable
+
+
+
+
+trainable_variables
+
+
+
+
+trainable_weights
+
+
+
+
+updates
+
+
+
+
+variables
+
+Returns the list of all layer variables/weights.
+
+Alias of `self.weights`.
+
+#### Returns:
+
+A list of variables.
+
+
+weights
+
+Returns the list of all layer variables/weights.
+
+
+#### Returns:
+
+A list of variables.
+
+
+
+
+## Methods
+
+__call__
+
+``` python
+__call__(
+ inputs,
+ *args,
+ **kwargs
+)
+```
+
+Wraps `call`, applying pre- and post-processing steps.
+
+
+#### Arguments:
+
+
+* `inputs`: input tensor(s).
+* `*args`: additional positional arguments to be passed to `self.call`.
+* `**kwargs`: additional keyword arguments to be passed to `self.call`.
+
+
+#### Returns:
+
+Output tensor(s).
+
+
+
+#### Note:
+
+- The following optional keyword arguments are reserved for specific uses:
+ * `training`: Boolean scalar tensor of Python boolean indicating
+ whether the `call` is meant for training or inference.
+ * `mask`: Boolean input mask.
+- If the layer's `call` method takes a `mask` argument (as some Keras
+ layers do), its default value will be set to the mask generated
+ for `inputs` by the previous layer (if `input` did come from
+ a layer that generated a corresponding mask, i.e. if it came from
+ a Keras layer with masking support.
+
+
+
+#### Raises:
+
+
+* `ValueError`: if the layer's `call` method returns None (an invalid value).
+
+apply
+
+``` python
+apply(
+ inputs,
+ *args,
+ **kwargs
+)
+```
+
+Apply the layer on a input.
+
+This is an alias of `self.__call__`.
+
+#### Arguments:
+
+
+* `inputs`: Input tensor(s).
+* `*args`: additional positional arguments to be passed to `self.call`.
+* `**kwargs`: additional keyword arguments to be passed to `self.call`.
+
+
+#### Returns:
+
+Output tensor(s).
+
+
+build
+
+``` python
+build(input_shape)
+```
+
+Creates the variables of the layer (optional, for subclass implementers).
+
+This is a method that implementers of subclasses of `Layer` or `Model`
+can override if they need a state-creation step in-between
+layer instantiation and layer call.
+
+This is typically used to create the weights of `Layer` subclasses.
+
+#### Arguments:
+
+
+* `input_shape`: Instance of `TensorShape`, or list of instances of
+ `TensorShape` if the layer expects a list of inputs
+ (one instance per input).
+
+compute_mask
+
+``` python
+compute_mask(
+ inputs,
+ mask=None
+)
+```
+
+Computes an output mask tensor.
+
+
+#### Arguments:
+
+
+* `inputs`: Tensor or list of tensors.
+* `mask`: Tensor or list of tensors.
+
+
+#### Returns:
+
+None or a tensor (or list of tensors,
+ one per output tensor of the layer).
+
+
+compute_output_shape
+
+``` python
+compute_output_shape(input_shape)
+```
+
+Computes the output shape of the layer.
+
+Assumes that the layer will be built
+to match that input shape provided.
+
+#### Arguments:
+
+
+* `input_shape`: Shape tuple (tuple of integers)
+ or list of shape tuples (one per output tensor of the layer).
+ Shape tuples can include None for free dimensions,
+ instead of an integer.
+
+
+#### Returns:
+
+An input shape tuple.
+
+
+count_params
+
+``` python
+count_params()
+```
+
+Count the total number of scalars composing the weights.
+
+
+#### Returns:
+
+An integer count.
+
+
+
+#### Raises:
+
+
+* `ValueError`: if the layer isn't yet built
+ (in which case its weights aren't yet defined).
+
+finalize
+
+``` python
+finalize(
+ outputs,
+ final_state,
+ sequence_lengths
+)
+```
+
+
+
+
+from_config
+
+``` python
+from_config(
+ cls,
+ config
+)
+```
+
+Creates a layer from its config.
+
+This method is the reverse of `get_config`,
+capable of instantiating the same layer from the config
+dictionary. It does not handle layer connectivity
+(handled by Network), nor weights (handled by `set_weights`).
+
+#### Arguments:
+
+
+* `config`: A Python dictionary, typically the
+ output of get_config.
+
+
+#### Returns:
+
+A layer instance.
+
+
+get_config
+
+``` python
+get_config()
+```
+
+Returns the config of the layer.
+
+A layer config is a Python dictionary (serializable)
+containing the configuration of a layer.
+The same layer can be reinstantiated later
+(without its trained weights) from this configuration.
+
+The config of a layer does not include connectivity
+information, nor the layer class name. These are handled
+by `Network` (one layer of abstraction above).
+
+#### Returns:
+
+Python dictionary.
+
+
+
+
+``` python
+get_input_at(node_index)
+```
+
+Retrieves the input tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A tensor (or list of tensors if the layer has multiple inputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+
+
+``` python
+get_input_mask_at(node_index)
+```
+
+Retrieves the input mask tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A mask tensor
+(or list of tensors if the layer has multiple inputs).
+
+
+
+
+``` python
+get_input_shape_at(node_index)
+```
+
+Retrieves the input shape(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A shape tuple
+(or list of shape tuples if the layer has multiple inputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_losses_for
+
+``` python
+get_losses_for(inputs)
+```
+
+Retrieves losses relevant to a specific set of inputs.
+
+
+#### Arguments:
+
+
+* `inputs`: Input tensor or list/tuple of input tensors.
+
+
+#### Returns:
+
+List of loss tensors of the layer that depend on `inputs`.
+
+
+get_output_at
+
+``` python
+get_output_at(node_index)
+```
+
+Retrieves the output tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A tensor (or list of tensors if the layer has multiple outputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_output_mask_at
+
+``` python
+get_output_mask_at(node_index)
+```
+
+Retrieves the output mask tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A mask tensor
+(or list of tensors if the layer has multiple outputs).
+
+
+get_output_shape_at
+
+``` python
+get_output_shape_at(node_index)
+```
+
+Retrieves the output shape(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A shape tuple
+(or list of shape tuples if the layer has multiple outputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_updates_for
+
+``` python
+get_updates_for(inputs)
+```
+
+Retrieves updates relevant to a specific set of inputs.
+
+
+#### Arguments:
+
+
+* `inputs`: Input tensor or list/tuple of input tensors.
+
+
+#### Returns:
+
+List of update ops of the layer that depend on `inputs`.
+
+
+get_weights
+
+``` python
+get_weights()
+```
+
+Returns the current weights of the layer.
+
+
+#### Returns:
+
+Weights values as a list of numpy arrays.
+
+
+initialize
+
+``` python
+initialize(
+ inputs,
+ initial_state=None,
+ **kwargs
+)
+```
+
+Initialize the decoder.
+
+
+set_weights
+
+``` python
+set_weights(weights)
+```
+
+Sets the weights of the layer, from Numpy arrays.
+
+
+#### Arguments:
+
+
+* `weights`: a list of Numpy arrays. The number
+ of arrays and their shape must match
+ number of the dimensions of the weights
+ of the layer (i.e. it should match the
+ output of `get_weights`).
+
+
+#### Raises:
+
+
+* `ValueError`: If the provided weights list does not match the
+ layer's specifications.
+
+step
+
+``` python
+step(
+ time,
+ inputs,
+ state
+)
+```
+
+Perform a decoding step.
+
+
+#### Args:
+
+
+* `time`: scalar `int32` tensor.
+* `inputs`: A (structure of) input tensors.
+* `state`: A (structure of) state tensors and TensorArrays.
+
+
+#### Returns:
+
+`(outputs, next_state, next_inputs, finished)`.
+
+
+with_name_scope
+
+``` python
+with_name_scope(
+ cls,
+ method
+)
+```
+
+Decorator to automatically enter the module name scope.
+
+```
+class MyModule(tf.Module):
+ @tf.Module.with_name_scope
+ def __call__(self, x):
+ if not hasattr(self, 'w'):
+ self.w = tf.Variable(tf.random.normal([x.shape[1], 64]))
+ return tf.matmul(x, self.w)
+```
+
+Using the above module would produce `tf.Variable`s and `tf.Tensor`s whose
+names included the module name:
+
+```
+mod = MyModule()
+mod(tf.ones([8, 32]))
+# ==>
+mod.w
+# ==>
+```
+
+#### Args:
+
+
+* `method`: The method to wrap.
+
+
+#### Returns:
+
+The original method wrapped such that it enters the module's name scope.
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/BasicDecoderOutput.md b/docs/api_docs/python/tfa/seq2seq/BasicDecoderOutput.md
new file mode 100644
index 0000000000..7008d26ef0
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/BasicDecoderOutput.md
@@ -0,0 +1,41 @@
+
+
+
+
+
+
+
+# tfa.seq2seq.BasicDecoderOutput
+
+## Class `BasicDecoderOutput`
+
+
+
+
+
+### Aliases:
+
+* Class `tfa.seq2seq.BasicDecoderOutput`
+* Class `tfa.seq2seq.basic_decoder.BasicDecoderOutput`
+
+
+
+Defined in [`seq2seq/basic_decoder.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/basic_decoder.py).
+
+
+
+
+## Properties
+
+rnn_output
+
+
+
+
+sample_id
+
+
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/BeamSearchDecoder.md b/docs/api_docs/python/tfa/seq2seq/BeamSearchDecoder.md
new file mode 100644
index 0000000000..406be683f0
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/BeamSearchDecoder.md
@@ -0,0 +1,1043 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+# tfa.seq2seq.BeamSearchDecoder
+
+## Class `BeamSearchDecoder`
+
+BeamSearch sampling decoder.
+
+Inherits From: [`BeamSearchDecoderMixin`](../../tfa/seq2seq/beam_search_decoder/BeamSearchDecoderMixin.md), [`BaseDecoder`](../../tfa/seq2seq/BaseDecoder.md)
+
+### Aliases:
+
+* Class `tfa.seq2seq.BeamSearchDecoder`
+* Class `tfa.seq2seq.beam_search_decoder.BeamSearchDecoder`
+
+
+
+Defined in [`seq2seq/beam_search_decoder.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/beam_search_decoder.py).
+
+
+
+**NOTE** If you are using the `BeamSearchDecoder` with a cell wrapped in
+`AttentionWrapper`, then you must ensure that:
+
+- The encoder output has been tiled to `beam_width` via
+ `tf.contrib.seq2seq.tile_batch` (NOT `tf.tile`).
+- The `batch_size` argument passed to the `get_initial_state` method of
+ this wrapper is equal to `true_batch_size * beam_width`.
+- The initial state created with `get_initial_state` above contains a
+ `cell_state` value containing properly tiled final state from the
+ encoder.
+
+#### An example:
+
+
+
+```
+tiled_encoder_outputs = tf.contrib.seq2seq.tile_batch(
+ encoder_outputs, multiplier=beam_width)
+tiled_encoder_final_state = tf.contrib.seq2seq.tile_batch(
+ encoder_final_state, multiplier=beam_width)
+tiled_sequence_length = tf.contrib.seq2seq.tile_batch(
+ sequence_length, multiplier=beam_width)
+attention_mechanism = MyFavoriteAttentionMechanism(
+ num_units=attention_depth,
+ memory=tiled_inputs,
+ memory_sequence_length=tiled_sequence_length)
+attention_cell = AttentionWrapper(cell, attention_mechanism, ...)
+decoder_initial_state = attention_cell.get_initial_state(
+ batch_size=true_batch_size * beam_width, dtype=dtype)
+decoder_initial_state = decoder_initial_state.clone(
+ cell_state=tiled_encoder_final_state)
+```
+
+Meanwhile, with `AttentionWrapper`, coverage penalty is suggested to use
+when computing scores (https://arxiv.org/pdf/1609.08144.pdf). It encourages
+the decoding to cover all inputs.
+
+__init__
+
+``` python
+__init__(
+ cell,
+ beam_width,
+ embedding_fn=None,
+ output_layer=None,
+ length_penalty_weight=0.0,
+ coverage_penalty_weight=0.0,
+ reorder_tensor_arrays=True,
+ **kwargs
+)
+```
+
+Initialize the BeamSearchDecoder.
+
+
+#### Args:
+
+
+* `cell`: An `RNNCell` instance.
+* `beam_width`: Python integer, the number of beams.
+* `embedding_fn`: A callable that takes a vector tensor of `ids`
+ (argmax ids).
+* `output_layer`: (Optional) An instance of `tf.keras.layers.Layer`,
+ i.e., `tf.keras.layers.Dense`. Optional layer to apply to the RNN
+ output prior to storing the result or sampling.
+* `length_penalty_weight`: Float weight to penalize length. Disabled with
+ 0.0.
+* `coverage_penalty_weight`: Float weight to penalize the coverage of
+ source sentence. Disabled with 0.0.
+* `reorder_tensor_arrays`: If `True`, `TensorArray`s' elements within the
+ cell state will be reordered according to the beam search path. If
+ the `TensorArray` can be reordered, the stacked form will be
+ returned. Otherwise, the `TensorArray` will be returned as is. Set
+ this flag to `False` if the cell state contains `TensorArray`s that
+ are not amenable to reordering.
+* `**kwargs`: Dict, other keyword arguments for initialization.
+
+
+#### Raises:
+
+
+* `TypeError`: if `cell` is not an instance of `RNNCell`,
+ or `output_layer` is not an instance of `tf.keras.layers.Layer`.
+
+
+
+## Properties
+
+activity_regularizer
+
+Optional regularizer function for the output of this layer.
+
+
+batch_size
+
+
+
+
+dtype
+
+
+
+
+dynamic
+
+
+
+
+
+
+Retrieves the input tensor(s) of a layer.
+
+Only applicable if the layer has exactly one input,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Input tensor or list of input tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+* `AttributeError`: If no inbound nodes are found.
+
+
+
+Retrieves the input mask tensor(s) of a layer.
+
+Only applicable if the layer has exactly one inbound node,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Input mask tensor (potentially None) or list of input
+mask tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+
+
+Retrieves the input shape(s) of a layer.
+
+Only applicable if the layer has exactly one input,
+i.e. if it is connected to one incoming layer, or if all inputs
+have the same shape.
+
+#### Returns:
+
+Input shape, as an integer shape tuple
+(or list of shape tuples, one tuple per input tensor).
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer has no defined input_shape.
+* `RuntimeError`: if called in Eager mode.
+
+
+
+
+
+
+losses
+
+Losses which are associated with this `Layer`.
+
+Variable regularization tensors are created when this property is accessed,
+so it is eager safe: accessing `losses` under a `tf.GradientTape` will
+propagate gradients back to the corresponding variables.
+
+#### Returns:
+
+A list of tensors.
+
+
+metrics
+
+
+
+
+name
+
+
+
+
+name_scope
+
+Returns a `tf.name_scope` instance for this class.
+
+
+non_trainable_variables
+
+
+
+
+non_trainable_weights
+
+
+
+
+output
+
+Retrieves the output tensor(s) of a layer.
+
+Only applicable if the layer has exactly one output,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Output tensor or list of output tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to more than one incoming
+ layers.
+* `RuntimeError`: if called in Eager mode.
+
+output_dtype
+
+
+
+
+output_mask
+
+Retrieves the output mask tensor(s) of a layer.
+
+Only applicable if the layer has exactly one inbound node,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Output mask tensor (potentially None) or list of output
+mask tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+output_shape
+
+Retrieves the output shape(s) of a layer.
+
+Only applicable if the layer has one output,
+or if all outputs have the same shape.
+
+#### Returns:
+
+Output shape, as an integer shape tuple
+(or list of shape tuples, one tuple per output tensor).
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer has no defined output shape.
+* `RuntimeError`: if called in Eager mode.
+
+output_size
+
+
+
+
+submodules
+
+Sequence of all sub-modules.
+
+Submodules are modules which are properties of this module, or found as
+properties of modules which are properties of this module (and so on).
+
+```
+a = tf.Module()
+b = tf.Module()
+c = tf.Module()
+a.b = b
+b.c = c
+assert list(a.submodules) == [b, c]
+assert list(b.submodules) == [c]
+assert list(c.submodules) == []
+```
+
+#### Returns:
+
+A sequence of all submodules.
+
+
+tracks_own_finished
+
+The BeamSearchDecoder shuffles its beams and their finished state.
+
+For this reason, it conflicts with the `dynamic_decode` function's
+tracking of finished states. Setting this property to true avoids
+early stopping of decoding due to mismanagement of the finished state
+in `dynamic_decode`.
+
+#### Returns:
+
+`True`.
+
+
+trainable
+
+
+
+
+trainable_variables
+
+
+
+
+trainable_weights
+
+
+
+
+updates
+
+
+
+
+variables
+
+Returns the list of all layer variables/weights.
+
+Alias of `self.weights`.
+
+#### Returns:
+
+A list of variables.
+
+
+weights
+
+Returns the list of all layer variables/weights.
+
+
+#### Returns:
+
+A list of variables.
+
+
+
+
+## Methods
+
+__call__
+
+``` python
+__call__(
+ inputs,
+ *args,
+ **kwargs
+)
+```
+
+Wraps `call`, applying pre- and post-processing steps.
+
+
+#### Arguments:
+
+
+* `inputs`: input tensor(s).
+* `*args`: additional positional arguments to be passed to `self.call`.
+* `**kwargs`: additional keyword arguments to be passed to `self.call`.
+
+
+#### Returns:
+
+Output tensor(s).
+
+
+
+#### Note:
+
+- The following optional keyword arguments are reserved for specific uses:
+ * `training`: Boolean scalar tensor of Python boolean indicating
+ whether the `call` is meant for training or inference.
+ * `mask`: Boolean input mask.
+- If the layer's `call` method takes a `mask` argument (as some Keras
+ layers do), its default value will be set to the mask generated
+ for `inputs` by the previous layer (if `input` did come from
+ a layer that generated a corresponding mask, i.e. if it came from
+ a Keras layer with masking support.
+
+
+
+#### Raises:
+
+
+* `ValueError`: if the layer's `call` method returns None (an invalid value).
+
+apply
+
+``` python
+apply(
+ inputs,
+ *args,
+ **kwargs
+)
+```
+
+Apply the layer on a input.
+
+This is an alias of `self.__call__`.
+
+#### Arguments:
+
+
+* `inputs`: Input tensor(s).
+* `*args`: additional positional arguments to be passed to `self.call`.
+* `**kwargs`: additional keyword arguments to be passed to `self.call`.
+
+
+#### Returns:
+
+Output tensor(s).
+
+
+build
+
+``` python
+build(input_shape)
+```
+
+Creates the variables of the layer (optional, for subclass implementers).
+
+This is a method that implementers of subclasses of `Layer` or `Model`
+can override if they need a state-creation step in-between
+layer instantiation and layer call.
+
+This is typically used to create the weights of `Layer` subclasses.
+
+#### Arguments:
+
+
+* `input_shape`: Instance of `TensorShape`, or list of instances of
+ `TensorShape` if the layer expects a list of inputs
+ (one instance per input).
+
+compute_mask
+
+``` python
+compute_mask(
+ inputs,
+ mask=None
+)
+```
+
+Computes an output mask tensor.
+
+
+#### Arguments:
+
+
+* `inputs`: Tensor or list of tensors.
+* `mask`: Tensor or list of tensors.
+
+
+#### Returns:
+
+None or a tensor (or list of tensors,
+ one per output tensor of the layer).
+
+
+compute_output_shape
+
+``` python
+compute_output_shape(input_shape)
+```
+
+Computes the output shape of the layer.
+
+Assumes that the layer will be built
+to match that input shape provided.
+
+#### Arguments:
+
+
+* `input_shape`: Shape tuple (tuple of integers)
+ or list of shape tuples (one per output tensor of the layer).
+ Shape tuples can include None for free dimensions,
+ instead of an integer.
+
+
+#### Returns:
+
+An input shape tuple.
+
+
+count_params
+
+``` python
+count_params()
+```
+
+Count the total number of scalars composing the weights.
+
+
+#### Returns:
+
+An integer count.
+
+
+
+#### Raises:
+
+
+* `ValueError`: if the layer isn't yet built
+ (in which case its weights aren't yet defined).
+
+finalize
+
+``` python
+finalize(
+ outputs,
+ final_state,
+ sequence_lengths
+)
+```
+
+Finalize and return the predicted_ids.
+
+
+#### Args:
+
+
+* `outputs`: An instance of BeamSearchDecoderOutput.
+* `final_state`: An instance of BeamSearchDecoderState. Passed through to
+ the output.
+* `sequence_lengths`: An `int64` tensor shaped
+ `[batch_size, beam_width]`. The sequence lengths determined for
+ each beam during decode. **NOTE** These are ignored; the updated
+ sequence lengths are stored in `final_state.lengths`.
+
+
+#### Returns:
+
+
+* `outputs`: An instance of `FinalBeamSearchDecoderOutput` where the
+ predicted_ids are the result of calling _gather_tree.
+* `final_state`: The same input instance of `BeamSearchDecoderState`.
+
+from_config
+
+``` python
+from_config(
+ cls,
+ config
+)
+```
+
+Creates a layer from its config.
+
+This method is the reverse of `get_config`,
+capable of instantiating the same layer from the config
+dictionary. It does not handle layer connectivity
+(handled by Network), nor weights (handled by `set_weights`).
+
+#### Arguments:
+
+
+* `config`: A Python dictionary, typically the
+ output of get_config.
+
+
+#### Returns:
+
+A layer instance.
+
+
+get_config
+
+``` python
+get_config()
+```
+
+Returns the config of the layer.
+
+A layer config is a Python dictionary (serializable)
+containing the configuration of a layer.
+The same layer can be reinstantiated later
+(without its trained weights) from this configuration.
+
+The config of a layer does not include connectivity
+information, nor the layer class name. These are handled
+by `Network` (one layer of abstraction above).
+
+#### Returns:
+
+Python dictionary.
+
+
+
+
+``` python
+get_input_at(node_index)
+```
+
+Retrieves the input tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A tensor (or list of tensors if the layer has multiple inputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+
+
+``` python
+get_input_mask_at(node_index)
+```
+
+Retrieves the input mask tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A mask tensor
+(or list of tensors if the layer has multiple inputs).
+
+
+
+
+``` python
+get_input_shape_at(node_index)
+```
+
+Retrieves the input shape(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A shape tuple
+(or list of shape tuples if the layer has multiple inputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_losses_for
+
+``` python
+get_losses_for(inputs)
+```
+
+Retrieves losses relevant to a specific set of inputs.
+
+
+#### Arguments:
+
+
+* `inputs`: Input tensor or list/tuple of input tensors.
+
+
+#### Returns:
+
+List of loss tensors of the layer that depend on `inputs`.
+
+
+get_output_at
+
+``` python
+get_output_at(node_index)
+```
+
+Retrieves the output tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A tensor (or list of tensors if the layer has multiple outputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_output_mask_at
+
+``` python
+get_output_mask_at(node_index)
+```
+
+Retrieves the output mask tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A mask tensor
+(or list of tensors if the layer has multiple outputs).
+
+
+get_output_shape_at
+
+``` python
+get_output_shape_at(node_index)
+```
+
+Retrieves the output shape(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A shape tuple
+(or list of shape tuples if the layer has multiple outputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_updates_for
+
+``` python
+get_updates_for(inputs)
+```
+
+Retrieves updates relevant to a specific set of inputs.
+
+
+#### Arguments:
+
+
+* `inputs`: Input tensor or list/tuple of input tensors.
+
+
+#### Returns:
+
+List of update ops of the layer that depend on `inputs`.
+
+
+get_weights
+
+``` python
+get_weights()
+```
+
+Returns the current weights of the layer.
+
+
+#### Returns:
+
+Weights values as a list of numpy arrays.
+
+
+initialize
+
+``` python
+initialize(
+ embedding,
+ start_tokens,
+ end_token,
+ initial_state
+)
+```
+
+Initialize the decoder.
+
+
+#### Args:
+
+
+* `embedding`: A tensor from the embedding layer output, which is the
+ `params` argument for `embedding_lookup`.
+* `start_tokens`: `int32` vector shaped `[batch_size]`, the start tokens.
+* `end_token`: `int32` scalar, the token that marks end of decoding.
+* `initial_state`: A (possibly nested tuple of...) tensors and
+TensorArrays.
+
+#### Returns:
+
+`(finished, start_inputs, initial_state)`.
+
+
+#### Raises:
+
+
+* `ValueError`: If `start_tokens` is not a vector or `end_token` is not a
+ scalar.
+
+set_weights
+
+``` python
+set_weights(weights)
+```
+
+Sets the weights of the layer, from Numpy arrays.
+
+
+#### Arguments:
+
+
+* `weights`: a list of Numpy arrays. The number
+ of arrays and their shape must match
+ number of the dimensions of the weights
+ of the layer (i.e. it should match the
+ output of `get_weights`).
+
+
+#### Raises:
+
+
+* `ValueError`: If the provided weights list does not match the
+ layer's specifications.
+
+step
+
+``` python
+step(
+ time,
+ inputs,
+ state,
+ name=None
+)
+```
+
+Perform a decoding step.
+
+
+#### Args:
+
+
+* `time`: scalar `int32` tensor.
+* `inputs`: A (structure of) input tensors.
+* `state`: A (structure of) state tensors and TensorArrays.
+* `name`: Name scope for any created operations.
+
+
+#### Returns:
+
+`(outputs, next_state, next_inputs, finished)`.
+
+
+with_name_scope
+
+``` python
+with_name_scope(
+ cls,
+ method
+)
+```
+
+Decorator to automatically enter the module name scope.
+
+```
+class MyModule(tf.Module):
+ @tf.Module.with_name_scope
+ def __call__(self, x):
+ if not hasattr(self, 'w'):
+ self.w = tf.Variable(tf.random.normal([x.shape[1], 64]))
+ return tf.matmul(x, self.w)
+```
+
+Using the above module would produce `tf.Variable`s and `tf.Tensor`s whose
+names included the module name:
+
+```
+mod = MyModule()
+mod(tf.ones([8, 32]))
+# ==>
+mod.w
+# ==>
+```
+
+#### Args:
+
+
+* `method`: The method to wrap.
+
+
+#### Returns:
+
+The original method wrapped such that it enters the module's name scope.
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/BeamSearchDecoderOutput.md b/docs/api_docs/python/tfa/seq2seq/BeamSearchDecoderOutput.md
new file mode 100644
index 0000000000..c7a3ed1ba3
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/BeamSearchDecoderOutput.md
@@ -0,0 +1,47 @@
+
+
+
+
+
+
+
+
+# tfa.seq2seq.BeamSearchDecoderOutput
+
+## Class `BeamSearchDecoderOutput`
+
+
+
+
+
+### Aliases:
+
+* Class `tfa.seq2seq.BeamSearchDecoderOutput`
+* Class `tfa.seq2seq.beam_search_decoder.BeamSearchDecoderOutput`
+
+
+
+Defined in [`seq2seq/beam_search_decoder.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/beam_search_decoder.py).
+
+
+
+
+## Properties
+
+scores
+
+
+
+
+predicted_ids
+
+
+
+
+parent_ids
+
+
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/BeamSearchDecoderState.md b/docs/api_docs/python/tfa/seq2seq/BeamSearchDecoderState.md
new file mode 100644
index 0000000000..ee2ff86db4
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/BeamSearchDecoderState.md
@@ -0,0 +1,59 @@
+
+
+
+
+
+
+
+
+
+
+# tfa.seq2seq.BeamSearchDecoderState
+
+## Class `BeamSearchDecoderState`
+
+
+
+
+
+### Aliases:
+
+* Class `tfa.seq2seq.BeamSearchDecoderState`
+* Class `tfa.seq2seq.beam_search_decoder.BeamSearchDecoderState`
+
+
+
+Defined in [`seq2seq/beam_search_decoder.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/beam_search_decoder.py).
+
+
+
+
+## Properties
+
+cell_state
+
+
+
+
+log_probs
+
+
+
+
+finished
+
+
+
+
+lengths
+
+
+
+
+accumulated_attention_probs
+
+
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/CustomSampler.md b/docs/api_docs/python/tfa/seq2seq/CustomSampler.md
new file mode 100644
index 0000000000..be8b08328b
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/CustomSampler.md
@@ -0,0 +1,127 @@
+
+
+
+
+
+
+
+
+
+
+
+
+# tfa.seq2seq.CustomSampler
+
+## Class `CustomSampler`
+
+Base abstract class that allows the user to customize sampling.
+
+Inherits From: [`Sampler`](../../tfa/seq2seq/Sampler.md)
+
+### Aliases:
+
+* Class `tfa.seq2seq.CustomSampler`
+* Class `tfa.seq2seq.sampler.CustomSampler`
+
+
+
+Defined in [`seq2seq/sampler.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/sampler.py).
+
+
+
+
+__init__
+
+``` python
+__init__(
+ initialize_fn,
+ sample_fn,
+ next_inputs_fn,
+ sample_ids_shape=None,
+ sample_ids_dtype=None
+)
+```
+
+Initializer.
+
+
+#### Args:
+
+
+* `initialize_fn`: callable that returns `(finished, next_inputs)` for
+ the first iteration.
+* `sample_fn`: callable that takes `(time, outputs, state)` and emits
+ tensor `sample_ids`.
+* `next_inputs_fn`: callable that takes
+ `(time, outputs, state, sample_ids)` and emits
+ `(finished, next_inputs, next_state)`.
+* `sample_ids_shape`: Either a list of integers, or a 1-D Tensor of type
+ `int32`, the shape of each value in the `sample_ids` batch.
+ Defaults to a scalar.
+* `sample_ids_dtype`: The dtype of the `sample_ids` tensor. Defaults to
+ int32.
+
+
+
+## Properties
+
+batch_size
+
+
+
+
+sample_ids_dtype
+
+
+
+
+sample_ids_shape
+
+
+
+
+
+
+## Methods
+
+initialize
+
+``` python
+initialize(
+ inputs,
+ **kwargs
+)
+```
+
+
+
+
+
+
+``` python
+next_inputs(
+ time,
+ outputs,
+ state,
+ sample_ids
+)
+```
+
+
+
+
+sample
+
+``` python
+sample(
+ time,
+ outputs,
+ state
+)
+```
+
+
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/Decoder.md b/docs/api_docs/python/tfa/seq2seq/Decoder.md
new file mode 100644
index 0000000000..da123f73dc
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/Decoder.md
@@ -0,0 +1,154 @@
+
+
+
+
+
+
+
+
+
+
+
+
+# tfa.seq2seq.Decoder
+
+## Class `Decoder`
+
+An RNN Decoder abstract interface object.
+
+
+
+### Aliases:
+
+* Class `tfa.seq2seq.Decoder`
+* Class `tfa.seq2seq.decoder.Decoder`
+
+
+
+Defined in [`seq2seq/decoder.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/decoder.py).
+
+
+
+Concepts used by this interface:
+- `inputs`: (structure of) tensors and TensorArrays that is passed as input
+ to the RNNCell composing the decoder, at each time step.
+- `state`: (structure of) tensors and TensorArrays that is passed to the
+ RNNCell instance as the state.
+- `finished`: boolean tensor telling whether each sequence in the batch is
+ finished.
+- `outputs`: Instance of BasicDecoderOutput. Result of the decoding, at
+ each time step.
+
+## Properties
+
+batch_size
+
+The batch size of input values.
+
+
+output_dtype
+
+A (possibly nested tuple of...) dtype[s].
+
+
+output_size
+
+A (possibly nested tuple of...) integer[s] or `TensorShape`
+object[s].
+
+tracks_own_finished
+
+Describes whether the Decoder keeps track of finished states.
+
+Most decoders will emit a true/false `finished` value independently
+at each time step. In this case, the `dynamic_decode` function keeps
+track of which batch entries are already finished, and performs a
+logical OR to insert new batches to the finished set.
+
+Some decoders, however, shuffle batches / beams between time steps and
+`dynamic_decode` will mix up the finished state across these entries
+because it does not track the reshuffle across time steps. In this
+case, it is up to the decoder to declare that it will keep track of its
+own finished state by setting this property to `True`.
+
+#### Returns:
+
+Python bool.
+
+
+
+
+## Methods
+
+finalize
+
+``` python
+finalize(
+ outputs,
+ final_state,
+ sequence_lengths
+)
+```
+
+
+
+
+initialize
+
+``` python
+initialize(name=None)
+```
+
+Called before any decoding iterations.
+
+This methods must compute initial input values and initial state.
+
+#### Args:
+
+
+* `name`: Name scope for any created operations.
+
+
+#### Returns:
+
+`(finished, initial_inputs, initial_state)`: initial values of
+'finished' flags, inputs and state.
+
+
+step
+
+``` python
+step(
+ time,
+ inputs,
+ state,
+ name=None
+)
+```
+
+Called per step of decoding (but only once for dynamic decoding).
+
+
+#### Args:
+
+
+* `time`: Scalar `int32` tensor. Current step number.
+* `inputs`: RNNCell input (possibly nested tuple of) tensor[s] for this
+ time step.
+* `state`: RNNCell state (possibly nested tuple of) tensor[s] from
+ previous time step.
+* `name`: Name scope for any created operations.
+
+
+#### Returns:
+
+`(outputs, next_state, next_inputs, finished)`: `outputs` is an
+object containing the decoder output, `next_state` is a (structure
+of) state tensors and TensorArrays, `next_inputs` is the tensor that
+should be used as input for the next step, `finished` is a boolean
+tensor telling whether the sequence is complete, for each sequence in
+the batch.
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/FinalBeamSearchDecoderOutput.md b/docs/api_docs/python/tfa/seq2seq/FinalBeamSearchDecoderOutput.md
new file mode 100644
index 0000000000..90d2b3712f
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/FinalBeamSearchDecoderOutput.md
@@ -0,0 +1,50 @@
+
+
+
+
+
+
+
+# tfa.seq2seq.FinalBeamSearchDecoderOutput
+
+## Class `FinalBeamSearchDecoderOutput`
+
+Final outputs returned by the beam search after all decoding is
+
+
+
+### Aliases:
+
+* Class `tfa.seq2seq.FinalBeamSearchDecoderOutput`
+* Class `tfa.seq2seq.beam_search_decoder.FinalBeamSearchDecoderOutput`
+
+
+
+Defined in [`seq2seq/beam_search_decoder.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/beam_search_decoder.py).
+
+
+finished.
+
+#### Args:
+
+
+* `predicted_ids`: The final prediction. A tensor of shape
+ `[batch_size, T, beam_width]` (or `[T, batch_size, beam_width]` if
+ `output_time_major` is True). Beams are ordered from best to worst.
+* `beam_search_decoder_output`: An instance of `BeamSearchDecoderOutput` that
+ describes the state of the beam search.
+
+## Properties
+
+predicted_ids
+
+
+
+
+beam_search_decoder_output
+
+
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/GreedyEmbeddingSampler.md b/docs/api_docs/python/tfa/seq2seq/GreedyEmbeddingSampler.md
new file mode 100644
index 0000000000..7f7d726744
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/GreedyEmbeddingSampler.md
@@ -0,0 +1,138 @@
+
+
+
+
+
+
+
+
+
+
+
+
+# tfa.seq2seq.GreedyEmbeddingSampler
+
+## Class `GreedyEmbeddingSampler`
+
+A sampler for use during inference.
+
+Inherits From: [`Sampler`](../../tfa/seq2seq/Sampler.md)
+
+### Aliases:
+
+* Class `tfa.seq2seq.GreedyEmbeddingSampler`
+* Class `tfa.seq2seq.sampler.GreedyEmbeddingSampler`
+
+
+
+Defined in [`seq2seq/sampler.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/sampler.py).
+
+
+
+Uses the argmax of the output (treated as logits) and passes the
+result through an embedding layer to get the next input.
+
+__init__
+
+``` python
+__init__(embedding_fn=None)
+```
+
+Initializer.
+
+
+#### Args:
+
+
+* `embedding_fn`: A optional callable that takes a vector tensor of `ids`
+ (argmax ids), or the `params` argument for `embedding_lookup`. The
+ returned tensor will be passed to the decoder input. Default to use
+ `tf.nn.embedding_lookup`.
+
+
+
+## Properties
+
+batch_size
+
+
+
+
+sample_ids_dtype
+
+
+
+
+sample_ids_shape
+
+
+
+
+
+
+## Methods
+
+initialize
+
+``` python
+initialize(
+ embedding,
+ start_tokens=None,
+ end_token=None
+)
+```
+
+Initialize the GreedyEmbeddingSampler.
+
+
+#### Args:
+
+
+* `embedding`: tensor that contains embedding states matrix. It will be
+ used to generate generate outputs with start_tokens and end_tokens.
+ The embedding will be ignored if the embedding_fn has been provided
+ at __init__().
+* `start_tokens`: `int32` vector shaped `[batch_size]`, the start tokens.
+* `end_token`: `int32` scalar, the token that marks end of decoding.
+
+
+#### Returns:
+
+Tuple of two items: `(finished, self.start_inputs)`.
+
+
+#### Raises:
+
+
+* `ValueError`: if `start_tokens` is not a 1D tensor or `end_token` is
+ not a scalar.
+
+
+
+``` python
+next_inputs(
+ time,
+ outputs,
+ state,
+ sample_ids
+)
+```
+
+next_inputs_fn for GreedyEmbeddingHelper.
+
+
+sample
+
+``` python
+sample(
+ time,
+ outputs,
+ state
+)
+```
+
+sample for GreedyEmbeddingHelper.
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/InferenceSampler.md b/docs/api_docs/python/tfa/seq2seq/InferenceSampler.md
new file mode 100644
index 0000000000..5c2d4bd043
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/InferenceSampler.md
@@ -0,0 +1,124 @@
+
+
+
+
+
+
+
+
+
+
+
+
+# tfa.seq2seq.InferenceSampler
+
+## Class `InferenceSampler`
+
+A helper to use during inference with a custom sampling function.
+
+Inherits From: [`Sampler`](../../tfa/seq2seq/Sampler.md)
+
+### Aliases:
+
+* Class `tfa.seq2seq.InferenceSampler`
+* Class `tfa.seq2seq.sampler.InferenceSampler`
+
+
+
+Defined in [`seq2seq/sampler.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/sampler.py).
+
+
+
+
+__init__
+
+``` python
+__init__(
+ sample_fn,
+ sample_shape,
+ sample_dtype,
+ end_fn,
+ next_inputs_fn=None
+)
+```
+
+Initializer.
+
+
+#### Args:
+
+
+* `sample_fn`: A callable that takes `outputs` and emits tensor
+ `sample_ids`.
+* `sample_shape`: Either a list of integers, or a 1-D Tensor of type
+ `int32`, the shape of the each sample in the batch returned by
+ `sample_fn`.
+* `sample_dtype`: the dtype of the sample returned by `sample_fn`.
+* `end_fn`: A callable that takes `sample_ids` and emits a `bool` vector
+ shaped `[batch_size]` indicating whether each sample is an end
+ token.
+* `next_inputs_fn`: (Optional) A callable that takes `sample_ids` and
+ returns the next batch of inputs. If not provided, `sample_ids` is
+ used as the next batch of inputs.
+
+
+
+## Properties
+
+batch_size
+
+
+
+
+sample_ids_dtype
+
+
+
+
+sample_ids_shape
+
+
+
+
+
+
+## Methods
+
+initialize
+
+``` python
+initialize(start_inputs)
+```
+
+
+
+
+
+
+``` python
+next_inputs(
+ time,
+ outputs,
+ state,
+ sample_ids
+)
+```
+
+
+
+
+sample
+
+``` python
+sample(
+ time,
+ outputs,
+ state
+)
+```
+
+
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/LuongAttention.md b/docs/api_docs/python/tfa/seq2seq/LuongAttention.md
new file mode 100644
index 0000000000..b727880f34
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/LuongAttention.md
@@ -0,0 +1,918 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+# tfa.seq2seq.LuongAttention
+
+## Class `LuongAttention`
+
+Implements Luong-style (multiplicative) attention scoring.
+
+
+
+### Aliases:
+
+* Class `tfa.seq2seq.LuongAttention`
+* Class `tfa.seq2seq.attention_wrapper.LuongAttention`
+
+
+
+Defined in [`seq2seq/attention_wrapper.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/attention_wrapper.py).
+
+
+
+This attention has two forms. The first is standard Luong attention,
+as described in:
+
+Minh-Thang Luong, Hieu Pham, Christopher D. Manning.
+[Effective Approaches to Attention-based Neural Machine Translation.
+EMNLP 2015.](https://arxiv.org/abs/1508.04025)
+
+The second is the scaled form inspired partly by the normalized form of
+Bahdanau attention.
+
+To enable the second form, construct the object with parameter
+`scale=True`.
+
+__init__
+
+``` python
+__init__(
+ units,
+ memory,
+ memory_sequence_length=None,
+ scale=False,
+ probability_fn='softmax',
+ dtype=None,
+ name='LuongAttention',
+ **kwargs
+)
+```
+
+Construct the AttentionMechanism mechanism.
+
+
+#### Args:
+
+
+* `units`: The depth of the attention mechanism.
+* `memory`: The memory to query; usually the output of an RNN encoder.
+ This tensor should be shaped `[batch_size, max_time, ...]`.
+* `memory_sequence_length`: (optional): Sequence lengths for the batch
+ entries in memory. If provided, the memory tensor rows are masked
+ with zeros for values past the respective sequence lengths.
+* `scale`: Python boolean. Whether to scale the energy term.
+* `probability_fn`: (optional) string, the name of function to convert
+ the attention score to probabilities. The default is `softmax`
+ which is `tf.nn.softmax`. Other options is `hardmax`, which is
+ hardmax() within this module. Any other value will result
+ intovalidation error. Default to use `softmax`.
+* `dtype`: The data type for the memory layer of the attention mechanism.
+* `name`: Name to use when creating ops.
+* `**kwargs`: Dictionary that contains other common arguments for layer
+ creation.
+
+
+
+## Properties
+
+activity_regularizer
+
+Optional regularizer function for the output of this layer.
+
+
+alignments_size
+
+
+
+
+dtype
+
+
+
+
+dynamic
+
+
+
+
+
+
+Retrieves the input tensor(s) of a layer.
+
+Only applicable if the layer has exactly one input,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Input tensor or list of input tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+* `AttributeError`: If no inbound nodes are found.
+
+
+
+Retrieves the input mask tensor(s) of a layer.
+
+Only applicable if the layer has exactly one inbound node,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Input mask tensor (potentially None) or list of input
+mask tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+
+
+Retrieves the input shape(s) of a layer.
+
+Only applicable if the layer has exactly one input,
+i.e. if it is connected to one incoming layer, or if all inputs
+have the same shape.
+
+#### Returns:
+
+Input shape, as an integer shape tuple
+(or list of shape tuples, one tuple per input tensor).
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer has no defined input_shape.
+* `RuntimeError`: if called in Eager mode.
+
+
+
+
+
+
+losses
+
+Losses which are associated with this `Layer`.
+
+Variable regularization tensors are created when this property is accessed,
+so it is eager safe: accessing `losses` under a `tf.GradientTape` will
+propagate gradients back to the corresponding variables.
+
+#### Returns:
+
+A list of tensors.
+
+
+metrics
+
+
+
+
+name
+
+
+
+
+name_scope
+
+Returns a `tf.name_scope` instance for this class.
+
+
+non_trainable_variables
+
+
+
+
+non_trainable_weights
+
+
+
+
+output
+
+Retrieves the output tensor(s) of a layer.
+
+Only applicable if the layer has exactly one output,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Output tensor or list of output tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to more than one incoming
+ layers.
+* `RuntimeError`: if called in Eager mode.
+
+output_mask
+
+Retrieves the output mask tensor(s) of a layer.
+
+Only applicable if the layer has exactly one inbound node,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Output mask tensor (potentially None) or list of output
+mask tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+output_shape
+
+Retrieves the output shape(s) of a layer.
+
+Only applicable if the layer has one output,
+or if all outputs have the same shape.
+
+#### Returns:
+
+Output shape, as an integer shape tuple
+(or list of shape tuples, one tuple per output tensor).
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer has no defined output shape.
+* `RuntimeError`: if called in Eager mode.
+
+state_size
+
+
+
+
+submodules
+
+Sequence of all sub-modules.
+
+Submodules are modules which are properties of this module, or found as
+properties of modules which are properties of this module (and so on).
+
+```
+a = tf.Module()
+b = tf.Module()
+c = tf.Module()
+a.b = b
+b.c = c
+assert list(a.submodules) == [b, c]
+assert list(b.submodules) == [c]
+assert list(c.submodules) == []
+```
+
+#### Returns:
+
+A sequence of all submodules.
+
+
+trainable
+
+
+
+
+trainable_variables
+
+
+
+
+trainable_weights
+
+
+
+
+updates
+
+
+
+
+variables
+
+Returns the list of all layer variables/weights.
+
+Alias of `self.weights`.
+
+#### Returns:
+
+A list of variables.
+
+
+weights
+
+Returns the list of all layer variables/weights.
+
+
+#### Returns:
+
+A list of variables.
+
+
+
+
+## Methods
+
+__call__
+
+``` python
+__call__(
+ inputs,
+ **kwargs
+)
+```
+
+Preprocess the inputs before calling `base_layer.__call__()`.
+
+Note that there are situation here, one for setup memory, and one with
+actual query and state.
+1. When the memory has not been configured, we just pass all the param
+ to base_layer.__call__(), which will then invoke self.call() with
+ proper inputs, which allows this class to setup memory.
+2. When the memory has already been setup, the input should contain
+ query and state, and optionally processed memory. If the processed
+ memory is not included in the input, we will have to append it to
+ the inputs and give it to the base_layer.__call__(). The processed
+ memory is the output of first invocation of self.__call__(). If we
+ don't add it here, then from keras perspective, the graph is
+ disconnected since the output from previous call is never used.
+
+#### Args:
+
+
+* `inputs`: the inputs tensors.
+* `**kwargs`: dict, other keyeword arguments for the `__call__()`
+
+apply
+
+``` python
+apply(
+ inputs,
+ *args,
+ **kwargs
+)
+```
+
+Apply the layer on a input.
+
+This is an alias of `self.__call__`.
+
+#### Arguments:
+
+
+* `inputs`: Input tensor(s).
+* `*args`: additional positional arguments to be passed to `self.call`.
+* `**kwargs`: additional keyword arguments to be passed to `self.call`.
+
+
+#### Returns:
+
+Output tensor(s).
+
+
+build
+
+``` python
+build(input_shape)
+```
+
+
+
+
+compute_mask
+
+``` python
+compute_mask(
+ inputs,
+ mask=None
+)
+```
+
+
+
+
+compute_output_shape
+
+``` python
+compute_output_shape(input_shape)
+```
+
+Computes the output shape of the layer.
+
+Assumes that the layer will be built
+to match that input shape provided.
+
+#### Arguments:
+
+
+* `input_shape`: Shape tuple (tuple of integers)
+ or list of shape tuples (one per output tensor of the layer).
+ Shape tuples can include None for free dimensions,
+ instead of an integer.
+
+
+#### Returns:
+
+An input shape tuple.
+
+
+count_params
+
+``` python
+count_params()
+```
+
+Count the total number of scalars composing the weights.
+
+
+#### Returns:
+
+An integer count.
+
+
+
+#### Raises:
+
+
+* `ValueError`: if the layer isn't yet built
+ (in which case its weights aren't yet defined).
+
+deserialize_inner_layer_from_config
+
+``` python
+deserialize_inner_layer_from_config(
+ cls,
+ config,
+ custom_objects
+)
+```
+
+Helper method that reconstruct the query and memory from the config.
+
+In the get_config() method, the query and memory layer configs are
+serialized into dict for persistence, this method perform the reverse
+action to reconstruct the layer from the config.
+
+#### Args:
+
+
+* `config`: dict, the configs that will be used to reconstruct the
+ object.
+* `custom_objects`: dict mapping class names (or function names) of
+ custom (non-Keras) objects to class/functions.
+
+#### Returns:
+
+
+* `config`: dict, the config with layer instance created, which is ready
+ to be used as init parameters.
+
+from_config
+
+``` python
+@classmethod
+from_config(
+ cls,
+ config,
+ custom_objects=None
+)
+```
+
+
+
+
+get_config
+
+``` python
+get_config()
+```
+
+
+
+
+
+
+``` python
+get_input_at(node_index)
+```
+
+Retrieves the input tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A tensor (or list of tensors if the layer has multiple inputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+
+
+``` python
+get_input_mask_at(node_index)
+```
+
+Retrieves the input mask tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A mask tensor
+(or list of tensors if the layer has multiple inputs).
+
+
+
+
+``` python
+get_input_shape_at(node_index)
+```
+
+Retrieves the input shape(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A shape tuple
+(or list of shape tuples if the layer has multiple inputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_losses_for
+
+``` python
+get_losses_for(inputs)
+```
+
+Retrieves losses relevant to a specific set of inputs.
+
+
+#### Arguments:
+
+
+* `inputs`: Input tensor or list/tuple of input tensors.
+
+
+#### Returns:
+
+List of loss tensors of the layer that depend on `inputs`.
+
+
+get_output_at
+
+``` python
+get_output_at(node_index)
+```
+
+Retrieves the output tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A tensor (or list of tensors if the layer has multiple outputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_output_mask_at
+
+``` python
+get_output_mask_at(node_index)
+```
+
+Retrieves the output mask tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A mask tensor
+(or list of tensors if the layer has multiple outputs).
+
+
+get_output_shape_at
+
+``` python
+get_output_shape_at(node_index)
+```
+
+Retrieves the output shape(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A shape tuple
+(or list of shape tuples if the layer has multiple outputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_updates_for
+
+``` python
+get_updates_for(inputs)
+```
+
+Retrieves updates relevant to a specific set of inputs.
+
+
+#### Arguments:
+
+
+* `inputs`: Input tensor or list/tuple of input tensors.
+
+
+#### Returns:
+
+List of update ops of the layer that depend on `inputs`.
+
+
+get_weights
+
+``` python
+get_weights()
+```
+
+Returns the current weights of the layer.
+
+
+#### Returns:
+
+Weights values as a list of numpy arrays.
+
+
+initial_alignments
+
+``` python
+initial_alignments(
+ batch_size,
+ dtype
+)
+```
+
+Creates the initial alignment values for the `AttentionWrapper`
+class.
+
+This is important for AttentionMechanisms that use the previous
+alignment to calculate the alignment at the next time step
+(e.g. monotonic attention).
+
+The default behavior is to return a tensor of all zeros.
+
+#### Args:
+
+
+* `batch_size`: `int32` scalar, the batch_size.
+* `dtype`: The `dtype`.
+
+
+#### Returns:
+
+A `dtype` tensor shaped `[batch_size, alignments_size]`
+(`alignments_size` is the values' `max_time`).
+
+
+initial_state
+
+``` python
+initial_state(
+ batch_size,
+ dtype
+)
+```
+
+Creates the initial state values for the `AttentionWrapper` class.
+
+This is important for AttentionMechanisms that use the previous
+alignment to calculate the alignment at the next time step
+(e.g. monotonic attention).
+
+The default behavior is to return the same output as
+initial_alignments.
+
+#### Args:
+
+
+* `batch_size`: `int32` scalar, the batch_size.
+* `dtype`: The `dtype`.
+
+
+#### Returns:
+
+A structure of all-zero tensors with shapes as described by
+`state_size`.
+
+
+set_weights
+
+``` python
+set_weights(weights)
+```
+
+Sets the weights of the layer, from Numpy arrays.
+
+
+#### Arguments:
+
+
+* `weights`: a list of Numpy arrays. The number
+ of arrays and their shape must match
+ number of the dimensions of the weights
+ of the layer (i.e. it should match the
+ output of `get_weights`).
+
+
+#### Raises:
+
+
+* `ValueError`: If the provided weights list does not match the
+ layer's specifications.
+
+with_name_scope
+
+``` python
+with_name_scope(
+ cls,
+ method
+)
+```
+
+Decorator to automatically enter the module name scope.
+
+```
+class MyModule(tf.Module):
+ @tf.Module.with_name_scope
+ def __call__(self, x):
+ if not hasattr(self, 'w'):
+ self.w = tf.Variable(tf.random.normal([x.shape[1], 64]))
+ return tf.matmul(x, self.w)
+```
+
+Using the above module would produce `tf.Variable`s and `tf.Tensor`s whose
+names included the module name:
+
+```
+mod = MyModule()
+mod(tf.ones([8, 32]))
+# ==>
+mod.w
+# ==>
+```
+
+#### Args:
+
+
+* `method`: The method to wrap.
+
+
+#### Returns:
+
+The original method wrapped such that it enters the module's name scope.
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/LuongMonotonicAttention.md b/docs/api_docs/python/tfa/seq2seq/LuongMonotonicAttention.md
new file mode 100644
index 0000000000..c1d6721f9c
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/LuongMonotonicAttention.md
@@ -0,0 +1,920 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+# tfa.seq2seq.LuongMonotonicAttention
+
+## Class `LuongMonotonicAttention`
+
+Monotonic attention mechanism with Luong-style energy function.
+
+
+
+### Aliases:
+
+* Class `tfa.seq2seq.LuongMonotonicAttention`
+* Class `tfa.seq2seq.attention_wrapper.LuongMonotonicAttention`
+
+
+
+Defined in [`seq2seq/attention_wrapper.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/attention_wrapper.py).
+
+
+
+This type of attention enforces a monotonic constraint on the attention
+distributions; that is once the model attends to a given point in the
+memory it can't attend to any prior points at subsequence output timesteps.
+It achieves this by using the _monotonic_probability_fn instead of softmax
+to construct its attention distributions. Otherwise, it is equivalent to
+LuongAttention. This approach is proposed in
+
+[Colin Raffel, Minh-Thang Luong, Peter J. Liu, Ron J. Weiss, Douglas Eck,
+"Online and Linear-Time Attention by Enforcing Monotonic Alignments."
+ICML 2017.](https://arxiv.org/abs/1704.00784)
+
+__init__
+
+``` python
+__init__(
+ units,
+ memory,
+ memory_sequence_length=None,
+ scale=False,
+ sigmoid_noise=0.0,
+ sigmoid_noise_seed=None,
+ score_bias_init=0.0,
+ mode='parallel',
+ dtype=None,
+ name='LuongMonotonicAttention',
+ **kwargs
+)
+```
+
+Construct the Attention mechanism.
+
+
+#### Args:
+
+
+* `units`: The depth of the query mechanism.
+* `memory`: The memory to query; usually the output of an RNN encoder.
+ This tensor should be shaped `[batch_size, max_time, ...]`.
+* `memory_sequence_length`: (optional): Sequence lengths for the batch
+ entries in memory. If provided, the memory tensor rows are masked
+ with zeros for values past the respective sequence lengths.
+* `scale`: Python boolean. Whether to scale the energy term.
+* `sigmoid_noise`: Standard deviation of pre-sigmoid noise. See the
+ docstring for `_monotonic_probability_fn` for more information.
+* `sigmoid_noise_seed`: (optional) Random seed for pre-sigmoid noise.
+* `score_bias_init`: Initial value for score bias scalar. It's
+ recommended to initialize this to a negative value when the length
+ of the memory is large.
+* `mode`: How to compute the attention distribution. Must be one of
+ 'recursive', 'parallel', or 'hard'. See the docstring for
+ `tf.contrib.seq2seq.monotonic_attention` for more information.
+* `dtype`: The data type for the query and memory layers of the attention
+ mechanism.
+* `name`: Name to use when creating ops.
+* `**kwargs`: Dictionary that contains other common arguments for layer
+ creation.
+
+
+
+## Properties
+
+activity_regularizer
+
+Optional regularizer function for the output of this layer.
+
+
+alignments_size
+
+
+
+
+dtype
+
+
+
+
+dynamic
+
+
+
+
+
+
+Retrieves the input tensor(s) of a layer.
+
+Only applicable if the layer has exactly one input,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Input tensor or list of input tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+* `AttributeError`: If no inbound nodes are found.
+
+
+
+Retrieves the input mask tensor(s) of a layer.
+
+Only applicable if the layer has exactly one inbound node,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Input mask tensor (potentially None) or list of input
+mask tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+
+
+Retrieves the input shape(s) of a layer.
+
+Only applicable if the layer has exactly one input,
+i.e. if it is connected to one incoming layer, or if all inputs
+have the same shape.
+
+#### Returns:
+
+Input shape, as an integer shape tuple
+(or list of shape tuples, one tuple per input tensor).
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer has no defined input_shape.
+* `RuntimeError`: if called in Eager mode.
+
+
+
+
+
+
+losses
+
+Losses which are associated with this `Layer`.
+
+Variable regularization tensors are created when this property is accessed,
+so it is eager safe: accessing `losses` under a `tf.GradientTape` will
+propagate gradients back to the corresponding variables.
+
+#### Returns:
+
+A list of tensors.
+
+
+metrics
+
+
+
+
+name
+
+
+
+
+name_scope
+
+Returns a `tf.name_scope` instance for this class.
+
+
+non_trainable_variables
+
+
+
+
+non_trainable_weights
+
+
+
+
+output
+
+Retrieves the output tensor(s) of a layer.
+
+Only applicable if the layer has exactly one output,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Output tensor or list of output tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to more than one incoming
+ layers.
+* `RuntimeError`: if called in Eager mode.
+
+output_mask
+
+Retrieves the output mask tensor(s) of a layer.
+
+Only applicable if the layer has exactly one inbound node,
+i.e. if it is connected to one incoming layer.
+
+#### Returns:
+
+Output mask tensor (potentially None) or list of output
+mask tensors.
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer is connected to
+more than one incoming layers.
+
+output_shape
+
+Retrieves the output shape(s) of a layer.
+
+Only applicable if the layer has one output,
+or if all outputs have the same shape.
+
+#### Returns:
+
+Output shape, as an integer shape tuple
+(or list of shape tuples, one tuple per output tensor).
+
+
+
+#### Raises:
+
+
+* `AttributeError`: if the layer has no defined output shape.
+* `RuntimeError`: if called in Eager mode.
+
+state_size
+
+
+
+
+submodules
+
+Sequence of all sub-modules.
+
+Submodules are modules which are properties of this module, or found as
+properties of modules which are properties of this module (and so on).
+
+```
+a = tf.Module()
+b = tf.Module()
+c = tf.Module()
+a.b = b
+b.c = c
+assert list(a.submodules) == [b, c]
+assert list(b.submodules) == [c]
+assert list(c.submodules) == []
+```
+
+#### Returns:
+
+A sequence of all submodules.
+
+
+trainable
+
+
+
+
+trainable_variables
+
+
+
+
+trainable_weights
+
+
+
+
+updates
+
+
+
+
+variables
+
+Returns the list of all layer variables/weights.
+
+Alias of `self.weights`.
+
+#### Returns:
+
+A list of variables.
+
+
+weights
+
+Returns the list of all layer variables/weights.
+
+
+#### Returns:
+
+A list of variables.
+
+
+
+
+## Methods
+
+__call__
+
+``` python
+__call__(
+ inputs,
+ **kwargs
+)
+```
+
+Preprocess the inputs before calling `base_layer.__call__()`.
+
+Note that there are situation here, one for setup memory, and one with
+actual query and state.
+1. When the memory has not been configured, we just pass all the param
+ to base_layer.__call__(), which will then invoke self.call() with
+ proper inputs, which allows this class to setup memory.
+2. When the memory has already been setup, the input should contain
+ query and state, and optionally processed memory. If the processed
+ memory is not included in the input, we will have to append it to
+ the inputs and give it to the base_layer.__call__(). The processed
+ memory is the output of first invocation of self.__call__(). If we
+ don't add it here, then from keras perspective, the graph is
+ disconnected since the output from previous call is never used.
+
+#### Args:
+
+
+* `inputs`: the inputs tensors.
+* `**kwargs`: dict, other keyeword arguments for the `__call__()`
+
+apply
+
+``` python
+apply(
+ inputs,
+ *args,
+ **kwargs
+)
+```
+
+Apply the layer on a input.
+
+This is an alias of `self.__call__`.
+
+#### Arguments:
+
+
+* `inputs`: Input tensor(s).
+* `*args`: additional positional arguments to be passed to `self.call`.
+* `**kwargs`: additional keyword arguments to be passed to `self.call`.
+
+
+#### Returns:
+
+Output tensor(s).
+
+
+build
+
+``` python
+build(input_shape)
+```
+
+
+
+
+compute_mask
+
+``` python
+compute_mask(
+ inputs,
+ mask=None
+)
+```
+
+
+
+
+compute_output_shape
+
+``` python
+compute_output_shape(input_shape)
+```
+
+Computes the output shape of the layer.
+
+Assumes that the layer will be built
+to match that input shape provided.
+
+#### Arguments:
+
+
+* `input_shape`: Shape tuple (tuple of integers)
+ or list of shape tuples (one per output tensor of the layer).
+ Shape tuples can include None for free dimensions,
+ instead of an integer.
+
+
+#### Returns:
+
+An input shape tuple.
+
+
+count_params
+
+``` python
+count_params()
+```
+
+Count the total number of scalars composing the weights.
+
+
+#### Returns:
+
+An integer count.
+
+
+
+#### Raises:
+
+
+* `ValueError`: if the layer isn't yet built
+ (in which case its weights aren't yet defined).
+
+deserialize_inner_layer_from_config
+
+``` python
+deserialize_inner_layer_from_config(
+ cls,
+ config,
+ custom_objects
+)
+```
+
+Helper method that reconstruct the query and memory from the config.
+
+In the get_config() method, the query and memory layer configs are
+serialized into dict for persistence, this method perform the reverse
+action to reconstruct the layer from the config.
+
+#### Args:
+
+
+* `config`: dict, the configs that will be used to reconstruct the
+ object.
+* `custom_objects`: dict mapping class names (or function names) of
+ custom (non-Keras) objects to class/functions.
+
+#### Returns:
+
+
+* `config`: dict, the config with layer instance created, which is ready
+ to be used as init parameters.
+
+from_config
+
+``` python
+@classmethod
+from_config(
+ cls,
+ config,
+ custom_objects=None
+)
+```
+
+
+
+
+get_config
+
+``` python
+get_config()
+```
+
+
+
+
+
+
+``` python
+get_input_at(node_index)
+```
+
+Retrieves the input tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A tensor (or list of tensors if the layer has multiple inputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+
+
+``` python
+get_input_mask_at(node_index)
+```
+
+Retrieves the input mask tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A mask tensor
+(or list of tensors if the layer has multiple inputs).
+
+
+
+
+``` python
+get_input_shape_at(node_index)
+```
+
+Retrieves the input shape(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A shape tuple
+(or list of shape tuples if the layer has multiple inputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_losses_for
+
+``` python
+get_losses_for(inputs)
+```
+
+Retrieves losses relevant to a specific set of inputs.
+
+
+#### Arguments:
+
+
+* `inputs`: Input tensor or list/tuple of input tensors.
+
+
+#### Returns:
+
+List of loss tensors of the layer that depend on `inputs`.
+
+
+get_output_at
+
+``` python
+get_output_at(node_index)
+```
+
+Retrieves the output tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A tensor (or list of tensors if the layer has multiple outputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_output_mask_at
+
+``` python
+get_output_mask_at(node_index)
+```
+
+Retrieves the output mask tensor(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A mask tensor
+(or list of tensors if the layer has multiple outputs).
+
+
+get_output_shape_at
+
+``` python
+get_output_shape_at(node_index)
+```
+
+Retrieves the output shape(s) of a layer at a given node.
+
+
+#### Arguments:
+
+
+* `node_index`: Integer, index of the node
+ from which to retrieve the attribute.
+ E.g. `node_index=0` will correspond to the
+ first time the layer was called.
+
+
+#### Returns:
+
+A shape tuple
+(or list of shape tuples if the layer has multiple outputs).
+
+
+
+#### Raises:
+
+
+* `RuntimeError`: If called in Eager mode.
+
+get_updates_for
+
+``` python
+get_updates_for(inputs)
+```
+
+Retrieves updates relevant to a specific set of inputs.
+
+
+#### Arguments:
+
+
+* `inputs`: Input tensor or list/tuple of input tensors.
+
+
+#### Returns:
+
+List of update ops of the layer that depend on `inputs`.
+
+
+get_weights
+
+``` python
+get_weights()
+```
+
+Returns the current weights of the layer.
+
+
+#### Returns:
+
+Weights values as a list of numpy arrays.
+
+
+initial_alignments
+
+``` python
+initial_alignments(
+ batch_size,
+ dtype
+)
+```
+
+Creates the initial alignment values for the monotonic attentions.
+
+Initializes to dirac distributions, i.e.
+[1, 0, 0, ...memory length..., 0] for all entries in the batch.
+
+#### Args:
+
+
+* `batch_size`: `int32` scalar, the batch_size.
+* `dtype`: The `dtype`.
+
+
+#### Returns:
+
+A `dtype` tensor shaped `[batch_size, alignments_size]`
+(`alignments_size` is the values' `max_time`).
+
+
+initial_state
+
+``` python
+initial_state(
+ batch_size,
+ dtype
+)
+```
+
+Creates the initial state values for the `AttentionWrapper` class.
+
+This is important for AttentionMechanisms that use the previous
+alignment to calculate the alignment at the next time step
+(e.g. monotonic attention).
+
+The default behavior is to return the same output as
+initial_alignments.
+
+#### Args:
+
+
+* `batch_size`: `int32` scalar, the batch_size.
+* `dtype`: The `dtype`.
+
+
+#### Returns:
+
+A structure of all-zero tensors with shapes as described by
+`state_size`.
+
+
+set_weights
+
+``` python
+set_weights(weights)
+```
+
+Sets the weights of the layer, from Numpy arrays.
+
+
+#### Arguments:
+
+
+* `weights`: a list of Numpy arrays. The number
+ of arrays and their shape must match
+ number of the dimensions of the weights
+ of the layer (i.e. it should match the
+ output of `get_weights`).
+
+
+#### Raises:
+
+
+* `ValueError`: If the provided weights list does not match the
+ layer's specifications.
+
+with_name_scope
+
+``` python
+with_name_scope(
+ cls,
+ method
+)
+```
+
+Decorator to automatically enter the module name scope.
+
+```
+class MyModule(tf.Module):
+ @tf.Module.with_name_scope
+ def __call__(self, x):
+ if not hasattr(self, 'w'):
+ self.w = tf.Variable(tf.random.normal([x.shape[1], 64]))
+ return tf.matmul(x, self.w)
+```
+
+Using the above module would produce `tf.Variable`s and `tf.Tensor`s whose
+names included the module name:
+
+```
+mod = MyModule()
+mod(tf.ones([8, 32]))
+# ==>
+mod.w
+# ==>
+```
+
+#### Args:
+
+
+* `method`: The method to wrap.
+
+
+#### Returns:
+
+The original method wrapped such that it enters the module's name scope.
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/SampleEmbeddingSampler.md b/docs/api_docs/python/tfa/seq2seq/SampleEmbeddingSampler.md
new file mode 100644
index 0000000000..0e5c21769f
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/SampleEmbeddingSampler.md
@@ -0,0 +1,155 @@
+
+
+
+
+
+
+
+
+
+
+
+
+# tfa.seq2seq.SampleEmbeddingSampler
+
+## Class `SampleEmbeddingSampler`
+
+A sampler for use during inference.
+
+Inherits From: [`GreedyEmbeddingSampler`](../../tfa/seq2seq/GreedyEmbeddingSampler.md)
+
+### Aliases:
+
+* Class `tfa.seq2seq.SampleEmbeddingSampler`
+* Class `tfa.seq2seq.sampler.SampleEmbeddingSampler`
+
+
+
+Defined in [`seq2seq/sampler.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/sampler.py).
+
+
+
+Uses sampling (from a distribution) instead of argmax and passes the
+result through an embedding layer to get the next input.
+
+__init__
+
+``` python
+__init__(
+ embedding_fn=None,
+ softmax_temperature=None,
+ seed=None
+)
+```
+
+Initializer.
+
+
+#### Args:
+
+
+* `embedding_fn`: (Optional) A callable that takes a vector tensor of
+ `ids` (argmax ids), or the `params` argument for
+ `embedding_lookup`. The returned tensor will be passed to the
+ decoder input.
+* `softmax_temperature`: (Optional) `float32` scalar, value to divide the
+ logits by before computing the softmax. Larger values (above 1.0)
+ result in more random samples, while smaller values push the
+ sampling distribution towards the argmax. Must be strictly greater
+ than 0. Defaults to 1.0.
+* `seed`: (Optional) The sampling seed.
+
+
+#### Raises:
+
+
+* `ValueError`: if `start_tokens` is not a 1D tensor or `end_token` is
+ not a scalar.
+
+
+
+## Properties
+
+batch_size
+
+
+
+
+sample_ids_dtype
+
+
+
+
+sample_ids_shape
+
+
+
+
+
+
+## Methods
+
+initialize
+
+``` python
+initialize(
+ embedding,
+ start_tokens=None,
+ end_token=None
+)
+```
+
+Initialize the GreedyEmbeddingSampler.
+
+
+#### Args:
+
+
+* `embedding`: tensor that contains embedding states matrix. It will be
+ used to generate generate outputs with start_tokens and end_tokens.
+ The embedding will be ignored if the embedding_fn has been provided
+ at __init__().
+* `start_tokens`: `int32` vector shaped `[batch_size]`, the start tokens.
+* `end_token`: `int32` scalar, the token that marks end of decoding.
+
+
+#### Returns:
+
+Tuple of two items: `(finished, self.start_inputs)`.
+
+
+#### Raises:
+
+
+* `ValueError`: if `start_tokens` is not a 1D tensor or `end_token` is
+ not a scalar.
+
+
+
+``` python
+next_inputs(
+ time,
+ outputs,
+ state,
+ sample_ids
+)
+```
+
+next_inputs_fn for GreedyEmbeddingHelper.
+
+
+sample
+
+``` python
+sample(
+ time,
+ outputs,
+ state
+)
+```
+
+sample for SampleEmbeddingHelper.
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/Sampler.md b/docs/api_docs/python/tfa/seq2seq/Sampler.md
new file mode 100644
index 0000000000..cfd9219217
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/Sampler.md
@@ -0,0 +1,128 @@
+
+
+
+
+
+
+
+
+
+
+
+# tfa.seq2seq.Sampler
+
+## Class `Sampler`
+
+Interface for implementing sampling in seq2seq decoders.
+
+
+
+### Aliases:
+
+* Class `tfa.seq2seq.Sampler`
+* Class `tfa.seq2seq.sampler.Sampler`
+
+
+
+Defined in [`seq2seq/sampler.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/sampler.py).
+
+
+
+Sampler instances are used by `BasicDecoder`. The normal usage of a sampler
+is like below:
+sampler = Sampler(init_args)
+(initial_finished, initial_inputs) = sampler.initialize(input_tensors)
+for time_step in range(time):
+ cell_output, cell_state = cell.call(cell_input, previous_state)
+ sample_ids = sampler.sample(time_step, cell_output, cell_state)
+ (finished, next_inputs, next_state) = sampler.next_inputs(
+ time_step,cell_output, cell_state)
+
+Note that all the tensor input should not be feed to Sampler as __init__()
+parameters, instead, they should be feed by decoders via initialize().
+
+## Properties
+
+batch_size
+
+Batch size of tensor returned by `sample`.
+
+Returns a scalar int32 tensor. The return value might not
+available before the invocation of initialize(), in this case,
+ValueError is raised.
+
+sample_ids_dtype
+
+DType of tensor returned by `sample`.
+
+Returns a DType. The return value might not available before the
+invocation of initialize().
+
+sample_ids_shape
+
+Shape of tensor returned by `sample`, excluding the batch dimension.
+
+Returns a `TensorShape`. The return value might not available
+before the invocation of initialize().
+
+
+
+## Methods
+
+initialize
+
+``` python
+initialize(
+ inputs,
+ **kwargs
+)
+```
+
+initialize the sampler with the input tensors.
+
+This method suppose to be only invoke once before the calling other
+methods of the Sampler.
+
+#### Args:
+
+
+* `inputs`: A (structure of) input tensors, it could be a nested tuple or
+ a single tensor.
+* `**kwargs`: Other kwargs for initialization. It could contain tensors
+ like mask for inputs, or non tensor parameter.
+
+
+#### Returns:
+
+`(initial_finished, initial_inputs)`.
+
+
+
+
+``` python
+next_inputs(
+ time,
+ outputs,
+ state,
+ sample_ids
+)
+```
+
+Returns `(finished, next_inputs, next_state)`.
+
+
+sample
+
+``` python
+sample(
+ time,
+ outputs,
+ state
+)
+```
+
+Returns `sample_ids`.
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/ScheduledEmbeddingTrainingSampler.md b/docs/api_docs/python/tfa/seq2seq/ScheduledEmbeddingTrainingSampler.md
new file mode 100644
index 0000000000..5f4f46f74d
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/ScheduledEmbeddingTrainingSampler.md
@@ -0,0 +1,133 @@
+
+
+
+
+
+
+
+
+
+
+
+
+# tfa.seq2seq.ScheduledEmbeddingTrainingSampler
+
+## Class `ScheduledEmbeddingTrainingSampler`
+
+A training sampler that adds scheduled sampling.
+
+Inherits From: [`TrainingSampler`](../../tfa/seq2seq/TrainingSampler.md)
+
+### Aliases:
+
+* Class `tfa.seq2seq.ScheduledEmbeddingTrainingSampler`
+* Class `tfa.seq2seq.sampler.ScheduledEmbeddingTrainingSampler`
+
+
+
+Defined in [`seq2seq/sampler.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/sampler.py).
+
+
+
+Returns -1s for sample_ids where no sampling took place; valid
+sample id values elsewhere.
+
+__init__
+
+``` python
+__init__(
+ sampling_probability,
+ embedding_fn=None,
+ time_major=False,
+ seed=None,
+ scheduling_seed=None
+)
+```
+
+Initializer.
+
+
+#### Args:
+
+
+* `sampling_probability`: A `float32` 0-D or 1-D tensor: the probability
+ of sampling categorically from the output ids instead of reading
+ directly from the inputs.
+* `embedding_fn`: A callable that takes a vector tensor of `ids`
+ (argmax ids), or the `params` argument for `embedding_lookup`.
+* `time_major`: Python bool. Whether the tensors in `inputs` are time
+ major. If `False` (default), they are assumed to be batch major.
+* `seed`: The sampling seed.
+* `scheduling_seed`: The schedule decision rule sampling seed.
+
+
+#### Raises:
+
+
+* `ValueError`: if `sampling_probability` is not a scalar or vector.
+
+
+
+## Properties
+
+batch_size
+
+
+
+
+sample_ids_dtype
+
+
+
+
+sample_ids_shape
+
+
+
+
+
+
+## Methods
+
+initialize
+
+``` python
+initialize(
+ inputs,
+ sequence_length=None,
+ embedding=None
+)
+```
+
+
+
+
+
+
+``` python
+next_inputs(
+ time,
+ outputs,
+ state,
+ sample_ids
+)
+```
+
+
+
+
+sample
+
+``` python
+sample(
+ time,
+ outputs,
+ state
+)
+```
+
+
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/ScheduledOutputTrainingSampler.md b/docs/api_docs/python/tfa/seq2seq/ScheduledOutputTrainingSampler.md
new file mode 100644
index 0000000000..fdeaa44a86
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/ScheduledOutputTrainingSampler.md
@@ -0,0 +1,132 @@
+
+
+
+
+
+
+
+
+
+
+
+
+# tfa.seq2seq.ScheduledOutputTrainingSampler
+
+## Class `ScheduledOutputTrainingSampler`
+
+A training sampler that adds scheduled sampling directly to outputs.
+
+Inherits From: [`TrainingSampler`](../../tfa/seq2seq/TrainingSampler.md)
+
+### Aliases:
+
+* Class `tfa.seq2seq.ScheduledOutputTrainingSampler`
+* Class `tfa.seq2seq.sampler.ScheduledOutputTrainingSampler`
+
+
+
+Defined in [`seq2seq/sampler.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/sampler.py).
+
+
+
+Returns False for sample_ids where no sampling took place; True
+elsewhere.
+
+__init__
+
+``` python
+__init__(
+ sampling_probability,
+ time_major=False,
+ seed=None,
+ next_inputs_fn=None
+)
+```
+
+Initializer.
+
+
+#### Args:
+
+
+* `sampling_probability`: A `float32` scalar tensor: the probability of
+ sampling from the outputs instead of reading directly from the
+ inputs.
+* `time_major`: Python bool. Whether the tensors in `inputs` are time
+ major. If `False` (default), they are assumed to be batch major.
+* `seed`: The sampling seed.
+* `next_inputs_fn`: (Optional) callable to apply to the RNN outputs to
+ create the next input when sampling. If `None` (default), the RNN
+ outputs will be used as the next inputs.
+
+
+#### Raises:
+
+
+* `ValueError`: if `sampling_probability` is not a scalar or vector.
+
+
+
+## Properties
+
+batch_size
+
+
+
+
+sample_ids_dtype
+
+
+
+
+sample_ids_shape
+
+
+
+
+
+
+## Methods
+
+initialize
+
+``` python
+initialize(
+ inputs,
+ sequence_length=None,
+ auxiliary_inputs=None
+)
+```
+
+
+
+
+
+
+``` python
+next_inputs(
+ time,
+ outputs,
+ state,
+ sample_ids
+)
+```
+
+
+
+
+sample
+
+``` python
+sample(
+ time,
+ outputs,
+ state
+)
+```
+
+
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/SequenceLoss.md b/docs/api_docs/python/tfa/seq2seq/SequenceLoss.md
new file mode 100644
index 0000000000..fe49b5b542
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/SequenceLoss.md
@@ -0,0 +1,96 @@
+
+
+
+
+
+
+
+
+
+# tfa.seq2seq.SequenceLoss
+
+## Class `SequenceLoss`
+
+Weighted cross-entropy loss for a sequence of logits.
+
+
+
+### Aliases:
+
+* Class `tfa.seq2seq.SequenceLoss`
+* Class `tfa.seq2seq.loss.SequenceLoss`
+
+
+
+Defined in [`seq2seq/loss.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/loss.py).
+
+
+
+
+__init__
+
+``` python
+__init__(
+ average_across_timesteps=False,
+ average_across_batch=False,
+ sum_over_timesteps=True,
+ sum_over_batch=True,
+ softmax_loss_function=None,
+ name=None
+)
+```
+
+
+
+
+
+
+## Methods
+
+__call__
+
+``` python
+__call__(
+ y_true,
+ y_pred,
+ sample_weight=None
+)
+```
+
+Override the parent __call__ to have a customized reduce
+behavior.
+
+from_config
+
+``` python
+from_config(
+ cls,
+ config
+)
+```
+
+Instantiates a `Loss` from its config (output of `get_config()`).
+
+
+#### Args:
+
+
+* `config`: Output of `get_config()`.
+
+
+#### Returns:
+
+A `Loss` instance.
+
+
+get_config
+
+``` python
+get_config()
+```
+
+
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/TrainingSampler.md b/docs/api_docs/python/tfa/seq2seq/TrainingSampler.md
new file mode 100644
index 0000000000..df8e1300be
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/TrainingSampler.md
@@ -0,0 +1,135 @@
+
+
+
+
+
+
+
+
+
+
+
+
+# tfa.seq2seq.TrainingSampler
+
+## Class `TrainingSampler`
+
+A Sampler for use during training.
+
+Inherits From: [`Sampler`](../../tfa/seq2seq/Sampler.md)
+
+### Aliases:
+
+* Class `tfa.seq2seq.TrainingSampler`
+* Class `tfa.seq2seq.sampler.TrainingSampler`
+
+
+
+Defined in [`seq2seq/sampler.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/sampler.py).
+
+
+
+Only reads inputs.
+
+Returned sample_ids are the argmax of the RNN output logits.
+
+__init__
+
+``` python
+__init__(time_major=False)
+```
+
+Initializer.
+
+
+#### Args:
+
+
+* `time_major`: Python bool. Whether the tensors in `inputs` are time
+ major. If `False` (default), they are assumed to be batch major.
+
+
+#### Raises:
+
+
+* `ValueError`: if `sequence_length` is not a 1D tensor.
+
+
+
+## Properties
+
+batch_size
+
+
+
+
+sample_ids_dtype
+
+
+
+
+sample_ids_shape
+
+
+
+
+
+
+## Methods
+
+initialize
+
+``` python
+initialize(
+ inputs,
+ sequence_length=None
+)
+```
+
+Initialize the TrainSampler.
+
+
+#### Args:
+
+
+* `inputs`: A (structure of) input tensors.
+* `sequence_length`: An int32 vector tensor.
+
+
+#### Returns:
+
+(finished, next_inputs), a tuple of two items. The first item is a
+ boolean vector to indicate whether the item in the batch has
+ finished. The second item is the first slide of input data based on
+ the timestep dimension (usually the second dim of the input).
+
+
+
+
+``` python
+next_inputs(
+ time,
+ outputs,
+ state,
+ sample_ids
+)
+```
+
+
+
+
+sample
+
+``` python
+sample(
+ time,
+ outputs,
+ state
+)
+```
+
+
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/attention_wrapper.md b/docs/api_docs/python/tfa/seq2seq/attention_wrapper.md
new file mode 100644
index 0000000000..23a464ed62
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/attention_wrapper.md
@@ -0,0 +1,40 @@
+
+
+
+
+
+# Module: tfa.seq2seq.attention_wrapper
+
+A powerful dynamic attention wrapper object.
+
+
+
+Defined in [`seq2seq/attention_wrapper.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/attention_wrapper.py).
+
+
+
+
+## Classes
+
+[`class AttentionMechanism`](../../tfa/seq2seq/AttentionMechanism.md)
+
+[`class AttentionWrapper`](../../tfa/seq2seq/AttentionWrapper.md): Wraps another `RNNCell` with attention.
+
+[`class AttentionWrapperState`](../../tfa/seq2seq/AttentionWrapperState.md): `namedtuple` storing the state of a `AttentionWrapper`.
+
+[`class BahdanauAttention`](../../tfa/seq2seq/BahdanauAttention.md): Implements Bahdanau-style (additive) attention.
+
+[`class BahdanauMonotonicAttention`](../../tfa/seq2seq/BahdanauMonotonicAttention.md): Monotonic attention mechanism with Bahadanau-style energy function.
+
+[`class LuongAttention`](../../tfa/seq2seq/LuongAttention.md): Implements Luong-style (multiplicative) attention scoring.
+
+[`class LuongMonotonicAttention`](../../tfa/seq2seq/LuongMonotonicAttention.md): Monotonic attention mechanism with Luong-style energy function.
+
+## Functions
+
+[`hardmax(...)`](../../tfa/seq2seq/hardmax.md): Returns batched one-hot vectors.
+
+[`monotonic_attention(...)`](../../tfa/seq2seq/monotonic_attention.md): Compute monotonic attention distribution from choosing probabilities.
+
+[`safe_cumprod(...)`](../../tfa/seq2seq/safe_cumprod.md): Computes cumprod of x in logspace using cumsum to avoid underflow.
+
diff --git a/docs/api_docs/python/tfa/seq2seq/basic_decoder.md b/docs/api_docs/python/tfa/seq2seq/basic_decoder.md
new file mode 100644
index 0000000000..8cc74c9532
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/basic_decoder.md
@@ -0,0 +1,22 @@
+
+
+
+
+
+# Module: tfa.seq2seq.basic_decoder
+
+A class of Decoders that may sample to generate the next input.
+
+
+
+Defined in [`seq2seq/basic_decoder.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/basic_decoder.py).
+
+
+
+
+## Classes
+
+[`class BasicDecoder`](../../tfa/seq2seq/BasicDecoder.md): Basic sampling decoder.
+
+[`class BasicDecoderOutput`](../../tfa/seq2seq/BasicDecoderOutput.md)
+
diff --git a/docs/api_docs/python/tfa/seq2seq/beam_search_decoder.md b/docs/api_docs/python/tfa/seq2seq/beam_search_decoder.md
new file mode 100644
index 0000000000..0a359ea60e
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/beam_search_decoder.md
@@ -0,0 +1,38 @@
+
+
+
+
+
+# Module: tfa.seq2seq.beam_search_decoder
+
+A decoder that performs beam search.
+
+
+
+Defined in [`seq2seq/beam_search_decoder.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/beam_search_decoder.py).
+
+
+
+
+## Classes
+
+[`class BeamSearchDecoder`](../../tfa/seq2seq/BeamSearchDecoder.md): BeamSearch sampling decoder.
+
+[`class BeamSearchDecoderMixin`](../../tfa/seq2seq/beam_search_decoder/BeamSearchDecoderMixin.md): BeamSearchDecoderMixin contains the common methods for
+
+[`class BeamSearchDecoderOutput`](../../tfa/seq2seq/BeamSearchDecoderOutput.md)
+
+[`class BeamSearchDecoderState`](../../tfa/seq2seq/BeamSearchDecoderState.md)
+
+[`class FinalBeamSearchDecoderOutput`](../../tfa/seq2seq/FinalBeamSearchDecoderOutput.md): Final outputs returned by the beam search after all decoding is
+
+## Functions
+
+[`attention_probs_from_attn_state(...)`](../../tfa/seq2seq/beam_search_decoder/attention_probs_from_attn_state.md): Calculates the average attention probabilities.
+
+[`gather_tree_from_array(...)`](../../tfa/seq2seq/gather_tree_from_array.md): Calculates the full beams for `TensorArray`s.
+
+[`get_attention_probs(...)`](../../tfa/seq2seq/beam_search_decoder/get_attention_probs.md): Get attention probabilities from the cell state.
+
+[`tile_batch(...)`](../../tfa/seq2seq/tile_batch.md): Tile the batch dimension of a (possibly nested structure of) tensor(s)
+
diff --git a/docs/api_docs/python/tfa/seq2seq/beam_search_decoder/BeamSearchDecoderMixin.md b/docs/api_docs/python/tfa/seq2seq/beam_search_decoder/BeamSearchDecoderMixin.md
new file mode 100644
index 0000000000..52efa9249f
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/beam_search_decoder/BeamSearchDecoderMixin.md
@@ -0,0 +1,168 @@
+
+
+
+
+
+
+
+
+
+
+
+# tfa.seq2seq.beam_search_decoder.BeamSearchDecoderMixin
+
+## Class `BeamSearchDecoderMixin`
+
+BeamSearchDecoderMixin contains the common methods for
+
+
+
+
+
+Defined in [`seq2seq/beam_search_decoder.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/beam_search_decoder.py).
+
+
+BeamSearchDecoder.
+
+It is expected to be used a base class for concrete
+BeamSearchDecoder. Since this is a mixin class, it is expected to be
+used together with other class as base.
+
+__init__
+
+``` python
+__init__(
+ cell,
+ beam_width,
+ output_layer=None,
+ length_penalty_weight=0.0,
+ coverage_penalty_weight=0.0,
+ reorder_tensor_arrays=True,
+ **kwargs
+)
+```
+
+Initialize the BeamSearchDecoderMixin.
+
+
+#### Args:
+
+
+* `cell`: An `RNNCell` instance.
+* `beam_width`: Python integer, the number of beams.
+* `output_layer`: (Optional) An instance of `tf.keras.layers.Layer`,
+ i.e., `tf.keras.layers.Dense`. Optional layer to apply to the RNN
+ output prior to storing the result or sampling.
+* `length_penalty_weight`: Float weight to penalize length. Disabled with
+ 0.0.
+* `coverage_penalty_weight`: Float weight to penalize the coverage of
+ source sentence. Disabled with 0.0.
+* `reorder_tensor_arrays`: If `True`, `TensorArray`s' elements within the
+ cell state will be reordered according to the beam search path. If
+ the `TensorArray` can be reordered, the stacked form will be
+ returned. Otherwise, the `TensorArray` will be returned as is. Set
+ this flag to `False` if the cell state contains `TensorArray`s that
+ are not amenable to reordering.
+* `**kwargs`: Dict, other keyword arguments for parent class.
+
+
+#### Raises:
+
+
+* `TypeError`: if `cell` is not an instance of `RNNCell`,
+ or `output_layer` is not an instance of `tf.keras.layers.Layer`.
+
+
+
+## Properties
+
+batch_size
+
+
+
+
+output_size
+
+
+
+
+tracks_own_finished
+
+The BeamSearchDecoder shuffles its beams and their finished state.
+
+For this reason, it conflicts with the `dynamic_decode` function's
+tracking of finished states. Setting this property to true avoids
+early stopping of decoding due to mismanagement of the finished state
+in `dynamic_decode`.
+
+#### Returns:
+
+`True`.
+
+
+
+
+## Methods
+
+finalize
+
+``` python
+finalize(
+ outputs,
+ final_state,
+ sequence_lengths
+)
+```
+
+Finalize and return the predicted_ids.
+
+
+#### Args:
+
+
+* `outputs`: An instance of BeamSearchDecoderOutput.
+* `final_state`: An instance of BeamSearchDecoderState. Passed through to
+ the output.
+* `sequence_lengths`: An `int64` tensor shaped
+ `[batch_size, beam_width]`. The sequence lengths determined for
+ each beam during decode. **NOTE** These are ignored; the updated
+ sequence lengths are stored in `final_state.lengths`.
+
+
+#### Returns:
+
+
+* `outputs`: An instance of `FinalBeamSearchDecoderOutput` where the
+ predicted_ids are the result of calling _gather_tree.
+* `final_state`: The same input instance of `BeamSearchDecoderState`.
+
+step
+
+``` python
+step(
+ time,
+ inputs,
+ state,
+ name=None
+)
+```
+
+Perform a decoding step.
+
+
+#### Args:
+
+
+* `time`: scalar `int32` tensor.
+* `inputs`: A (structure of) input tensors.
+* `state`: A (structure of) state tensors and TensorArrays.
+* `name`: Name scope for any created operations.
+
+
+#### Returns:
+
+`(outputs, next_state, next_inputs, finished)`.
+
+
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/beam_search_decoder/attention_probs_from_attn_state.md b/docs/api_docs/python/tfa/seq2seq/beam_search_decoder/attention_probs_from_attn_state.md
new file mode 100644
index 0000000000..c5e8495d04
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/beam_search_decoder/attention_probs_from_attn_state.md
@@ -0,0 +1,31 @@
+
+
+
+
+
+# tfa.seq2seq.beam_search_decoder.attention_probs_from_attn_state
+
+Calculates the average attention probabilities.
+
+``` python
+tfa.seq2seq.beam_search_decoder.attention_probs_from_attn_state(attention_state)
+```
+
+
+
+Defined in [`seq2seq/beam_search_decoder.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/beam_search_decoder.py).
+
+
+
+
+#### Args:
+
+
+* `attention_state`: An instance of `AttentionWrapperState`.
+
+
+#### Returns:
+
+The attention probabilities in the given AttentionWrapperState.
+If there're multiple attention mechanisms, return the average value from
+all attention mechanisms.
diff --git a/docs/api_docs/python/tfa/seq2seq/beam_search_decoder/get_attention_probs.md b/docs/api_docs/python/tfa/seq2seq/beam_search_decoder/get_attention_probs.md
new file mode 100644
index 0000000000..b21fd9b665
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/beam_search_decoder/get_attention_probs.md
@@ -0,0 +1,44 @@
+
+
+
+
+
+# tfa.seq2seq.beam_search_decoder.get_attention_probs
+
+Get attention probabilities from the cell state.
+
+``` python
+tfa.seq2seq.beam_search_decoder.get_attention_probs(
+ next_cell_state,
+ coverage_penalty_weight
+)
+```
+
+
+
+Defined in [`seq2seq/beam_search_decoder.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/beam_search_decoder.py).
+
+
+
+
+#### Args:
+
+
+* `next_cell_state`: The next state from the cell, e.g. an instance of
+ AttentionWrapperState if the cell is attentional.
+* `coverage_penalty_weight`: Float weight to penalize the coverage of source
+ sentence. Disabled with 0.0.
+
+
+#### Returns:
+
+The attention probabilities with shape
+ `[batch_size, beam_width, max_time]` if coverage penalty is enabled.
+ Otherwise, returns None.
+
+
+
+#### Raises:
+
+
+* `ValueError`: If no cell is attentional but coverage penalty is enabled.
\ No newline at end of file
diff --git a/docs/api_docs/python/tfa/seq2seq/decoder.md b/docs/api_docs/python/tfa/seq2seq/decoder.md
new file mode 100644
index 0000000000..7537ec70f1
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/decoder.md
@@ -0,0 +1,26 @@
+
+
+
+
+
+# Module: tfa.seq2seq.decoder
+
+Seq2seq layer operations for use in neural networks.
+
+
+
+Defined in [`seq2seq/decoder.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/decoder.py).
+
+
+
+
+## Classes
+
+[`class BaseDecoder`](../../tfa/seq2seq/BaseDecoder.md): An RNN Decoder that is based on a Keras layer.
+
+[`class Decoder`](../../tfa/seq2seq/Decoder.md): An RNN Decoder abstract interface object.
+
+## Functions
+
+[`dynamic_decode(...)`](../../tfa/seq2seq/dynamic_decode.md): Perform dynamic decoding with `decoder`.
+
diff --git a/docs/api_docs/python/tfa/seq2seq/dynamic_decode.md b/docs/api_docs/python/tfa/seq2seq/dynamic_decode.md
new file mode 100644
index 0000000000..005c496a9c
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/dynamic_decode.md
@@ -0,0 +1,70 @@
+
+
+
+
+
+# tfa.seq2seq.dynamic_decode
+
+Perform dynamic decoding with `decoder`.
+
+### Aliases:
+
+* `tfa.seq2seq.decoder.dynamic_decode`
+* `tfa.seq2seq.dynamic_decode`
+
+``` python
+tfa.seq2seq.dynamic_decode(
+ decoder,
+ output_time_major=False,
+ impute_finished=False,
+ maximum_iterations=None,
+ parallel_iterations=32,
+ swap_memory=False,
+ scope=None,
+ **kwargs
+)
+```
+
+
+
+Defined in [`seq2seq/decoder.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/decoder.py).
+
+
+
+Calls initialize() once and step() repeatedly on the Decoder object.
+
+#### Args:
+
+
+* `decoder`: A `Decoder` instance.
+* `output_time_major`: Python boolean. Default: `False` (batch major). If
+ `True`, outputs are returned as time major tensors (this mode is
+ faster). Otherwise, outputs are returned as batch major tensors (this
+ adds extra time to the computation).
+* `impute_finished`: Python boolean. If `True`, then states for batch
+ entries which are marked as finished get copied through and the
+ corresponding outputs get zeroed out. This causes some slowdown at
+ each time step, but ensures that the final state and outputs have
+ the correct values and that backprop ignores time steps that were
+ marked as finished.
+* `maximum_iterations`: `int32` scalar, maximum allowed number of decoding
+ steps. Default is `None` (decode until the decoder is fully done).
+* `parallel_iterations`: Argument passed to `tf.while_loop`.
+* `swap_memory`: Argument passed to `tf.while_loop`.
+* `scope`: Optional variable scope to use.
+* `**kwargs`: dict, other keyword arguments for dynamic_decode. It might
+ contain arguments for `BaseDecoder` to initialize, which takes all
+ tensor inputs during call().
+
+
+#### Returns:
+
+`(final_outputs, final_state, final_sequence_lengths)`.
+
+
+
+#### Raises:
+
+
+* `TypeError`: if `decoder` is not an instance of `Decoder`.
+* `ValueError`: if `maximum_iterations` is provided but is not a scalar.
\ No newline at end of file
diff --git a/docs/api_docs/python/tfa/seq2seq/gather_tree_from_array.md b/docs/api_docs/python/tfa/seq2seq/gather_tree_from_array.md
new file mode 100644
index 0000000000..8c884c4258
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/gather_tree_from_array.md
@@ -0,0 +1,44 @@
+
+
+
+
+
+# tfa.seq2seq.gather_tree_from_array
+
+Calculates the full beams for `TensorArray`s.
+
+### Aliases:
+
+* `tfa.seq2seq.beam_search_decoder.gather_tree_from_array`
+* `tfa.seq2seq.gather_tree_from_array`
+
+``` python
+tfa.seq2seq.gather_tree_from_array(
+ t,
+ parent_ids,
+ sequence_length
+)
+```
+
+
+
+Defined in [`seq2seq/beam_search_decoder.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/beam_search_decoder.py).
+
+
+
+
+#### Args:
+
+
+* `t`: A stacked `TensorArray` of size `max_time` that contains `Tensor`s of
+ shape `[batch_size, beam_width, s]` or `[batch_size * beam_width, s]`
+ where `s` is the depth shape.
+* `parent_ids`: The parent ids of shape `[max_time, batch_size, beam_width]`.
+* `sequence_length`: The sequence length of shape `[batch_size, beam_width]`.
+
+
+#### Returns:
+
+A `Tensor` which is a stacked `TensorArray` of the same size and type as
+`t` and where beams are sorted in each `Tensor` according to
+`parent_ids`.
diff --git a/docs/api_docs/python/tfa/seq2seq/hardmax.md b/docs/api_docs/python/tfa/seq2seq/hardmax.md
new file mode 100644
index 0000000000..ca6aed2e15
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/hardmax.md
@@ -0,0 +1,38 @@
+
+
+
+
+
+# tfa.seq2seq.hardmax
+
+Returns batched one-hot vectors.
+
+### Aliases:
+
+* `tfa.seq2seq.attention_wrapper.hardmax`
+* `tfa.seq2seq.hardmax`
+
+``` python
+tfa.seq2seq.hardmax(
+ logits,
+ name=None
+)
+```
+
+
+
+Defined in [`seq2seq/attention_wrapper.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/attention_wrapper.py).
+
+
+
+The depth index containing the `1` is that of the maximum logit value.
+
+#### Args:
+
+
+* `logits`: A batch tensor of logit values.
+* `name`: Name to use when creating ops.
+
+#### Returns:
+
+A batched one-hot tensor.
diff --git a/docs/api_docs/python/tfa/seq2seq/loss.md b/docs/api_docs/python/tfa/seq2seq/loss.md
new file mode 100644
index 0000000000..1408f71d38
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/loss.md
@@ -0,0 +1,24 @@
+
+
+
+
+
+# Module: tfa.seq2seq.loss
+
+Seq2seq loss operations for use in sequence models.
+
+
+
+Defined in [`seq2seq/loss.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/loss.py).
+
+
+
+
+## Classes
+
+[`class SequenceLoss`](../../tfa/seq2seq/SequenceLoss.md): Weighted cross-entropy loss for a sequence of logits.
+
+## Functions
+
+[`sequence_loss(...)`](../../tfa/seq2seq/sequence_loss.md): Weighted cross-entropy loss for a sequence of logits.
+
diff --git a/docs/api_docs/python/tfa/seq2seq/monotonic_attention.md b/docs/api_docs/python/tfa/seq2seq/monotonic_attention.md
new file mode 100644
index 0000000000..2392214a12
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/monotonic_attention.md
@@ -0,0 +1,73 @@
+
+
+
+
+
+# tfa.seq2seq.monotonic_attention
+
+Compute monotonic attention distribution from choosing probabilities.
+
+### Aliases:
+
+* `tfa.seq2seq.attention_wrapper.monotonic_attention`
+* `tfa.seq2seq.monotonic_attention`
+
+``` python
+tfa.seq2seq.monotonic_attention(
+ p_choose_i,
+ previous_attention,
+ mode
+)
+```
+
+
+
+Defined in [`seq2seq/attention_wrapper.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/attention_wrapper.py).
+
+
+
+Monotonic attention implies that the input sequence is processed in an
+explicitly left-to-right manner when generating the output sequence. In
+addition, once an input sequence element is attended to at a given output
+timestep, elements occurring before it cannot be attended to at subsequent
+output timesteps. This function generates attention distributions
+according to these assumptions. For more information, see `Online and
+Linear-Time Attention by Enforcing Monotonic Alignments`.
+
+#### Args:
+
+
+* `p_choose_i`: Probability of choosing input sequence/memory element i.
+ Should be of shape (batch_size, input_sequence_length), and should all
+ be in the range [0, 1].
+* `previous_attention`: The attention distribution from the previous output
+ timestep. Should be of shape (batch_size, input_sequence_length). For
+ the first output timestep, preevious_attention[n] should be
+ [1, 0, 0, ..., 0] for all n in [0, ... batch_size - 1].
+* `mode`: How to compute the attention distribution. Must be one of
+ 'recursive', 'parallel', or 'hard'.
+ * 'recursive' uses tf.scan to recursively compute the distribution.
+ This is slowest but is exact, general, and does not suffer from
+ numerical instabilities.
+ * 'parallel' uses parallelized cumulative-sum and cumulative-product
+ operations to compute a closed-form solution to the recurrence
+ relation defining the attention distribution. This makes it more
+ efficient than 'recursive', but it requires numerical checks which
+ make the distribution non-exact. This can be a problem in
+ particular when input_sequence_length is long and/or p_choose_i has
+ entries very close to 0 or 1.
+ * 'hard' requires that the probabilities in p_choose_i are all either
+ 0 or 1, and subsequently uses a more efficient and exact solution.
+
+
+#### Returns:
+
+A tensor of shape (batch_size, input_sequence_length) representing the
+attention distributions for each sequence in the batch.
+
+
+
+#### Raises:
+
+
+* `ValueError`: mode is not one of 'recursive', 'parallel', 'hard'.
\ No newline at end of file
diff --git a/docs/api_docs/python/tfa/seq2seq/safe_cumprod.md b/docs/api_docs/python/tfa/seq2seq/safe_cumprod.md
new file mode 100644
index 0000000000..ceee331bb7
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/safe_cumprod.md
@@ -0,0 +1,44 @@
+
+
+
+
+
+# tfa.seq2seq.safe_cumprod
+
+Computes cumprod of x in logspace using cumsum to avoid underflow.
+
+### Aliases:
+
+* `tfa.seq2seq.attention_wrapper.safe_cumprod`
+* `tfa.seq2seq.safe_cumprod`
+
+``` python
+tfa.seq2seq.safe_cumprod(
+ x,
+ *args,
+ **kwargs
+)
+```
+
+
+
+Defined in [`seq2seq/attention_wrapper.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/attention_wrapper.py).
+
+
+
+The cumprod function and its gradient can result in numerical instabilities
+when its argument has very small and/or zero values. As long as the
+argument is all positive, we can instead compute the cumulative product as
+exp(cumsum(log(x))). This function can be called identically to
+tf.cumprod.
+
+#### Args:
+
+
+* `x`: Tensor to take the cumulative product of.
+* `*args`: Passed on to cumsum; these are identical to those in cumprod.
+* `**kwargs`: Passed on to cumsum; these are identical to those in cumprod.
+
+#### Returns:
+
+Cumulative product of x.
diff --git a/docs/api_docs/python/tfa/seq2seq/sampler.md b/docs/api_docs/python/tfa/seq2seq/sampler.md
new file mode 100644
index 0000000000..c544e9d4ec
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/sampler.md
@@ -0,0 +1,40 @@
+
+
+
+
+
+# Module: tfa.seq2seq.sampler
+
+A library of sampler for use with SamplingDecoders.
+
+
+
+Defined in [`seq2seq/sampler.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/sampler.py).
+
+
+
+
+## Classes
+
+[`class CustomSampler`](../../tfa/seq2seq/CustomSampler.md): Base abstract class that allows the user to customize sampling.
+
+[`class GreedyEmbeddingSampler`](../../tfa/seq2seq/GreedyEmbeddingSampler.md): A sampler for use during inference.
+
+[`class InferenceSampler`](../../tfa/seq2seq/InferenceSampler.md): A helper to use during inference with a custom sampling function.
+
+[`class SampleEmbeddingSampler`](../../tfa/seq2seq/SampleEmbeddingSampler.md): A sampler for use during inference.
+
+[`class Sampler`](../../tfa/seq2seq/Sampler.md): Interface for implementing sampling in seq2seq decoders.
+
+[`class ScheduledEmbeddingTrainingSampler`](../../tfa/seq2seq/ScheduledEmbeddingTrainingSampler.md): A training sampler that adds scheduled sampling.
+
+[`class ScheduledOutputTrainingSampler`](../../tfa/seq2seq/ScheduledOutputTrainingSampler.md): A training sampler that adds scheduled sampling directly to outputs.
+
+[`class TrainingSampler`](../../tfa/seq2seq/TrainingSampler.md): A Sampler for use during training.
+
+## Functions
+
+[`bernoulli_sample(...)`](../../tfa/seq2seq/sampler/bernoulli_sample.md): Samples from Bernoulli distribution.
+
+[`categorical_sample(...)`](../../tfa/seq2seq/sampler/categorical_sample.md): Samples from categorical distribution.
+
diff --git a/docs/api_docs/python/tfa/seq2seq/sampler/bernoulli_sample.md b/docs/api_docs/python/tfa/seq2seq/sampler/bernoulli_sample.md
new file mode 100644
index 0000000000..bca0a10698
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/sampler/bernoulli_sample.md
@@ -0,0 +1,24 @@
+
+
+
+
+
+# tfa.seq2seq.sampler.bernoulli_sample
+
+Samples from Bernoulli distribution.
+
+``` python
+tfa.seq2seq.sampler.bernoulli_sample(
+ probs=None,
+ logits=None,
+ dtype=tf.int32,
+ sample_shape=(),
+ seed=None
+)
+```
+
+
+
+Defined in [`seq2seq/sampler.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/sampler.py).
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/sampler/categorical_sample.md b/docs/api_docs/python/tfa/seq2seq/sampler/categorical_sample.md
new file mode 100644
index 0000000000..d13ef6b34e
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/sampler/categorical_sample.md
@@ -0,0 +1,23 @@
+
+
+
+
+
+# tfa.seq2seq.sampler.categorical_sample
+
+Samples from categorical distribution.
+
+``` python
+tfa.seq2seq.sampler.categorical_sample(
+ logits,
+ dtype=tf.int32,
+ sample_shape=(),
+ seed=None
+)
+```
+
+
+
+Defined in [`seq2seq/sampler.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/sampler.py).
+
+
diff --git a/docs/api_docs/python/tfa/seq2seq/sequence_loss.md b/docs/api_docs/python/tfa/seq2seq/sequence_loss.md
new file mode 100644
index 0000000000..5fe50b724a
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/sequence_loss.md
@@ -0,0 +1,95 @@
+
+
+
+
+
+# tfa.seq2seq.sequence_loss
+
+Weighted cross-entropy loss for a sequence of logits.
+
+### Aliases:
+
+* `tfa.seq2seq.loss.sequence_loss`
+* `tfa.seq2seq.sequence_loss`
+
+``` python
+tfa.seq2seq.sequence_loss(
+ logits,
+ targets,
+ weights,
+ average_across_timesteps=True,
+ average_across_batch=True,
+ sum_over_timesteps=False,
+ sum_over_batch=False,
+ softmax_loss_function=None,
+ name=None
+)
+```
+
+
+
+Defined in [`seq2seq/loss.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/loss.py).
+
+
+
+Depending on the values of `average_across_timesteps` /
+`sum_over_timesteps` and `average_across_batch` / `sum_over_batch`, the
+return Tensor will have rank 0, 1, or 2 as these arguments reduce the
+cross-entropy at each target, which has shape
+`[batch_size, sequence_length]`, over their respective dimensions. For
+example, if `average_across_timesteps` is `True` and `average_across_batch`
+is `False`, then the return Tensor will have shape `[batch_size]`.
+
+Note that `average_across_timesteps` and `sum_over_timesteps` cannot be
+True at same time. Same for `average_across_batch` and `sum_over_batch`.
+
+The recommended loss reduction in tf 2.0 has been changed to sum_over,
+instead of weighted average. User are recommend to use `sum_over_timesteps`
+and `sum_over_batch` for reduction.
+
+#### Args:
+
+
+* `logits`: A Tensor of shape
+ `[batch_size, sequence_length, num_decoder_symbols]` and dtype float.
+ The logits correspond to the prediction across all classes at each
+ timestep.
+* `targets`: A Tensor of shape `[batch_size, sequence_length]` and dtype
+ int. The target represents the true class at each timestep.
+* `weights`: A Tensor of shape `[batch_size, sequence_length]` and dtype
+ float. `weights` constitutes the weighting of each prediction in the
+ sequence. When using `weights` as masking, set all valid timesteps to 1
+ and all padded timesteps to 0, e.g. a mask returned by
+ `tf.sequence_mask`.
+* `average_across_timesteps`: If set, sum the cost across the sequence
+ dimension and divide the cost by the total label weight across
+ timesteps.
+* `average_across_batch`: If set, sum the cost across the batch dimension and
+ divide the returned cost by the batch size.
+* `sum_over_timesteps`: If set, sum the cost across the sequence dimension
+ and divide the size of the sequence. Note that any element with 0
+ weights will be excluded from size calculation.
+* `sum_over_batch`: if set, sum the cost across the batch dimension and
+ divide the total cost by the batch size. Not that any element with 0
+ weights will be excluded from size calculation.
+* `softmax_loss_function`: Function (labels, logits) -> loss-batch
+ to be used instead of the standard softmax (the default if this is
+ None). **Note that to avoid confusion, it is required for the function
+ to accept named arguments.**
+* `name`: Optional name for this operation, defaults to "sequence_loss".
+
+
+#### Returns:
+
+A float Tensor of rank 0, 1, or 2 depending on the
+`average_across_timesteps` and `average_across_batch` arguments. By
+default, it has rank 0 (scalar) and is the weighted average cross-entropy
+(log-perplexity) per symbol.
+
+
+
+#### Raises:
+
+
+* `ValueError`: logits does not have 3 dimensions or targets does not have 2
+ dimensions or weights does not have 2 dimensions.
\ No newline at end of file
diff --git a/docs/api_docs/python/tfa/seq2seq/tile_batch.md b/docs/api_docs/python/tfa/seq2seq/tile_batch.md
new file mode 100644
index 0000000000..16a849718a
--- /dev/null
+++ b/docs/api_docs/python/tfa/seq2seq/tile_batch.md
@@ -0,0 +1,56 @@
+
+
+
+
+
+# tfa.seq2seq.tile_batch
+
+Tile the batch dimension of a (possibly nested structure of) tensor(s)
+
+### Aliases:
+
+* `tfa.seq2seq.beam_search_decoder.tile_batch`
+* `tfa.seq2seq.tile_batch`
+
+``` python
+tfa.seq2seq.tile_batch(
+ t,
+ multiplier,
+ name=None
+)
+```
+
+
+
+Defined in [`seq2seq/beam_search_decoder.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/seq2seq/beam_search_decoder.py).
+
+
+t.
+
+For each tensor t in a (possibly nested structure) of tensors,
+this function takes a tensor t shaped `[batch_size, s0, s1, ...]` composed
+of minibatch entries `t[0], ..., t[batch_size - 1]` and tiles it to have a
+shape `[batch_size * multiplier, s0, s1, ...]` composed of minibatch
+entries `t[0], t[0], ..., t[1], t[1], ...` where each minibatch entry is
+repeated `multiplier` times.
+
+#### Args:
+
+
+* `t`: `Tensor` shaped `[batch_size, ...]`.
+* `multiplier`: Python int.
+* `name`: Name scope for any created operations.
+
+
+#### Returns:
+
+A (possibly nested structure of) `Tensor` shaped
+`[batch_size * multiplier, ...]`.
+
+
+
+#### Raises:
+
+
+* `ValueError`: if tensor(s) `t` do not have a statically known rank or
+the rank is < 1.
\ No newline at end of file
diff --git a/docs/api_docs/python/tfa/text.md b/docs/api_docs/python/tfa/text.md
new file mode 100644
index 0000000000..8b5f6fbbda
--- /dev/null
+++ b/docs/api_docs/python/tfa/text.md
@@ -0,0 +1,26 @@
+
+
+
+
+
+# Module: tfa.text
+
+Text-processing ops.
+
+
+
+Defined in [`text/__init__.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/text/__init__.py).
+
+
+
+
+## Modules
+
+[`skip_gram_ops`](../tfa/text/skip_gram_ops.md) module: Skip-gram sampling ops from https://arxiv.org/abs/1301.3781.
+
+## Functions
+
+[`skip_gram_sample(...)`](../tfa/text/skip_gram_sample.md): Generates skip-gram token and label paired Tensors from the input
+
+[`skip_gram_sample_with_text_vocab(...)`](../tfa/text/skip_gram_sample_with_text_vocab.md): Skip-gram sampling with a text vocabulary file.
+
diff --git a/docs/api_docs/python/tfa/text/skip_gram_ops.md b/docs/api_docs/python/tfa/text/skip_gram_ops.md
new file mode 100644
index 0000000000..a8d6cf949f
--- /dev/null
+++ b/docs/api_docs/python/tfa/text/skip_gram_ops.md
@@ -0,0 +1,22 @@
+
+
+
+
+
+# Module: tfa.text.skip_gram_ops
+
+Skip-gram sampling ops from https://arxiv.org/abs/1301.3781.
+
+
+
+Defined in [`text/skip_gram_ops.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/text/skip_gram_ops.py).
+
+
+
+
+## Functions
+
+[`skip_gram_sample(...)`](../../tfa/text/skip_gram_sample.md): Generates skip-gram token and label paired Tensors from the input
+
+[`skip_gram_sample_with_text_vocab(...)`](../../tfa/text/skip_gram_sample_with_text_vocab.md): Skip-gram sampling with a text vocabulary file.
+
diff --git a/docs/api_docs/python/tfa/text/skip_gram_sample.md b/docs/api_docs/python/tfa/text/skip_gram_sample.md
new file mode 100644
index 0000000000..763cd12b20
--- /dev/null
+++ b/docs/api_docs/python/tfa/text/skip_gram_sample.md
@@ -0,0 +1,143 @@
+
+
+
+
+
+# tfa.text.skip_gram_sample
+
+Generates skip-gram token and label paired Tensors from the input
+
+### Aliases:
+
+* `tfa.text.skip_gram_ops.skip_gram_sample`
+* `tfa.text.skip_gram_sample`
+
+``` python
+tfa.text.skip_gram_sample(
+ input_tensor,
+ min_skips=1,
+ max_skips=5,
+ start=0,
+ limit=-1,
+ emit_self_as_target=False,
+ vocab_freq_table=None,
+ vocab_min_count=None,
+ vocab_subsampling=None,
+ corpus_size=None,
+ batch_size=None,
+ batch_capacity=None,
+ seed=None,
+ name=None
+)
+```
+
+
+
+Defined in [`text/skip_gram_ops.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/text/skip_gram_ops.py).
+
+
+tensor.
+
+Generates skip-gram `("token", "label")` pairs using each element in the
+rank-1 `input_tensor` as a token. The window size used for each token will
+be randomly selected from the range specified by `[min_skips, max_skips]`,
+inclusive. See https://arxiv.org/abs/1301.3781 for more details about
+skip-gram.
+
+For example, given `input_tensor = ["the", "quick", "brown", "fox",
+"jumps"]`, `min_skips = 1`, `max_skips = 2`, `emit_self_as_target = False`,
+the output `(tokens, labels)` pairs for the token "quick" will be randomly
+selected from either `(tokens=["quick", "quick"], labels=["the", "brown"])`
+for 1 skip, or `(tokens=["quick", "quick", "quick"],
+labels=["the", "brown", "fox"])` for 2 skips.
+
+If `emit_self_as_target = True`, each token will also be emitted as a label
+for itself. From the previous example, the output will be either
+`(tokens=["quick", "quick", "quick"], labels=["the", "quick", "brown"])`
+for 1 skip, or `(tokens=["quick", "quick", "quick", "quick"],
+labels=["the", "quick", "brown", "fox"])` for 2 skips.
+
+The same process is repeated for each element of `input_tensor` and
+concatenated together into the two output rank-1 `Tensors` (one for all the
+tokens, another for all the labels).
+
+If `vocab_freq_table` is specified, tokens in `input_tensor` that are not
+present in the vocabulary are discarded. Tokens whose frequency counts are
+below `vocab_min_count` are also discarded. Tokens whose frequency
+proportions in the corpus exceed `vocab_subsampling` may be randomly
+down-sampled. See Eq. 5 in http://arxiv.org/abs/1310.4546 for more details
+about subsampling.
+
+Due to the random window sizes used for each token, the lengths of the
+outputs are non-deterministic, unless `batch_size` is specified to batch
+the outputs to always return `Tensors` of length `batch_size`.
+
+#### Args:
+
+
+* `input_tensor`: A rank-1 `Tensor` from which to generate skip-gram
+ candidates.
+* `min_skips`: `int` or scalar `Tensor` specifying the minimum window size to
+ randomly use for each token. Must be >= 0 and <= `max_skips`. If
+ `min_skips` and `max_skips` are both 0, the only label outputted will
+ be the token itself when `emit_self_as_target = True` -
+ or no output otherwise.
+* `max_skips`: `int` or scalar `Tensor` specifying the maximum window size to
+ randomly use for each token. Must be >= 0.
+* `start`: `int` or scalar `Tensor` specifying the position in
+ `input_tensor` from which to start generating skip-gram candidates.
+* `limit`: `int` or scalar `Tensor` specifying the maximum number of
+ elements in `input_tensor` to use in generating skip-gram candidates.
+ -1 means to use the rest of the `Tensor` after `start`.
+* `emit_self_as_target`: `bool` or scalar `Tensor` specifying whether to emit
+ each token as a label for itself.
+* `vocab_freq_table`: (Optional) A lookup table (subclass of
+ `lookup.InitializableLookupTableBase`) that maps tokens to their raw
+ frequency counts. If specified, any token in `input_tensor` that is not
+ found in `vocab_freq_table` will be filtered out before generating
+ skip-gram candidates. While this will typically map to integer raw
+ frequency counts, it could also map to float frequency proportions.
+ `vocab_min_count` and `corpus_size` should be in the same units
+ as this.
+* `vocab_min_count`: (Optional) `int`, `float`, or scalar `Tensor` specifying
+ minimum frequency threshold (from `vocab_freq_table`) for a token to be
+ kept in `input_tensor`. If this is specified, `vocab_freq_table` must
+ also be specified - and they should both be in the same units.
+* `vocab_subsampling`: (Optional) `float` specifying frequency proportion
+ threshold for tokens from `input_tensor`. Tokens that occur more
+ frequently (based on the ratio of the token's `vocab_freq_table` value
+ to the `corpus_size`) will be randomly down-sampled. Reasonable
+ starting values may be around 1e-3 or 1e-5. If this is specified, both
+ `vocab_freq_table` and `corpus_size` must also be specified. See Eq. 5
+ in http://arxiv.org/abs/1310.4546 for more details.
+* `corpus_size`: (Optional) `int`, `float`, or scalar `Tensor` specifying the
+ total number of tokens in the corpus (e.g., sum of all the frequency
+ counts of `vocab_freq_table`). Used with `vocab_subsampling` for
+ down-sampling frequently occurring tokens. If this is specified,
+ `vocab_freq_table` and `vocab_subsampling` must also be specified.
+* `batch_size`: (Optional) `int` specifying batch size of returned `Tensors`.
+* `batch_capacity`: (Optional) `int` specifying batch capacity for the queue
+ used for batching returned `Tensors`. Only has an effect if
+ `batch_size` > 0. Defaults to 100 * `batch_size` if not specified.
+* `seed`: (Optional) `int` used to create a random seed for window size and
+ subsampling. See `set_random_seed` docs for behavior.
+* `name`: (Optional) A `string` name or a name scope for the operations.
+
+
+#### Returns:
+
+A `tuple` containing (token, label) `Tensors`. Each output `Tensor` is of
+rank-1 and has the same type as `input_tensor`. The `Tensors` will be of
+length `batch_size`; if `batch_size` is not specified, they will be of
+random length, though they will be in sync with each other as long as
+they are evaluated together.
+
+
+
+#### Raises:
+
+
+* `ValueError`: If `vocab_freq_table` is not provided, but `vocab_min_count`,
+ `vocab_subsampling`, or `corpus_size` is specified.
+ If `vocab_subsampling` and `corpus_size` are not both present or
+ both absent.
\ No newline at end of file
diff --git a/docs/api_docs/python/tfa/text/skip_gram_sample_with_text_vocab.md b/docs/api_docs/python/tfa/text/skip_gram_sample_with_text_vocab.md
new file mode 100644
index 0000000000..e5543eee6f
--- /dev/null
+++ b/docs/api_docs/python/tfa/text/skip_gram_sample_with_text_vocab.md
@@ -0,0 +1,138 @@
+
+
+
+
+
+# tfa.text.skip_gram_sample_with_text_vocab
+
+Skip-gram sampling with a text vocabulary file.
+
+### Aliases:
+
+* `tfa.text.skip_gram_ops.skip_gram_sample_with_text_vocab`
+* `tfa.text.skip_gram_sample_with_text_vocab`
+
+``` python
+tfa.text.skip_gram_sample_with_text_vocab(
+ input_tensor,
+ vocab_freq_file,
+ vocab_token_index=0,
+ vocab_token_dtype=tf.dtypes.string,
+ vocab_freq_index=1,
+ vocab_freq_dtype=tf.dtypes.float64,
+ vocab_delimiter=',',
+ vocab_min_count=0,
+ vocab_subsampling=None,
+ corpus_size=None,
+ min_skips=1,
+ max_skips=5,
+ start=0,
+ limit=-1,
+ emit_self_as_target=False,
+ batch_size=None,
+ batch_capacity=None,
+ seed=None,
+ name=None
+)
+```
+
+
+
+Defined in [`text/skip_gram_ops.py`](https://github.com/tensorflow/addons/tree/0.4-release/tensorflow_addons/text/skip_gram_ops.py).
+
+
+
+Wrapper around `skip_gram_sample()` for use with a text vocabulary file.
+The vocabulary file is expected to be a plain-text file, with lines of
+`vocab_delimiter`-separated columns. The `vocab_token_index` column should
+contain the vocabulary term, while the `vocab_freq_index` column should
+contain the number of times that term occurs in the corpus. For example,
+with a text vocabulary file of:
+
+ ```
+ bonjour,fr,42
+ hello,en,777
+ hola,es,99
+ ```
+
+You should set `vocab_delimiter=","`, `vocab_token_index=0`, and
+`vocab_freq_index=2`.
+
+See `skip_gram_sample()` documentation for more details about the skip-gram
+sampling process.
+
+#### Args:
+
+
+* `input_tensor`: A rank-1 `Tensor` from which to generate skip-gram candidates.
+* `vocab_freq_file`: `string` specifying full file path to the text vocab file.
+* `vocab_token_index`: `int` specifying which column in the text vocab file
+ contains the tokens.
+* `vocab_token_dtype`: `DType` specifying the format of the tokens in the text vocab file.
+* `vocab_freq_index`: `int` specifying which column in the text vocab file
+ contains the frequency counts of the tokens.
+* `vocab_freq_dtype`: `DType` specifying the format of the frequency counts
+ in the text vocab file.
+* `vocab_delimiter`: `string` specifying the delimiter used in the text vocab
+ file.
+* `vocab_min_count`: `int`, `float`, or scalar `Tensor` specifying
+ minimum frequency threshold (from `vocab_freq_file`) for a token to be
+ kept in `input_tensor`. This should correspond with `vocab_freq_dtype`.
+* `vocab_subsampling`: (Optional) `float` specifying frequency proportion
+ threshold for tokens from `input_tensor`. Tokens that occur more
+ frequently will be randomly down-sampled. Reasonable starting values
+ may be around 1e-3 or 1e-5. See Eq. 5 in http://arxiv.org/abs/1310.4546
+ for more details.
+* `corpus_size`: (Optional) `int`, `float`, or scalar `Tensor` specifying the
+ total number of tokens in the corpus (e.g., sum of all the frequency
+ counts of `vocab_freq_file`). Used with `vocab_subsampling` for
+ down-sampling frequently occurring tokens. If this is specified,
+ `vocab_freq_file` and `vocab_subsampling` must also be specified.
+ If `corpus_size` is needed but not supplied, then it will be calculated
+ from `vocab_freq_file`. You might want to supply your own value if you
+ have already eliminated infrequent tokens from your vocabulary files
+ (where frequency < vocab_min_count) to save memory in the internal
+ token lookup table. Otherwise, the unused tokens' variables will waste
+ memory. The user-supplied `corpus_size` value must be greater than or
+ equal to the sum of all the frequency counts of `vocab_freq_file`.
+* `min_skips`: `int` or scalar `Tensor` specifying the minimum window size to
+ randomly use for each token. Must be >= 0 and <= `max_skips`. If
+ `min_skips` and `max_skips` are both 0, the only label outputted will
+ be the token itself.
+* `max_skips`: `int` or scalar `Tensor` specifying the maximum window size to
+ randomly use for each token. Must be >= 0.
+* `start`: `int` or scalar `Tensor` specifying the position in `input_tensor`
+ from which to start generating skip-gram candidates.
+* `limit`: `int` or scalar `Tensor` specifying the maximum number of elements
+ in `input_tensor` to use in generating skip-gram candidates. -1 means
+ to use the rest of the `Tensor` after `start`.
+* `emit_self_as_target`: `bool` or scalar `Tensor` specifying whether to emit
+ each token as a label for itself.
+* `batch_size`: (Optional) `int` specifying batch size of returned `Tensors`.
+* `batch_capacity`: (Optional) `int` specifying batch capacity for the queue
+ used for batching returned `Tensors`. Only has an effect if
+ `batch_size` > 0. Defaults to 100 * `batch_size` if not specified.
+* `seed`: (Optional) `int` used to create a random seed for window size and
+ subsampling. See
+ [`set_random_seed`](../../g3doc/python/constant_op.md#set_random_seed)
+ for behavior.
+* `name`: (Optional) A `string` name or a name scope for the operations.
+
+
+#### Returns:
+
+A `tuple` containing (token, label) `Tensors`. Each output `Tensor` is of
+rank-1 and has the same type as `input_tensor`. The `Tensors` will be of
+length `batch_size`; if `batch_size` is not specified, they will be of
+random length, though they will be in sync with each other as long as
+they are evaluated together.
+
+
+
+#### Raises:
+
+
+* `ValueError`: If `vocab_token_index` or `vocab_freq_index` is less than 0
+ or exceeds the number of columns in `vocab_freq_file`.
+ If `vocab_token_index` and `vocab_freq_index` are both set to the same
+ column. If any token in `vocab_freq_file` has a negative frequency.
\ No newline at end of file
diff --git a/requirements.txt b/requirements.txt
index 8b3c4f69bc..0349c9f771 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1 +1 @@
-tf-nightly-2.0-preview
\ No newline at end of file
+tensorflow==2.0.0b1
\ No newline at end of file
diff --git a/tensorflow_addons/version.py b/tensorflow_addons/version.py
index 008d683d2e..1c210762c6 100644
--- a/tensorflow_addons/version.py
+++ b/tensorflow_addons/version.py
@@ -27,7 +27,7 @@
# stable release (indicated by `_VERSION_SUFFIX = ''`). Outside the context of a
# release branch, the current version is by default assumed to be a
# 'development' version, labeled 'dev'.
-_VERSION_SUFFIX = 'dev'
+_VERSION_SUFFIX = ''
# Example, '0.1.0-dev'
__version__ = '.'.join([
diff --git a/tools/ci_build/builds/release_linux.sh b/tools/ci_build/builds/release_linux.sh
index e6403153e6..873ae704d4 100755
--- a/tools/ci_build/builds/release_linux.sh
+++ b/tools/ci_build/builds/release_linux.sh
@@ -38,7 +38,7 @@ for version in ${PYTHON_VERSIONS}; do
build_pip_pkg
# Package Whl
- bazel-bin/build_pip_pkg artifacts --nightly
+ bazel-bin/build_pip_pkg artifacts
# Uncomment and use this command for release branches
#bazel-bin/build_pip_pkg artifacts
diff --git a/tools/ci_build/builds/release_macos.sh b/tools/ci_build/builds/release_macos.sh
index 4baeba68db..9e8c97ad04 100644
--- a/tools/ci_build/builds/release_macos.sh
+++ b/tools/ci_build/builds/release_macos.sh
@@ -41,7 +41,7 @@ for version in ${PYTHON_VERSIONS}; do
build_pip_pkg
# Package Whl
- bazel-bin/build_pip_pkg artifacts --nightly
+ bazel-bin/build_pip_pkg artifacts
# Uncomment and use this command for release branches
#bazel-bin/build_pip_pkg artifacts