pip install keras-adaptive-softmax
Generally, AdaptiveEmbedding
and AdaptiveSoftmax
should be used together. AdaptiveEmbedding
provides variable length embeddings, while AdaptiveSoftmax
calculates the similarities between the outputs and the generated embeddings.
import keras
from keras_adaptive_softmax import AdaptiveEmbedding, AdaptiveSoftmax
input_layer = keras.layers.Input(shape=(None,))
embed_layer = AdaptiveEmbedding(
input_dim=30,
output_dim=32,
cutoffs=[5, 15, 25],
div_val=2,
return_embeddings=True,
return_projections=True,
mask_zero=True,
)(input_layer)
dense_layer = keras.layers.Dense(
units=32,
activation='tanh',
)(embed_layer[0])
softmax_layer = AdaptiveSoftmax(
input_dim=32,
output_dim=30,
cutoffs=[5, 15, 25],
div_val=2,
bind_embeddings=True,
bind_projections=True,
)([dense_layer] + embed_layer[1:])
model = keras.models.Model(inputs=input_layer, outputs=softmax_layer)
model.compile('adam', 'sparse_categorical_crossentropy')
model.summary()
cutoffs
and div_val
controls the length of embeddings for each token. Suppose we have 30 distinct tokens, in the above example:
- The lengths of the embeddings of the first 5 tokens are 32
- The lengths of the embeddings of the next 10 tokens are 16
- The lengths of the embeddings of the next 10 tokens are 8
- The lengths of the embeddings of the last 5 tokens are 4