Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
TimeDistributed(Dense) with Masking not masking bias #12495
I have come across a problem with Masking the inputs to a TimeDistributed(Dense) layer.
This seems somewhat similar to #1300.
In this example, I construct a TimeDistributed(Dense) layer that sums the values at each timestep and adds a constant value.
# Construct time series data set pad_value = 0.1 t1 = [3, 1, 2, 0.1] t2 = [1, 1, 1, 0.1] t_pad = [pad_value, pad_value, pad_value, pad_value] time_series = np.asarray([[t1, t2, t_pad]]) add_constant = 10 # Build model my_input = Input(shape=(None, 4)) mask = Masking(mask_value=pad_value)(my_input) out = TimeDistributed(Dense(1, activation='linear'))(mask) model = Model(inputs=my_input, outputs=out) # Set all weights multiplying the input to 1, and set the bias to add_constant model.set_weights([np.ones_like(model.get_weights()), add_constant*np.ones_like(model.get_weights())]) model.predict(time_series)
As you can see, the last entry should be zero if the masking was applied, instead it is returning the bias.
I am also having the same problem. The documentation for Masking indicates that the timesteps are skipped, which I assumed means "has no effect", not just mapped to zero and propagated forward. For my use cases I do not want the network to have information about how much padding there was, which seems to be in effect what's happening.