You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I came up with the following sanity check to ensure that the implementation and word embeddings etc are good.
I created a dataset of 100,000 lines, that has the following 6 lines repeated over and over again:
hi . $$$
hi , joey . $$$
hello ? $$$
who are you ? $$$
what are you doing ? $$$
nothing much . you ? $$$
I then ran your code with the following parameters and model:
TOKEN_REPRESENTATION_SIZE = 32 # word2vec parameter
HIDDEN_LAYER_DIMENSION = 4096 # number of nodes in each LSTM layer
seq2seq = Seq2seq(
batch_input_shape=(SAMPLES_BATCH_SIZE, INPUT_SEQUENCE_LENGTH, TOKEN_REPRESENTATION_SIZE),
hidden_dim = HIDDEN_LAYER_DIMENSION,
output_length=ANSWER_MAX_TOKEN_LENGTH,
output_dim=token_dict_size,
depth=2,
dropout=0.25,
peek=True
)
opt=adagrad(clipvalue=50)
model.compile(loss='sparse_categorical_crossentropy', optimizer=opt, metrics=["accuracy"])
After 10 data passes, my result look like this:
INFO:lib.nn_model.train:[hi. ] -> [$$$ doing who who $$$ $$$ $$$]
INFO:lib.nn_model.train:[hello ?] -> [$$$ doing who who $$$ $$$ $$$]
INFO:lib.nn_model.train:[who are you ?] -> [$$$ doing who who $$$ $$$ $$$]
INFO:lib.nn_model.train:[what are you doing ?] -> [$$$ doing who who $$$ $$$ $$$]
So basically, the sanity check fails. The model can't even learn the answer to these 6 lines, even though they were repeated so many times. Does anyone know why this is happening? What could be the problem?
The text was updated successfully, but these errors were encountered:
I came up with the following sanity check to ensure that the implementation and word embeddings etc are good.
I created a dataset of 100,000 lines, that has the following 6 lines repeated over and over again:
I then ran your code with the following parameters and model:
After 10 data passes, my result look like this:
So basically, the sanity check fails. The model can't even learn the answer to these 6 lines, even though they were repeated so many times. Does anyone know why this is happening? What could be the problem?
The text was updated successfully, but these errors were encountered: