Skip to content
This repository has been archived by the owner on Jul 27, 2023. It is now read-only.

Something wrong with the loss #10

Closed
katherinelyx opened this issue Jul 3, 2019 · 6 comments
Closed

Something wrong with the loss #10

katherinelyx opened this issue Jul 3, 2019 · 6 comments

Comments

@katherinelyx
Copy link

Hello, I have run your codes of 'chatbot' on a conversation dataset. But the loss seems unnormally low. The results on dailydialog dataset from paper 'DailyDialog: AManuallyLabelledMulti-turnDialogueDataset' show that perplexity is more than 30 and loss is more than 3. But the perplexity obtained by your codes is lower 3 which is absolutely wrong. Could you provide some advice? Thank you.
'
@huseinzol05
Copy link
Member

I cut the dataset to use 500 sentences only. you might want to use the whole dataset for better experiments.

@katherinelyx
Copy link
Author

Actually, I used the whole dataset but the perplexity shows too low (lower means better).

@huseinzol05
Copy link
Member

Oh, I believe thats good? how about test some BLEU or word position accuracy on dev / test split? I implemented word position accuracy in the notebooks, can you paste some logs here?

@katherinelyx
Copy link
Author

Epoch 0 Batch 50/225 - Loss: 3.2396 - perplextity: 25.5224
Epoch 0 Batch 50/225 - Test Loss: 3.1811 - Test perplextity: 24.0732
Epoch 0 Batch 100/225 - Loss: 3.1357 - perplextity: 23.0053
Epoch 0 Batch 100/225 - Test Loss: 2.7565 - Test perplextity: 15.7445
Epoch 0 Batch 150/225 - Loss: 2.6570 - perplextity: 14.2531
Epoch 0 Batch 150/225 - Test Loss: 2.6090 - Test perplextity: 13.5860
Epoch 1 Batch 50/225 - Loss: 2.5518 - perplextity: 12.8302
Epoch 1 Batch 50/225 - Test Loss: 2.5145 - Test perplextity: 12.3599
Epoch 1 Batch 100/225 - Loss: 2.7359 - perplextity: 15.4241
Epoch 1 Batch 100/225 - Test Loss: 2.4095 - Test perplextity: 11.1285
Epoch 1 Batch 150/225 - Loss: 2.3386 - perplextity: 10.3663
Epoch 1 Batch 150/225 - Test Loss: 2.3367 - Test perplextity: 10.3469
Epoch 2 Batch 50/225 - Loss: 2.2795 - perplextity: 9.7714
Epoch 2 Batch 50/225 - Test Loss: 2.2351 - Test perplextity: 9.3470
Epoch 2 Batch 100/225 - Loss: 2.4809 - perplextity: 11.9526
Epoch 2 Batch 100/225 - Test Loss: 2.1800 - Test perplextity: 8.8461
Epoch 2 Batch 150/225 - Loss: 2.1127 - perplextity: 8.2705
Epoch 2 Batch 150/225 - Test Loss: 2.1392 - Test perplextity: 8.4927
Epoch 3 Batch 50/225 - Loss: 2.0933 - perplextity: 8.1114
Epoch 3 Batch 50/225 - Test Loss: 2.0807 - Test perplextity: 8.0101
Epoch 3 Batch 100/225 - Loss: 2.3165 - perplextity: 10.1398
Epoch 3 Batch 100/225 - Test Loss: 2.0538 - Test perplextity: 7.7971
Epoch 3 Batch 150/225 - Loss: 1.9877 - perplextity: 7.2991
Epoch 3 Batch 150/225 - Test Loss: 2.0336 - Test perplextity: 7.6416
Epoch 4 Batch 50/225 - Loss: 1.9937 - perplextity: 7.3424
Epoch 4 Batch 50/225 - Test Loss: 2.0070 - Test perplextity: 7.4412
Epoch 4 Batch 100/225 - Loss: 2.2090 - perplextity: 9.1067
Epoch 4 Batch 100/225 - Test Loss: 1.9855 - Test perplextity: 7.2825
Epoch 4 Batch 150/225 - Loss: 1.9120 - perplextity: 6.7669
Epoch 4 Batch 150/225 - Test Loss: 1.9765 - Test perplextity: 7.2174
Epoch 5 Batch 50/225 - Loss: 1.9217 - perplextity: 6.8324
Epoch 5 Batch 50/225 - Test Loss: 1.9546 - Test perplextity: 7.0608
Epoch 5 Batch 100/225 - Loss: 2.1339 - perplextity: 8.4479
Epoch 5 Batch 100/225 - Test Loss: 1.9412 - Test perplextity: 6.9672
Epoch 5 Batch 150/225 - Loss: 1.8519 - perplextity: 6.3719
Epoch 5 Batch 150/225 - Test Loss: 1.9381 - Test perplextity: 6.9458
Epoch 6 Batch 50/225 - Loss: 1.8610 - perplextity: 6.4300
Epoch 6 Batch 50/225 - Test Loss: 1.9214 - Test perplextity: 6.8304
Epoch 6 Batch 100/225 - Loss: 2.0644 - perplextity: 7.8804
Epoch 6 Batch 100/225 - Test Loss: 1.9103 - Test perplextity: 6.7549
Epoch 6 Batch 150/225 - Loss: 1.7991 - perplextity: 6.0440
Epoch 6 Batch 150/225 - Test Loss: 1.9102 - Test perplextity: 6.7543
Epoch 7 Batch 50/225 - Loss: 1.8036 - perplextity: 6.0717
Epoch 7 Batch 50/225 - Test Loss: 1.8949 - Test perplextity: 6.6519
Epoch 7 Batch 100/225 - Loss: 2.0062 - perplextity: 7.4351
Epoch 7 Batch 100/225 - Test Loss: 1.8878 - Test perplextity: 6.6046
Epoch 7 Batch 150/225 - Loss: 1.7517 - perplextity: 5.7647
Epoch 7 Batch 150/225 - Test Loss: 1.8894 - Test perplextity: 6.6155
Epoch 8 Batch 50/225 - Loss: 1.7498 - perplextity: 5.7532
Epoch 8 Batch 50/225 - Test Loss: 1.8740 - Test perplextity: 6.5143
Epoch 8 Batch 100/225 - Loss: 1.9525 - perplextity: 7.0464
Epoch 8 Batch 100/225 - Test Loss: 1.8725 - Test perplextity: 6.5045
Epoch 8 Batch 150/225 - Loss: 1.7382 - perplextity: 5.6873
Epoch 8 Batch 150/225 - Test Loss: 1.8799 - Test perplextity: 6.5528
Epoch 9 Batch 50/225 - Loss: 1.7048 - perplextity: 5.5004
Epoch 9 Batch 50/225 - Test Loss: 1.8631 - Test perplextity: 6.4438
Epoch 9 Batch 100/225 - Loss: 1.9123 - perplextity: 6.7690
Epoch 9 Batch 100/225 - Test Loss: 1.8637 - Test perplextity: 6.4479
Epoch 9 Batch 150/225 - Loss: 1.6760 - perplextity: 5.3441
Epoch 9 Batch 150/225 - Test Loss: 1.8560 - Test perplextity: 6.3980
Epoch 10 Batch 50/225 - Loss: 1.6487 - perplextity: 5.2003
Epoch 10 Batch 50/225 - Test Loss: 1.8510 - Test perplextity: 6.3661
Epoch 10 Batch 100/225 - Loss: 1.8625 - perplextity: 6.4397
Epoch 10 Batch 100/225 - Test Loss: 1.8534 - Test perplextity: 6.3814
Epoch 10 Batch 150/225 - Loss: 1.6387 - perplextity: 5.1486
Epoch 10 Batch 150/225 - Test Loss: 1.8477 - Test perplextity: 6.3454
Epoch 11 Batch 50/225 - Loss: 1.6005 - perplextity: 4.9557
Epoch 11 Batch 50/225 - Test Loss: 1.8444 - Test perplextity: 6.3245
Epoch 11 Batch 100/225 - Loss: 1.8206 - perplextity: 6.1756
Epoch 11 Batch 100/225 - Test Loss: 1.8453 - Test perplextity: 6.3298
Epoch 11 Batch 150/225 - Loss: 1.6047 - perplextity: 4.9765
Epoch 11 Batch 150/225 - Test Loss: 1.8449 - Test perplextity: 6.3275
Epoch 12 Batch 50/225 - Loss: 1.5533 - perplextity: 4.7272
Epoch 12 Batch 50/225 - Test Loss: 1.8412 - Test perplextity: 6.3038
Epoch 12 Batch 100/225 - Loss: 1.7814 - perplextity: 5.9379
Epoch 12 Batch 100/225 - Test Loss: 1.8413 - Test perplextity: 6.3046
Epoch 12 Batch 150/225 - Loss: 1.5708 - perplextity: 4.8104
Epoch 12 Batch 150/225 - Test Loss: 1.8434 - Test perplextity: 6.3181
Epoch 13 Batch 50/225 - Loss: 1.5063 - perplextity: 4.5098
Epoch 13 Batch 50/225 - Test Loss: 1.8406 - Test perplextity: 6.3002
Epoch 13 Batch 100/225 - Loss: 1.7443 - perplextity: 5.7221
Epoch 13 Batch 100/225 - Test Loss: 1.8420 - Test perplextity: 6.3090
Epoch 13 Batch 150/225 - Loss: 1.5340 - perplextity: 4.6365
Epoch 13 Batch 150/225 - Test Loss: 1.8408 - Test perplextity: 6.3016
Epoch 14 Batch 50/225 - Loss: 1.4640 - perplextity: 4.3233
Epoch 14 Batch 50/225 - Test Loss: 1.8399 - Test perplextity: 6.2960
Epoch 14 Batch 100/225 - Loss: 1.7052 - perplextity: 5.5023
Epoch 14 Batch 100/225 - Test Loss: 1.8406 - Test perplextity: 6.3000
Epoch 14 Batch 150/225 - Loss: 1.5005 - perplextity: 4.4838
Epoch 14 Batch 150/225 - Test Loss: 1.8395 - Test perplextity: 6.2931
Epoch 15 Batch 50/225 - Loss: 1.4342 - perplextity: 4.1965
Epoch 15 Batch 50/225 - Test Loss: 1.8393 - Test perplextity: 6.2924
Epoch 15 Batch 100/225 - Loss: 1.6726 - perplextity: 5.3259
Epoch 15 Batch 100/225 - Test Loss: 1.8384 - Test perplextity: 6.2866
Epoch 15 Batch 150/225 - Loss: 1.4723 - perplextity: 4.3592
Epoch 15 Batch 150/225 - Test Loss: 1.8417 - Test perplextity: 6.3072
Epoch 16 Batch 50/225 - Loss: 1.3969 - perplextity: 4.0426
Epoch 16 Batch 50/225 - Test Loss: 1.8398 - Test perplextity: 6.2955
Epoch 16 Batch 100/225 - Loss: 1.6442 - perplextity: 5.1768
Epoch 16 Batch 100/225 - Test Loss: 1.8409 - Test perplextity: 6.3021
Epoch 16 Batch 150/225 - Loss: 1.4423 - perplextity: 4.2305
Epoch 16 Batch 150/225 - Test Loss: 1.8445 - Test perplextity: 6.3247
Epoch 17 Batch 50/225 - Loss: 1.3791 - perplextity: 3.9713
Epoch 17 Batch 50/225 - Test Loss: 1.8484 - Test perplextity: 6.3497
Epoch 17 Batch 100/225 - Loss: 1.6208 - perplextity: 5.0572
Epoch 17 Batch 100/225 - Test Loss: 1.8447 - Test perplextity: 6.3261
Epoch 17 Batch 150/225 - Loss: 1.4237 - perplextity: 4.1524
Epoch 17 Batch 150/225 - Test Loss: 1.8416 - Test perplextity: 6.3069
Epoch 18 Batch 50/225 - Loss: 1.3451 - perplextity: 3.8385
Epoch 18 Batch 50/225 - Test Loss: 1.8453 - Test perplextity: 6.3297
Epoch 18 Batch 100/225 - Loss: 1.5864 - perplextity: 4.8861
Epoch 18 Batch 100/225 - Test Loss: 1.8483 - Test perplextity: 6.3489
Epoch 18 Batch 150/225 - Loss: 1.3997 - perplextity: 4.0540
Epoch 18 Batch 150/225 - Test Loss: 1.8428 - Test perplextity: 6.3144
Epoch 19 Batch 50/225 - Loss: 1.3163 - perplextity: 3.7295
Epoch 19 Batch 50/225 - Test Loss: 1.8469 - Test perplextity: 6.3401
Epoch 19 Batch 100/225 - Loss: 1.5570 - perplextity: 4.7445
Epoch 19 Batch 100/225 - Test Loss: 1.8510 - Test perplextity: 6.3664
Epoch 19 Batch 150/225 - Loss: 1.3776 - perplextity: 3.9655
Epoch 19 Batch 150/225 - Test Loss: 1.8449 - Test perplextity: 6.3277
Model Trained and Saved

@katherinelyx
Copy link
Author

According to the results showed above, the test perplexity achieves 6.3277 after 20 epochs, which is too much lower than usual results from other published papers. I will try some other metrics.

@huseinzol05
Copy link
Member

skeptical much? haha. Maybe you read old research papers? maybe they use old seq2seq APIs?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@huseinzol05 @katherinelyx