Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Training on kraken 5.2.2 visual issue #603

Closed
johnlockejrr opened this issue May 10, 2024 · 1 comment
Closed

Training on kraken 5.2.2 visual issue #603

johnlockejrr opened this issue May 10, 2024 · 1 comment

Comments

@johnlockejrr
Copy link

johnlockejrr commented May 10, 2024

Epochs appear twice. Maybe terminal not cleared properly? Or is it intended so?

stage 0/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:20 • 0:00:00 31.22it/s  early_stopping: 0/10 -infSeed set to 42
stage 0/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:20 • 0:00:00 31.22it/s val_accuracy: 0.865 val_word_accuracy: 0.553 early_stopping: 0/10 0.86538
stage 1/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:23 • 0:00:00 25.97it/s val_accuracy: 0.865 val_word_accuracy: 0.553 early_stopping: 0/10 0.86538Seed set to 42
stage 1/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:23 • 0:00:00 25.97it/s val_accuracy: 0.908 val_word_accuracy: 0.696 early_stopping: 0/10 0.90846
stage 2/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:21 • 0:00:00 28.66it/s val_accuracy: 0.908 val_word_accuracy: 0.696 early_stopping: 0/10 0.90846Seed set to 42
stage 2/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:21 • 0:00:00 28.66it/s val_accuracy: 0.929 val_word_accuracy: 0.744 early_stopping: 0/10 0.92923
stage 3/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:28 • 0:00:00 22.00it/s val_accuracy: 0.929 val_word_accuracy: 0.744 early_stopping: 0/10 0.92923Seed set to 42
stage 3/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:28 • 0:00:00 22.00it/s val_accuracy: 0.939 val_word_accuracy: 0.766 early_stopping: 0/10 0.93923
stage 4/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:18 • 0:00:00 32.85it/s val_accuracy: 0.939 val_word_accuracy: 0.766 early_stopping: 0/10 0.93923Seed set to 42
stage 4/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:18 • 0:00:00 32.85it/s val_accuracy: 0.947 val_word_accuracy: 0.802 early_stopping: 0/10 0.94692
stage 5/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:21 • 0:00:00 28.45it/s val_accuracy: 0.947 val_word_accuracy: 0.802 early_stopping: 0/10 0.94692Seed set to 42
stage 5/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:21 • 0:00:00 28.45it/s val_accuracy: 0.962 val_word_accuracy: 0.846 early_stopping: 0/10 0.96231
stage 6/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:24 • 0:00:00 25.13it/s val_accuracy: 0.962 val_word_accuracy: 0.846 early_stopping: 0/10 0.96231Seed set to 42
stage 6/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:24 • 0:00:00 25.13it/s val_accuracy: 0.967 val_word_accuracy: 0.872 early_stopping: 0/10 0.96692
stage 7/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:24 • 0:00:00 25.74it/s val_accuracy: 0.967 val_word_accuracy: 0.872 early_stopping: 0/10 0.96692Seed set to 42
stage 7/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:24 • 0:00:00 25.74it/s val_accuracy: 0.963 val_word_accuracy: 0.864 early_stopping: 1/10 0.96692
stage 8/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:22 • 0:00:00 27.48it/s val_accuracy: 0.963 val_word_accuracy: 0.864 early_stopping: 1/10 0.96692Seed set to 42
stage 8/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:22 • 0:00:00 27.48it/s val_accuracy: 0.968 val_word_accuracy: 0.879 early_stopping: 0/10 0.96769
stage 9/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:34 • 0:00:00 19.50it/s val_accuracy: 0.968 val_word_accuracy: 0.879 early_stopping: 0/10 0.96769Seed set to 42
stage 9/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:34 • 0:00:00 19.50it/s val_accuracy: 0.974 val_word_accuracy: 0.894 early_stopping: 0/10 0.97385
stage 10/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:24 • 0:00:00 24.96it/s val_accuracy: 0.974 val_word_accuracy: 0.894 early_stopping: 0/10 0.97385Seed set to 42
stage 10/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:24 • 0:00:00 24.96it/s val_accuracy: 0.974 val_word_accuracy: 0.890 early_stopping: 1/10 0.97385
stage 11/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:18 • 0:00:00 33.61it/s val_accuracy: 0.974 val_word_accuracy: 0.890 early_stopping: 1/10 0.97385Seed set to 42
stage 11/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:18 • 0:00:00 33.61it/s val_accuracy: 0.972 val_word_accuracy: 0.886 early_stopping: 2/10 0.97385
stage 12/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:21 • 0:00:00 29.02it/s val_accuracy: 0.972 val_word_accuracy: 0.886 early_stopping: 2/10 0.97385Seed set to 42
stage 12/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:21 • 0:00:00 29.02it/s val_accuracy: 0.975 val_word_accuracy: 0.890 early_stopping: 0/10 0.97462
stage 13/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:28 • 0:00:00 21.45it/s val_accuracy: 0.975 val_word_accuracy: 0.890 early_stopping: 0/10 0.97462Seed set to 42
stage 13/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:28 • 0:00:00 21.45it/s val_accuracy: 0.975 val_word_accuracy: 0.890 early_stopping: 1/10 0.97462
stage 14/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:22 • 0:00:00 27.91it/s val_accuracy: 0.975 val_word_accuracy: 0.890 early_stopping: 1/10 0.97462Seed set to 42
stage 14/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:22 • 0:00:00 27.91it/s val_accuracy: 0.975 val_word_accuracy: 0.894 early_stopping: 2/10 0.97462
stage 15/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:21 • 0:00:00 28.36it/s val_accuracy: 0.975 val_word_accuracy: 0.894 early_stopping: 2/10 0.97462Seed set to 42
stage 15/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:21 • 0:00:00 28.36it/s val_accuracy: 0.977 val_word_accuracy: 0.894 early_stopping: 0/10 0.97692
stage 16/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:20 • 0:00:00 30.91it/s val_accuracy: 0.977 val_word_accuracy: 0.894 early_stopping: 0/10 0.97692Seed set to 42
stage 16/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:20 • 0:00:00 30.91it/s val_accuracy: 0.975 val_word_accuracy: 0.905 early_stopping: 1/10 0.97692
stage 17/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:21 • 0:00:00 28.96it/s val_accuracy: 0.975 val_word_accuracy: 0.905 early_stopping: 1/10 0.97692Seed set to 42
stage 17/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:21 • 0:00:00 28.96it/s val_accuracy: 0.978 val_word_accuracy: 0.905 early_stopping: 0/10 0.97846
stage 18/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:20 • 0:00:00 30.89it/s val_accuracy: 0.978 val_word_accuracy: 0.905 early_stopping: 0/10 0.97846Seed set to 42
stage 18/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:20 • 0:00:00 30.89it/s val_accuracy: 0.978 val_word_accuracy: 0.908 early_stopping: 1/10 0.97846
stage 19/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:19 • 0:00:00 31.26it/s val_accuracy: 0.978 val_word_accuracy: 0.908 early_stopping: 1/10 0.97846Seed set to 42
stage 19/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:19 • 0:00:00 31.26it/s val_accuracy: 0.976 val_word_accuracy: 0.897 early_stopping: 2/10 0.97846
stage 20/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:26 • 0:00:00 23.68it/s val_accuracy: 0.976 val_word_accuracy: 0.897 early_stopping: 2/10 0.97846Seed set to 42
stage 20/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:26 • 0:00:00 23.68it/s val_accuracy: 0.972 val_word_accuracy: 0.883 early_stopping: 3/10 0.97846
stage 21/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 616/616 0:00:23 • 0:00:00 26.16it/s val_accuracy: 0.972 val_word_accuracy: 0.883 early_stopping: 3/10 0.97846Seed set to 42
@mittagessen
Copy link
Owner

It's an artifact from lightning setting the log level to INFO so it prints the worker seeding onto the console. I've been trying to get rid of it for a couple of days now but haven't found a good solution yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants