Skip to content
This repository has been archived by the owner on Nov 3, 2023. It is now read-only.

ray_ddp the progressive bar is broken #176

Open
JiahaoYao opened this issue Jul 9, 2022 · 1 comment
Open

ray_ddp the progressive bar is broken #176

JiahaoYao opened this issue Jul 9, 2022 · 1 comment

Comments

@JiahaoYao
Copy link
Contributor

The progressive bar in the ray_ddp shows the results:

Epoch 0:   3%|▎         | 32/937 [00:00<00:06, 131.01it/s, loss=1.83, v_num=2]
Epoch 0:   3%|▎         | 32/937 [00:00<00:06, 131.70it/s, loss=1.83, v_num=2]
Epoch 0:   5%|▌         | 50/937 [00:00<00:06, 143.88it/s, loss=1.67, v_num=2]
Epoch 0:   5%|▌         | 50/937 [00:00<00:06, 144.36it/s, loss=1.67, v_num=2]
Epoch 0:   7%|▋         | 67/937 [00:00<00:05, 149.46it/s, loss=1.72, v_num=2]
Epoch 0:   7%|▋         | 68/937 [00:00<00:05, 150.15it/s, loss=1.74, v_num=2]
Epoch 0:   7%|▋         | 68/937 [00:00<00:05, 149.76it/s, loss=1.74, v_num=2]
Epoch 0:   9%|▉         | 86/937 [00:00<00:05, 154.51it/s, loss=1.6, v_num=2] 
Epoch 0:   9%|▉         | 86/937 [00:00<00:05, 154.74it/s, loss=1.6, v_num=2] 
Epoch 0:   9%|▉         | 87/937 [00:00<00:05, 154.71it/s, loss=1.62, v_num=2]
Epoch 0:   9%|▉         | 87/937 [00:00<00:05, 154.94it/s, loss=1.62, v_num=2]
Epoch 0:  11%|█         | 104/937 [00:00<00:05, 156.98it/s, loss=1.58, v_num=2]
Epoch 0:  11%|█         | 105/937 [00:00<00:05, 157.47it/s, loss=1.59, v_num=2]
Epoch 0:  11%|█         | 105/937 [00:00<00:05, 157.07it/s, loss=1.59, v_num=2]
Epoch 0:  13%|█▎        | 123/937 [00:00<00:05, 159.10it/s, loss=1.58, v_num=2]
Epoch 0:  13%|█▎        | 123/937 [00:00<00:05, 159.33it/s, loss=1.58, v_num=2]
Epoch 0:  15%|█▌        | 141/937 [00:00<00:04, 160.96it/s, loss=1.6, v_num=2] 
Epoch 0:  15%|█▌        | 141/937 [00:00<00:04, 161.13it/s, loss=1.6, v_num=2] 
Epoch 0:  17%|█▋        | 159/937 [00:00<00:04, 162.49it/s, loss=1.55, v_num=2]
Epoch 0:  17%|█▋        | 159/937 [00:00<00:04, 162.66it/s, loss=1.55, v_num=2]
Epoch 0:  17%|█▋        | 160/937 [00:00<00:04, 162.58it/s, loss=1.55, v_num=2]
Epoch 0:  17%|█▋        | 160/937 [00:00<00:04, 162.75it/s, loss=1.59, v_num=2]
Epoch 0:  17%|█▋        | 160/937 [00:00<00:04, 162.53it/s, loss=1.59, v_num=2]
Epoch 0:  17%|█▋        | 161/937 [00:00<00:04, 162.84it/s, loss=1.59, v_num=2]
Epoch 0:  17%|█▋        | 161/937 [00:00<00:04, 162.58it/s, loss=1.55, v_num=2]
Epoch 0:  17%|█▋        | 161/937 [00:00<00:04, 162.78it/s, loss=1.55, v_num=2]
Epoch 0:  19%|█▉        | 179/937 [00:01<00:04, 163.51it/s, loss=1.59, v_num=2]
Epoch 0:  19%|█▉        | 179/937 [00:01<00:04, 163.66it/s, loss=1.59, v_num=2]
Epoch 0:  21%|██        | 197/937 [00:01<00:04, 164.43it/s, loss=1.55, v_num=2]
Epoch 0:  21%|██        | 197/937 [00:01<00:04, 164.47it/s, loss=1.55, v_num=2]
Epoch 0:  23%|██▎       | 215/937 [00:01<00:04, 164.94it/s, loss=1.56, v_num=2]
Epoch 0:  23%|██▎       | 215/937 [00:01<00:04, 165.06it/s, loss=1.56, v_num=2]
Epoch 0:  23%|██▎       | 215/937 [00:01<00:04, 164.91it/s, loss=1.56, v_num=2]
Epoch 0:  25%|██▍       | 233/937 [00:01<00:04, 165.57it/s, loss=1.7, v_num=2] 
Epoch 0:  25%|██▍       | 233/937 [00:01<00:04, 165.72it/s, loss=1.7, v_num=2] 
Epoch 0:  25%|██▍       | 234/937 [00:01<00:04, 165.75it/s, loss=1.68, v_num=2]
Epoch 0:  25%|██▍       | 234/937 [00:01<00:04, 165.60it/s, loss=1.68, v_num=2]
Epoch 0:  27%|██▋       | 252/937 [00:01<00:04, 166.08it/s, loss=1.59, v_num=2]
Epoch 0:  27%|██▋       | 252/937 [00:01<00:04, 166.19it/s, loss=1.59, v_num=2]
Epoch 0:  29%|██▉       | 270/937 [00:01<00:04, 166.58it/s, loss=1.49, v_num=2]
Epoch 0:  29%|██▉       | 270/937 [00:01<00:04, 166.70it/s, loss=1.49, v_num=2]
Epoch 0:  31%|███       | 288/937 [00:01<00:03, 166.93it/s, loss=1.52, v_num=2]
Epoch 0:  31%|███       | 288/937 [00:01<00:03, 167.09it/s, loss=1.52, v_num=2]
Epoch 0:  31%|███       | 289/937 [00:01<00:03, 167.14it/s, loss=1.52, v_num=2]
Epoch 0:  31%|███       | 289/937 [00:01<00:03, 166.92it/s, loss=1.5, v_num=2] 
Epoch 0:  31%|███       | 289/937 [00:01<00:03, 167.08it/s, loss=1.5, v_num=2] 
Epoch 0:  33%|███▎      | 306/937 [00:01<00:03, 166.78it/s, loss=1.57, v_num=2]
Epoch 0:  33%|███▎      | 306/937 [00:01<00:03, 166.93it/s, loss=1.57, v_num=2]
Epoch 0:  34%|███▍      | 323/937 [00:01<00:03, 166.96it/s, loss=1.41, v_num=2]
Epoch 0:  35%|███▍      | 324/937 [00:01<00:03, 167.10it/s, loss=1.42, v_num=2]
Epoch 0:  35%|███▍      | 324/937 [00:01<00:03, 166.98it/s, loss=1.42, v_num=2]
Epoch 0:  36%|███▋      | 342/937 [00:02<00:03, 167.28it/s, loss=1.37, v_num=2]
Epoch 0:  36%|███▋      | 342/937 [00:02<00:03, 167.37it/s, loss=1.37, v_num=2]
Epoch 0:  38%|███▊      | 360/937 [00:02<00:03, 167.53it/s, loss=1.42, v_num=2]
Epoch 0:  38%|███▊      | 360/937 [00:02<00:03, 167.53it/s, loss=1.42, v_num=2]
Epoch 0:  40%|████      | 378/937 [00:02<00:03, 167.80it/s, loss=1.37, v_num=2]
Epoch 0:  40%|████      | 378/937 [00:02<00:03, 167.84it/s, loss=1.37, v_num=2]
Epoch 0:  42%|████▏     | 395/937 [00:02<00:03, 167.88it/s, loss=1.34, v_num=2]
Epoch 0:  42%|████▏     | 396/937 [00:02<00:03, 168.00it/s, loss=1.36, v_num=2]
Epoch 0:  42%|████▏     | 396/937 [00:02<00:03, 167.88it/s, loss=1.36, v_num=2]
Epoch 0:  42%|████▏     | 397/937 [00:02<00:03, 168.04it/s, loss=1.36, v_num=2]
Epoch 0:  42%|████▏     | 397/937 [00:02<00:03, 167.99it/s, loss=1.37, v_num=2]
Epoch 0:  42%|████▏     | 397/937 [00:02<00:03, 167.88it/s, loss=1.37, v_num=2]
Epoch 0:  44%|████▍     | 414/937 [00:02<00:03, 167.86it/s, loss=1.29, v_num=2]
Epoch 0:  44%|████▍     | 414/937 [00:02<00:03, 167.94it/s, loss=1.29, v_num=2]
Epoch 0:  44%|████▍     | 415/937 [00:02<00:03, 167.88it/s, loss=1.29, v_num=2]
Epoch 0:  44%|████▍     | 415/937 [00:02<00:03, 167.97it/s, loss=1.29, v_num=2]
Epoch 0:  46%|████▌     | 433/937 [00:02<00:02, 168.16it/s, loss=1.37, v_num=2]
Epoch 0:  46%|████▌     | 433/937 [00:02<00:02, 168.23it/s, loss=1.37, v_num=2]
Epoch 0:  48%|████▊     | 451/937 [00:02<00:02, 168.39it/s, loss=1.55, v_num=2]
Epoch 0:  48%|████▊     | 451/937 [00:02<00:02, 168.49it/s, loss=1.55, v_num=2]
Epoch 0:  48%|████▊     | 452/937 [00:02<00:02, 168.51it/s, loss=1.55, v_num=2]
Epoch 0:  48%|████▊     | 452/937 [00:02<00:02, 168.39it/s, loss=1.55, v_num=2]
Epoch 0:  48%|████▊     | 452/937 [00:02<00:02, 168.49it/s, loss=1.57, v_num=2]
Epoch 0:  48%|████▊     | 452/937 [00:02<00:02, 168.37it/s, loss=1.57, v_num=2]
Epoch 0:  50%|█████     | 470/937 [00:02<00:02, 168.57it/s, loss=1.87, v_num=2]
Epoch 0:  50%|█████     | 470/937 [00:02<00:02, 168.64it/s, loss=1.87, v_num=2]
Epoch 0:  50%|█████     | 471/937 [00:02<00:02, 168.66it/s, loss=1.87, v_num=2]
Epoch 0:  50%|█████     | 471/937 [00:02<00:02, 168.58it/s, loss=1.87, v_num=2]
Epoch 0:  50%|█████     | 471/937 [00:02<00:02, 168.65it/s, loss=1.87, v_num=2]
Epoch 0:  52%|█████▏    | 489/937 [00:02<00:02, 168.85it/s, loss=1.58, v_num=2]
Epoch 0:  52%|█████▏    | 489/937 [00:02<00:02, 168.90it/s, loss=1.58, v_num=2]
Epoch 0:  52%|█████▏    | 490/937 [00:02<00:02, 168.92it/s, loss=1.58, v_num=2]
Epoch 0:  52%|█████▏    | 490/937 [00:02<00:02, 168.86it/s, loss=1.56, v_num=2]
Epoch 0:  52%|█████▏    | 490/937 [00:02<00:02, 168.90it/s, loss=1.56, v_num=2]
Epoch 0:  54%|█████▍    | 508/937 [00:03<00:02, 169.07it/s, loss=1.64, v_num=2]
Epoch 0:  54%|█████▍    | 508/937 [00:03<00:02, 169.12it/s, loss=1.64, v_num=2]
Epoch 0:  56%|█████▌    | 526/937 [00:03<00:02, 169.28it/s, loss=1.61, v_num=2]
Epoch 0:  56%|█████▌    | 526/937 [00:03<00:02, 169.34it/s, loss=1.61, v_num=2]
Epoch 0:  56%|█████▌    | 527/937 [00:03<00:02, 169.29it/s, loss=1.58, v_num=2]
Epoch 0:  56%|█████▌    | 527/937 [00:03<00:02, 169.35it/s, loss=1.58, v_num=2]
Epoch 0:  58%|█████▊    | 545/937 [00:03<00:02, 169.52it/s, loss=1.51, v_num=2]
Epoch 0:  58%|█████▊    | 545/937 [00:03<00:02, 169.56it/s, loss=1.51, v_num=2]
Epoch 0:  60%|██████    | 563/937 [00:03<00:02, 169.70it/s, loss=1.56, v_num=2]
Epoch 0:  60%|██████    | 563/937 [00:03<00:02, 169.75it/s, loss=1.56, v_num=2]
Epoch 0:  60%|██████    | 564/937 [00:03<00:02, 169.71it/s, loss=1.55, v_num=2]
Epoch 0:  60%|██████    | 564/937 [00:03<00:02, 169.77it/s, loss=1.55, v_num=2]
Epoch 0:  62%|██████▏   | 582/937 [00:03<00:02, 169.91it/s, loss=1.63, v_num=2]
Epoch 0:  62%|██████▏   | 582/937 [00:03<00:02, 169.92it/s, loss=1.63, v_num=2]
Epoch 0:  64%|██████▍   | 600/937 [00:03<00:01, 170.08it/s, loss=1.64, v_num=2]
Epoch 0:  64%|██████▍   | 600/937 [00:03<00:01, 170.09it/s, loss=1.64, v_num=2]
Epoch 0:  64%|██████▍   | 601/937 [00:03<00:01, 170.09it/s, loss=1.65, v_num=2]
Epoch 0:  64%|██████▍   | 601/937 [00:03<00:01, 170.05it/s, loss=1.65, v_num=2]
Epoch 0:  66%|██████▌   | 619/937 [00:03<00:01, 170.19it/s, loss=1.59, v_num=2]
Epoch 0:  66%|██████▌   | 619/937 [00:03<00:01, 170.22it/s, loss=1.59, v_num=2]
Epoch 0:  68%|██████▊   | 637/937 [00:03<00:01, 170.34it/s, loss=1.61, v_num=2]
Epoch 0:  68%|██████▊   | 637/937 [00:03<00:01, 170.37it/s, loss=1.61, v_num=2]
Epoch 0:  68%|██████▊   | 638/937 [00:03<00:01, 170.35it/s, loss=1.61, v_num=2]
Epoch 0:  68%|██████▊   | 638/937 [00:03<00:01, 170.38it/s, loss=1.61, v_num=2]
Epoch 0:  70%|██████▉   | 655/937 [00:03<00:01, 170.40it/s, loss=1.57, v_num=2]
Epoch 0:  70%|███████   | 656/937 [00:03<00:01, 170.44it/s, loss=1.54, v_num=2]
Epoch 0:  70%|███████   | 656/937 [00:03<00:01, 170.41it/s, loss=1.54, v_num=2]
Epoch 0:  72%|███████▏  | 674/937 [00:03<00:01, 170.56it/s, loss=1.62, v_num=2]
Epoch 0:  72%|███████▏  | 674/937 [00:03<00:01, 170.60it/s, loss=1.62, v_num=2]
Epoch 0:  72%|███████▏  | 675/937 [00:03<00:01, 170.58it/s, loss=1.62, v_num=2]
Epoch 0:  72%|███████▏  | 675/937 [00:03<00:01, 170.60it/s, loss=1.62, v_num=2]
Epoch 0:  72%|███████▏  | 675/937 [00:03<00:01, 170.57it/s, loss=1.62, v_num=2]
Epoch 0:  74%|███████▍  | 692/937 [00:04<00:01, 170.53it/s, loss=1.72, v_num=2]
Epoch 0:  74%|███████▍  | 693/937 [00:04<00:01, 170.58it/s, loss=1.7, v_num=2] 
Epoch 0:  74%|███████▍  | 693/937 [00:04<00:01, 170.53it/s, loss=1.7, v_num=2] 
Epoch 0:  76%|███████▌  | 710/937 [00:04<00:01, 170.50it/s, loss=1.87, v_num=2]
Epoch 0:  76%|███████▌  | 711/937 [00:04<00:01, 170.55it/s, loss=1.86, v_num=2]
Epoch 0:  76%|███████▌  | 711/937 [00:04<00:01, 170.50it/s, loss=1.86, v_num=2]
Epoch 0:  78%|███████▊  | 729/937 [00:04<00:01, 170.55it/s, loss=1.85, v_num=2]
Epoch 0:  78%|███████▊  | 729/937 [00:04<00:01, 170.59it/s, loss=1.85, v_num=2]
Epoch 0:  80%|███████▉  | 747/937 [00:04<00:01, 170.64it/s, loss=1.84, v_num=2]
Epoch 0:  80%|███████▉  | 747/937 [00:04<00:01, 170.68it/s, loss=1.84, v_num=2]
Epoch 0:  80%|███████▉  | 748/937 [00:04<00:01, 170.68it/s, loss=1.87, v_num=2]
Epoch 0:  80%|███████▉  | 748/937 [00:04<00:01, 170.64it/s, loss=1.87, v_num=2]
^C/home/ubuntu/anaconda3/envs/plt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:726: UserWarning: Detected KeyboardInterrupt, attempting graceful shutdown...
  rank_zero_warn("Detected KeyboardInterrupt, attempting graceful shutdown...")
Epoch 0:  82%|████████▏ | 766/937 [00:04<00:01, 170.72it/s, loss=1.88, v_num=2]
Epoch 0:  82%|████████▏ | 766/937 [00:04<00:01, 170.75it/s, loss=1.88, v_num=2]
Epoch 0:  84%|████████▎ | 784/937 [00:04<00:00, 170.80it/s, loss=1.81, v_num=2]
Epoch 0:  84%|████████▎ | 784/937 [00:04<00:00, 170.84it/s, loss=1.81, v_num=2]
Epoch 0:  86%|████████▌ | 802/937 [00:04<00:00, 170.91it/s, loss=1.95, v_num=2]
Epoch 0:  86%|████████▌ | 802/937 [00:04<00:00, 170.94it/s, loss=1.95, v_num=2]
Epoch 0:  86%|████████▌ | 803/937 [00:04<00:00, 170.91it/s, loss=1.95, v_num=2]
Epoch 0:  86%|████████▌ | 803/937 [00:04<00:00, 170.93it/s, loss=1.95, v_num=2]
Epoch 0:  86%|████████▌ | 803/937 [00:04<00:00, 170.90it/s, loss=1.95, v_num=2]
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@JiahaoYao