Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GPU utilization patterns #11

Open
szilard opened this issue Apr 30, 2019 · 6 comments
Open

GPU utilization patterns #11

szilard opened this issue Apr 30, 2019 · 6 comments
Labels
analysis in-readme

Comments

@szilard
Copy link
Owner

szilard commented Apr 30, 2019 •
edited
Loading

xgboost:

1M:

[0] Tesla V100-SXM2-16GB | 29'C,   0 % |     0 / 16160 MB |
[0] Tesla V100-SXM2-16GB | 29'C,  62 % |  2278 / 16160 MB | root(2268M)
[0] Tesla V100-SXM2-16GB | 30'C,  80 % |  1096 / 16160 MB | root(1086M)
[0] Tesla V100-SXM2-16GB | 30'C,  80 % |  1096 / 16160 MB | root(1086M)
[0] Tesla V100-SXM2-16GB | 30'C,  79 % |  1096 / 16160 MB | root(1086M)
[0] Tesla V100-SXM2-16GB | 30'C,   0 % |  1098 / 16160 MB | root(1088M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |     0 / 16160 MB |

Screen Shot 2019-04-30 at 8 28 10 AM

03:10:48 PM  all    1.38    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00   98.37
03:10:49 PM  all   10.50    0.00    3.38    0.00    0.00    0.00    0.00    0.00    0.00   86.12
03:10:50 PM  all   10.26    0.00    3.88    0.00    0.00    0.00    0.00    0.00    0.00   85.86
03:10:51 PM  all   10.78    0.00    2.13    0.00    0.00    0.00    0.00    0.00    0.00   87.09
03:10:52 PM  all   11.36    0.00    2.50    0.00    0.00    0.00    0.00    0.00    0.00   86.14
03:10:53 PM  all   11.26    0.00    3.00    0.00    0.00    0.00    0.00    0.00    0.00   85.73
03:10:54 PM  all    7.18    0.00    2.52    0.13    0.00    0.00    0.13    0.00    0.00   90.05

Screen Shot 2019-04-30 at 8 27 56 AM

10M:

[0] Tesla V100-SXM2-16GB | 26'C,   0 % |     0 / 16160 MB |
[0] Tesla V100-SXM2-16GB | 27'C,  42 % |  2624 / 16160 MB | root(2614M)
[0] Tesla V100-SXM2-16GB | 27'C,  77 % |  2624 / 16160 MB | root(2614M)
[0] Tesla V100-SXM2-16GB | 28'C,  76 % |  2624 / 16160 MB | root(2614M)
[0] Tesla V100-SXM2-16GB | 28'C,  73 % |  2624 / 16160 MB | root(2614M)
[0] Tesla V100-SXM2-16GB | 28'C,  80 % |  2624 / 16160 MB | root(2614M)
[0] Tesla V100-SXM2-16GB | 29'C,  86 % |  1612 / 16160 MB | root(1602M)
[0] Tesla V100-SXM2-16GB | 30'C,  86 % |  1612 / 16160 MB | root(1602M)
[0] Tesla V100-SXM2-16GB | 30'C,  87 % |  1612 / 16160 MB | root(1602M)
[0] Tesla V100-SXM2-16GB | 30'C,  86 % |  1612 / 16160 MB | root(1602M)
[0] Tesla V100-SXM2-16GB | 30'C,  86 % |  1612 / 16160 MB | root(1602M)
[0] Tesla V100-SXM2-16GB | 30'C,  85 % |  1612 / 16160 MB | root(1602M)
[0] Tesla V100-SXM2-16GB | 31'C,  86 % |  1612 / 16160 MB | root(1602M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |     0 / 16160 MB |

08:41:22 PM  all    1.38    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00   98.37
08:41:23 PM  all    6.75    0.00    0.38    0.00    0.00    0.00    0.12    0.00    0.00   92.75
08:41:24 PM  all   11.50    0.00    2.00    0.00    0.00    0.00    0.00    0.00    0.00   86.50
08:41:25 PM  all   10.62    0.00    2.88    0.00    0.00    0.00    0.00    0.00    0.00   86.50
08:41:26 PM  all   11.53    0.00    2.63    0.00    0.00    0.00    0.00    0.00    0.00   85.84
08:41:27 PM  all   11.62    0.00    2.38    0.00    0.00    0.00    0.00    0.00    0.00   86.00
08:41:28 PM  all   11.79    0.00    2.26    0.00    0.00    0.00    0.00    0.00    0.00   85.95
08:41:29 PM  all   11.62    0.00    2.62    0.00    0.00    0.00    0.00    0.00    0.00   85.75
08:41:30 PM  all   11.38    0.00    2.75    0.00    0.00    0.00    0.00    0.00    0.00   85.88
08:41:31 PM  all   10.28    0.00    3.76    0.00    0.00    0.00    0.00    0.00    0.00   85.96

08:41:31 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
08:41:32 PM  all    9.75    0.00    4.50    0.00    0.00    0.00    0.00    0.00    0.00   85.75
08:41:33 PM  all    9.64    0.00    3.50    0.00    0.00    0.00    0.00    0.00    0.00   86.86
08:41:34 PM  all    9.91    0.00    3.39    0.00    0.00    0.00    0.00    0.00    0.00   86.70
08:41:35 PM  all   10.74    0.00    3.50    0.00    0.00    0.00    0.00    0.00    0.00   85.77
08:41:36 PM  all   10.14    0.00    4.01    0.00    0.00    0.00    0.00    0.00    0.00   85.86
08:41:37 PM  all   10.34    0.00    3.78    0.00    0.00    0.00    0.00    0.00    0.00   85.88
08:41:38 PM  all    1.39    0.00    1.01    0.13    0.00    0.00    0.00    0.00    0.00   97.48
@szilard
Copy link
Owner Author

szilard commented Apr 30, 2019 •
edited
Loading

h2o xgboost:

1M:

[0] Tesla V100-SXM2-16GB | 29'C,   0 % |     0 / 16160 MB |
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   454 / 16160 MB | root(444M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   454 / 16160 MB | root(444M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   510 / 16160 MB | root(500M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   510 / 16160 MB | root(500M)
[0] Tesla V100-SXM2-16GB | 29'C,  24 % |   510 / 16160 MB | root(500M)
[0] Tesla V100-SXM2-16GB | 30'C,  78 % |   510 / 16160 MB | root(500M)
[0] Tesla V100-SXM2-16GB | 31'C,  79 % |   510 / 16160 MB | root(500M)
[0] Tesla V100-SXM2-16GB | 31'C,  80 % |   510 / 16160 MB | root(500M)
[0] Tesla V100-SXM2-16GB | 30'C,   0 % |   510 / 16160 MB | root(500M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   510 / 16160 MB | root(500M)
[0] Tesla V100-SXM2-16GB | 30'C,   0 % |   510 / 16160 MB | root(500M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   510 / 16160 MB | root(500M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   444 / 16160 MB | root(434M)
[0] Tesla V100-SXM2-16GB | 29'C,   6 % |     0 / 16160 MB |

Screen Shot 2019-04-30 at 8 35 40 AM

03:22:51 PM  all    1.50    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00   98.12
03:22:52 PM  all   17.74    0.00    1.64    0.00    0.00    0.00    0.13    0.00    0.00   80.50
03:22:53 PM  all   51.19    0.00    1.38    0.00    0.00    0.00    0.00    0.00    0.00   47.42
03:22:54 PM  all   57.02    0.00    2.53    0.00    0.00    0.00    0.13    0.00    0.00   40.33
03:22:55 PM  all   85.82    0.00    2.03    0.00    0.00    0.00    0.13    0.00    0.00   12.03

03:22:55 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
03:22:56 PM  all   77.50    0.00    2.28    0.00    0.00    0.00    0.00    0.00    0.00   20.23
03:22:57 PM  all   78.63    0.00    1.39    0.00    0.00    0.00    0.00    0.00    0.00   19.97
03:22:58 PM  all   34.47    0.00    5.41    0.00    0.00    0.00    0.13    0.00    0.00   60.00
03:22:59 PM  all   17.00    0.00    5.16    0.00    0.00    0.00    0.00    0.00    0.00   77.83
03:23:00 PM  all   17.96    0.00    4.61    0.00    0.00    0.00    0.00    0.00    0.00   77.43
03:23:01 PM  all   37.44    0.00    4.02    0.00    0.00    0.00    0.00    0.00    0.00   58.54
03:23:02 PM  all   99.25    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    0.00
03:23:03 PM  all   88.51    0.00    1.26    0.00    0.00    0.00    0.00    0.00    0.00   10.23
03:23:04 PM  all   51.31    0.00    2.63    0.00    0.00    0.00    0.00    0.00    0.00   46.06
03:23:05 PM  all   99.12    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    0.25
03:23:06 PM  all   96.00    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    3.62
03:23:07 PM  all   38.74    0.00    1.64    0.88    0.00    0.00    0.00    0.00    0.00   58.74
03:23:08 PM  all   24.02    0.00    1.64    0.13    0.00    0.13    0.13    0.00    0.00   73.96
03:23:09 PM  all    1.25    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00   98.25

Screen Shot 2019-04-30 at 8 36 45 AM

10M:

[0] Tesla V100-SXM2-16GB | 26'C,   0 % |     0 / 16160 MB |
[0] Tesla V100-SXM2-16GB | 27'C,   4 % |   454 / 16160 MB | root(444M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % |   444 / 16160 MB | root(434M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % |   444 / 16160 MB | root(434M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % |   444 / 16160 MB | root(434M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % |   444 / 16160 MB | root(434M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % |   444 / 16160 MB | root(434M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % |   444 / 16160 MB | root(434M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % |   452 / 16160 MB | root(442M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % |   452 / 16160 MB | root(442M)
[0] Tesla V100-SXM2-16GB | 28'C,  58 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 29'C,  69 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 29'C,  72 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 31'C,  69 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 30'C,  70 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 31'C,  60 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 30'C,  69 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 32'C,  67 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 31'C,  68 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 31'C,  69 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 31'C,  63 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 31'C,  58 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 32'C,  64 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 32'C,  63 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 31'C,  69 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 30'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 30'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   444 / 16160 MB | root(434M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |     0 / 16160 MB |

08:49:38 PM  all    0.88    0.00    0.13    0.00    0.00    0.00    0.00    0.00    0.00   99.00
08:49:39 PM  all   46.39    0.00    1.77    0.00    0.00    0.00    0.38    0.00    0.00   51.46
08:49:40 PM  all   34.13    0.00    2.91    0.00    0.00    0.00    0.00    0.00    0.00   62.96
08:49:41 PM  all   13.88    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00   85.62
08:49:42 PM  all   12.16    0.00    2.26    0.00    0.00    0.00    0.00    0.00    0.00   85.59
08:49:43 PM  all   10.88    0.00    3.50    0.00    0.00    0.00    0.00    0.00    0.00   85.62
08:49:44 PM  all   74.34    0.00    2.89    0.00    0.00    0.00    0.13    0.00    0.00   22.64
08:49:45 PM  all   92.69    0.00    1.89    0.00    0.00    0.00    0.00    0.00    0.00    5.42
08:49:46 PM  all   80.90    0.00    2.76    0.00    0.00    0.00    0.00    0.00    0.00   16.33
08:49:47 PM  all   46.88    0.00    1.00    0.00    0.00    0.00    0.00    0.00    0.00   52.12
08:49:48 PM  all   28.07    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00   71.43
08:49:49 PM  all   40.28    0.00    2.01    0.00    0.00    0.00    0.00    0.00    0.00   57.72
08:49:50 PM  all   32.75    0.00    2.88    0.00    0.00    0.00    0.00    0.00    0.00   64.38
08:49:51 PM  all   26.53    0.00    5.51    0.25    0.00    0.00    0.13    0.00    0.00   67.58
08:49:52 PM  all   30.61    0.00    4.64    0.00    0.00    0.00    0.00    0.00    0.00   64.74
08:49:53 PM  all   28.57    0.00    4.76    0.00    0.00    0.00    0.00    0.00    0.00   66.67
08:49:54 PM  all   28.45    0.00    4.14    0.00    0.00    0.00    0.00    0.00    0.00   67.42
08:49:55 PM  all   31.45    0.00    3.65    0.00    0.00    0.00    0.00    0.00    0.00   64.91
08:49:56 PM  all   31.45    0.00    3.88    0.00    0.00    0.00    0.00    0.00    0.00   64.66
08:49:57 PM  all   31.03    0.00    3.89    0.00    0.00    0.00    0.00    0.00    0.00   65.08
08:49:58 PM  all   30.61    0.00    3.76    0.00    0.00    0.00    0.00    0.00    0.00   65.62
08:49:59 PM  all   31.37    0.00    4.39    0.00    0.00    0.00    0.00    0.00    0.00   64.24
08:50:00 PM  all   31.53    0.00    3.77    0.00    0.00    0.00    0.00    0.00    0.00   64.70
08:50:01 PM  all   30.33    0.00    3.63    0.00    0.00    0.00    0.00    0.00    0.00   66.04
08:50:02 PM  all   30.65    0.00    4.40    0.00    0.00    0.00    0.00    0.00    0.00   64.95
08:50:03 PM  all   32.46    0.00    4.14    0.00    0.00    0.00    0.00    0.00    0.00   63.41
08:50:04 PM  all   34.00    0.00    3.50    0.00    0.00    0.00    0.00    0.00    0.00   62.50
08:50:05 PM  all   30.70    0.00    4.26    0.00    0.00    0.00    0.00    0.00    0.00   65.04
08:50:06 PM  all   36.51    0.00    4.27    0.00    0.00    0.00    0.00    0.00    0.00   59.22
08:50:07 PM  all   98.88    0.00    1.12    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:08 PM  all   99.75    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:09 PM  all   99.62    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:10 PM  all   99.38    0.00    0.62    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:11 PM  all   99.25    0.00    0.62    0.00    0.00    0.00    0.00    0.00    0.00    0.12
08:50:12 PM  all   96.12    0.00    2.00    0.00    0.00    0.00    0.00    0.00    0.00    1.88
08:50:13 PM  all   99.88    0.00    0.12    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:14 PM  all   99.62    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:15 PM  all   99.38    0.00    0.62    0.00    0.00    0.00    0.00    0.00    0.00    0.00

08:50:15 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
08:50:16 PM  all   99.62    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:17 PM  all   98.75    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    0.63
08:50:18 PM  all   99.75    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:19 PM  all   99.88    0.00    0.12    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:20 PM  all   99.50    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:21 PM  all   99.75    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:22 PM  all   99.38    0.00    0.62    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:23 PM  all   99.50    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:24 PM  all   99.62    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:25 PM  all   99.50    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:26 PM  all   99.38    0.00    0.62    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:27 PM  all   99.50    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:28 PM  all   85.98    0.00    1.63    0.00    0.00    0.00    0.00    0.00    0.00   12.39
08:50:29 PM  all   86.36    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00   13.39
08:50:30 PM  all   44.65    0.00    2.52    0.00    0.00    0.00    0.00    0.00    0.00   52.83
08:50:31 PM  all   99.50    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:32 PM  all   99.75    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:33 PM  all   99.62    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:34 PM  all   99.62    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:35 PM  all   99.87    0.00    0.13    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:36 PM  all   98.75    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    0.75
08:50:37 PM  all   99.75    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:38 PM  all   99.50    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:39 PM  all   99.50    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:40 PM  all   99.75    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:41 PM  all   99.12    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.50
08:50:42 PM  all   98.13    0.00    1.87    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:43 PM  all   99.62    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:44 PM  all   99.62    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:45 PM  all   99.75    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:46 PM  all   99.75    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:47 PM  all   99.50    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.12
08:50:48 PM  all   99.62    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:49 PM  all   99.50    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:50 PM  all   99.88    0.00    0.12    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:51 PM  all   99.75    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:52 PM  all   90.99    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    8.26
08:50:53 PM  all   89.92    0.00    0.50    0.00    0.00    0.00    0.13    0.00    0.00    9.45
08:50:54 PM  all   19.90    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00   79.60
08:50:55 PM  all   53.69    0.00    2.67    0.00    0.00    0.00    0.13    0.00    0.00   43.51
08:50:56 PM  all    1.51    0.00    1.13    0.00    0.00    0.00    0.00    0.00    0.00   97.36

@szilard szilard changed the title GPU usage patterns GPU utilization patterns Apr 30, 2019
@szilard
Copy link
Owner Author

szilard commented Apr 30, 2019 •
edited
Loading

catboost:

1M:

[0] Tesla V100-SXM2-16GB | 28'C,   0 % |     0 / 16160 MB |
[0] Tesla V100-SXM2-16GB | 28'C,   6 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 28'C,   4 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,  85 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,  86 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,  81 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C,  82 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C,  84 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 29'C,  92 % |   452 / 16160 MB | root(442M)
[0] Tesla V100-SXM2-16GB | 29'C,   7 % |     0 / 16160 MB |

Screen Shot 2019-05-01 at 11 58 16 AM

08:35:45 PM  all    1.25    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00   98.38
08:35:46 PM  all   15.31    0.00    2.01    0.00    0.00    0.00    0.00    0.00    0.00   82.69
08:35:47 PM  all   19.15    0.00   23.90    0.00    0.00    0.00    0.00    0.00    0.00   56.95
08:35:48 PM  all   20.05    0.00   10.47    0.00    0.00    0.00    0.00    0.00    0.00   69.48
08:35:49 PM  all   14.14    0.00    8.96    0.00    0.00    0.00    0.13    0.00    0.00   76.77
08:35:50 PM  all    8.33    0.00    6.15    0.00    0.00    0.00    0.13    0.00    0.00   85.38
08:35:51 PM  all    9.71    0.00    6.26    0.00    0.00    0.00    0.13    0.00    0.00   83.91
08:35:52 PM  all    9.24    0.00    7.45    0.00    0.00    0.00    0.13    0.00    0.00   83.18
08:35:53 PM  all    8.25    0.00    7.60    0.00    0.00    0.00    0.13    0.00    0.00   84.02
08:35:54 PM  all    8.71    0.00    7.68    0.00    0.00    0.00    0.00    0.00    0.00   83.61
08:35:55 PM  all   54.24    0.00   10.24    0.00    0.00    0.00    0.25    0.00    0.00   35.27
08:35:56 PM  all   73.56    0.00    8.77    0.00    0.00    0.00    0.00    0.00    0.00   17.67
08:35:57 PM  all   30.36    0.00   10.04    0.00    0.00    0.00    0.00    0.00    0.00   59.60
08:35:58 PM  all    1.65    0.00    0.76    0.25    0.00    0.00    0.00    0.00    0.00   97.34

Screen Shot 2019-05-01 at 11 58 53 AM

10M:

[0] Tesla V100-SXM2-16GB | 26'C,   0 % |     0 / 16160 MB |
[0] Tesla V100-SXM2-16GB | 27'C,  16 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   8 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,  10 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,  13 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   7 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   3 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   4 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   7 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   2 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   7 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   2 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   4 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   2 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 28'C,  11 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 29'C,  71 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,  97 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  97 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 33'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  97 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  96 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 33'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 33'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 33'C,  97 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 33'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 33'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 33'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 33'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 29'C, 100 % |   452 / 16160 MB | root(442M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   452 / 16160 MB | root(442M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |     0 / 16160 MB |

Screen Shot 2019-05-01 at 12 01 56 PM

09:06:58 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
09:06:59 PM  all    1.25    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00   98.37
09:07:00 PM  all    5.00    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00   94.62
09:07:01 PM  all   67.12    0.00    4.25    0.00    0.00    0.00    0.00    0.00    0.00   28.62
09:07:02 PM  all   94.63    0.00    4.49    0.00    0.00    0.00    0.00    0.00    0.00    0.87
09:07:03 PM  all   25.97    0.00    2.89    0.00    0.00    0.00    0.00    0.00    0.00   71.14
09:07:04 PM  all   27.00    0.00   32.12    0.00    0.00    0.00    0.00    0.00    0.00   40.88
09:07:05 PM  all   41.43    0.00   58.32    0.00    0.00    0.00    0.00    0.00    0.00    0.25
09:07:06 PM  all   18.02    0.00    1.75    0.00    0.00    0.00    0.00    0.00    0.00   80.23
09:07:07 PM  all   19.00    0.00   29.12    0.00    0.00    0.00    0.00    0.00    0.00   51.88
09:07:08 PM  all   27.78    0.00   65.08    0.00    0.00    0.00    0.00    0.00    0.00    7.13
09:07:09 PM  all   12.17    0.00    2.01    0.00    0.00    0.00    0.00    0.00    0.00   85.82
09:07:10 PM  all   21.31    0.00   11.98    0.00    0.00    0.00    0.00    0.00    0.00   66.71
09:07:11 PM  all   19.10    0.00   10.24    0.00    0.00    0.00    0.12    0.00    0.00   70.54
09:07:12 PM  all   19.02    0.00   10.26    0.00    0.00    0.00    0.00    0.00    0.00   70.71
09:07:13 PM  all   18.52    0.00   10.51    0.00    0.00    0.00    0.00    0.00    0.00   70.96
09:07:14 PM  all   21.03    0.00   10.26    0.00    0.00    0.00    0.00    0.00    0.00   68.71
09:07:15 PM  all   18.57    0.00   10.29    0.00    0.00    0.00    0.00    0.00    0.00   71.14
09:07:16 PM  all   19.40    0.00   10.01    0.00    0.00    0.00    0.00    0.00    0.00   70.59
09:07:17 PM  all   20.73    0.00    9.55    0.00    0.00    0.00    0.00    0.00    0.00   69.72
09:07:18 PM  all   21.03    0.00   10.45    0.00    0.00    0.00    0.00    0.00    0.00   68.51
09:07:19 PM  all   22.00    0.00   10.12    0.00    0.00    0.00    0.00    0.00    0.00   67.88
09:07:20 PM  all   20.08    0.00   10.41    0.00    0.00    0.00    0.00    0.00    0.00   69.51
09:07:21 PM  all   22.31    0.00   10.40    0.00    0.00    0.00    0.00    0.00    0.00   67.29
09:07:22 PM  all   20.43    0.00    9.77    0.00    0.00    0.00    0.00    0.00    0.00   69.80
09:07:23 PM  all   22.22    0.00   10.24    0.00    0.00    0.00    0.00    0.00    0.00   67.54
09:07:24 PM  all   21.96    0.00    9.91    0.00    0.00    0.00    0.00    0.00    0.00   68.13
09:07:25 PM  all   20.57    0.00    9.98    0.00    0.00    0.00    0.12    0.00    0.00   69.33
09:07:26 PM  all   18.47    0.00   10.05    0.00    0.00    0.00    0.00    0.00    0.00   71.48
09:07:27 PM  all   12.92    0.00    8.66    0.00    0.00    0.00    0.00    0.00    0.00   78.42
09:07:28 PM  all    6.78    0.00    7.54    0.00    0.00    0.00    0.00    0.00    0.00   85.68
09:07:29 PM  all    7.89    0.00    6.77    0.00    0.00    0.00    0.00    0.00    0.00   85.34
09:07:30 PM  all    7.13    0.00    7.38    0.00    0.00    0.00    0.00    0.00    0.00   85.48
09:07:31 PM  all    7.65    0.00    6.52    0.00    0.00    0.00    0.00    0.00    0.00   85.82
09:07:32 PM  all    7.76    0.00    6.63    0.00    0.00    0.00    0.00    0.00    0.00   85.61
09:07:33 PM  all    7.15    0.00    7.28    0.00    0.00    0.00    0.00    0.00    0.00   85.57
09:07:34 PM  all    6.64    0.00    6.52    0.00    0.00    0.00    0.00    0.00    0.00   86.84
09:07:35 PM  all    8.76    0.00    6.13    0.00    0.00    0.00    0.00    0.00    0.00   85.11
09:07:36 PM  all    7.28    0.00    7.28    0.00    0.00    0.00    0.00    0.00    0.00   85.45
09:07:37 PM  all    6.88    0.00    7.51    0.00    0.00    0.00    0.00    0.00    0.00   85.61
09:07:38 PM  all    6.89    0.00    7.52    0.00    0.00    0.00    0.00    0.00    0.00   85.59
09:07:39 PM  all    7.38    0.00    7.01    0.00    0.00    0.00    0.13    0.00    0.00   85.48
09:07:40 PM  all    7.28    0.00    7.40    0.00    0.00    0.00    0.00    0.00    0.00   85.32
09:07:41 PM  all    7.38    0.00    7.01    0.00    0.00    0.00    0.00    0.00    0.00   85.61
09:07:42 PM  all    8.15    0.00    6.27    0.00    0.00    0.00    0.00    0.00    0.00   85.59
09:07:43 PM  all    7.13    0.00    6.13    0.00    0.00    0.00    0.00    0.00    0.00   86.73
09:07:44 PM  all    7.14    0.00    7.27    0.00    0.00    0.00    0.00    0.00    0.00   85.59
09:07:45 PM  all    7.24    0.00    7.49    0.00    0.00    0.00    0.00    0.00    0.00   85.27
09:07:46 PM  all    7.39    0.00    7.02    0.00    0.00    0.00    0.00    0.00    0.00   85.59
09:07:47 PM  all    7.40    0.00    6.90    0.00    0.00    0.00    0.00    0.00    0.00   85.70
09:07:48 PM  all    7.88    0.00    6.51    0.00    0.00    0.00    0.00    0.00    0.00   85.61
09:07:49 PM  all    7.13    0.00    7.26    0.00    0.00    0.00    0.00    0.00    0.00   85.61
09:07:50 PM  all    6.90    0.00    7.53    0.00    0.00    0.00    0.00    0.00    0.00   85.57
09:07:51 PM  all    7.39    0.00    6.77    0.00    0.00    0.00    0.00    0.00    0.00   85.84

09:07:51 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
09:07:52 PM  all    6.76    0.00    7.38    0.00    0.00    0.00    0.00    0.00    0.00   85.86
09:07:53 PM  all    8.27    0.00    6.14    0.00    0.00    0.00    0.13    0.00    0.00   85.46
09:07:54 PM  all    7.77    0.00    6.52    0.00    0.00    0.00    0.00    0.00    0.00   85.71
09:07:55 PM  all    7.14    0.00    7.52    0.00    0.00    0.00    0.00    0.00    0.00   85.34
09:07:56 PM  all    8.14    0.00    6.38    0.00    0.00    0.00    0.00    0.00    0.00   85.48
09:07:57 PM  all    8.27    0.00    6.02    0.00    0.00    0.00    0.13    0.00    0.00   85.59
09:07:58 PM  all    7.17    0.00    7.30    0.00    0.00    0.00    0.00    0.00    0.00   85.53
09:07:59 PM  all    7.27    0.00    7.27    0.00    0.00    0.00    0.00    0.00    0.00   85.46
09:08:00 PM  all    6.52    0.00    6.89    0.00    0.00    0.00    0.00    0.00    0.00   86.59
09:08:01 PM  all    7.39    0.00    6.52    0.00    0.00    0.00    0.00    0.00    0.00   86.09
09:08:02 PM  all    7.53    0.00    7.03    0.00    0.00    0.00    0.00    0.00    0.00   85.45
09:08:03 PM  all    7.88    0.00    6.63    0.00    0.00    0.00    0.00    0.00    0.00   85.48
09:08:04 PM  all    8.66    0.00    6.15    0.00    0.00    0.00    0.00    0.00    0.00   85.19
09:08:05 PM  all    7.01    0.00    8.26    0.00    0.00    0.00    0.00    0.00    0.00   84.73
09:08:06 PM  all    8.65    0.00    5.76    0.00    0.00    0.00    0.00    0.00    0.00   85.59
09:08:07 PM  all    7.89    0.00    6.64    0.00    0.00    0.00    0.00    0.00    0.00   85.46
09:08:08 PM  all    7.27    0.00    7.27    0.00    0.00    0.00    0.13    0.00    0.00   85.34
09:08:09 PM  all    7.14    0.00    6.39    0.00    0.00    0.00    0.00    0.00    0.00   86.47
09:08:10 PM  all    7.27    0.00    6.89    0.00    0.00    0.00    0.00    0.00    0.00   85.84
09:08:11 PM  all    7.63    0.00    7.13    0.00    0.00    0.00    0.00    0.00    0.00   85.23
09:08:12 PM  all    7.64    0.00    6.89    0.00    0.00    0.00    0.00    0.00    0.00   85.46
09:08:13 PM  all    7.77    0.00    7.02    0.00    0.00    0.00    0.00    0.00    0.00   85.21
09:08:14 PM  all    8.40    0.00    6.39    0.00    0.00    0.00    0.00    0.00    0.00   85.21
09:08:15 PM  all    7.89    0.00    6.77    0.00    0.00    0.00    0.00    0.00    0.00   85.34
09:08:16 PM  all    7.66    0.00    6.78    0.00    0.00    0.00    0.00    0.00    0.00   85.55
09:08:17 PM  all    6.90    0.00    7.53    0.00    0.00    0.00    0.00    0.00    0.00   85.57
09:08:18 PM  all    7.15    0.00    6.27    0.00    0.00    0.00    0.00    0.00    0.00   86.57
09:08:19 PM  all    7.01    0.00    7.38    0.00    0.00    0.00    0.13    0.00    0.00   85.48
09:08:20 PM  all    8.78    0.00    5.65    0.00    0.00    0.00    0.00    0.00    0.00   85.57
09:08:21 PM  all    7.52    0.00    7.02    0.00    0.00    0.00    0.00    0.00    0.00   85.46
09:08:22 PM  all    7.52    0.00    7.52    0.00    0.00    0.00    0.00    0.00    0.00   84.96
09:08:23 PM  all    7.15    0.00    7.65    0.00    0.00    0.00    0.00    0.00    0.00   85.19
09:08:24 PM  all    7.02    0.00    7.39    0.00    0.00    0.00    0.00    0.00    0.00   85.59
09:08:25 PM  all    7.90    0.00    6.90    0.00    0.00    0.00    0.00    0.00    0.00   85.19
09:08:26 PM  all    6.78    0.00    7.65    0.00    0.00    0.00    0.00    0.00    0.00   85.57
09:08:27 PM  all    6.74    0.00    6.99    0.00    0.00    0.00    0.00    0.00    0.00   86.27
09:08:28 PM  all    7.76    0.00    7.26    0.00    0.00    0.00    0.13    0.00    0.00   84.86
09:08:29 PM  all    7.52    0.00    7.14    0.00    0.00    0.00    0.00    0.00    0.00   85.34
09:08:30 PM  all    7.78    0.00    7.03    0.00    0.00    0.00    0.00    0.00    0.00   85.19
09:08:31 PM  all    7.52    0.00    7.39    0.00    0.00    0.00    0.00    0.00    0.00   85.09
09:08:32 PM  all    7.76    0.00    7.01    0.00    0.00    0.00    0.00    0.00    0.00   85.23
09:08:33 PM  all    7.30    0.00    7.30    0.00    0.00    0.00    0.00    0.00    0.00   85.41
09:08:34 PM  all    7.74    0.00    7.12    0.00    0.00    0.00    0.00    0.00    0.00   85.14
09:08:35 PM  all    7.25    0.00    6.75    0.00    0.00    0.00    0.00    0.00    0.00   86.00
09:08:36 PM  all    6.78    0.00    7.40    0.00    0.00    0.00    0.00    0.00    0.00   85.82
09:08:37 PM  all    7.28    0.00    7.40    0.00    0.00    0.00    0.00    0.00    0.00   85.32
09:08:38 PM  all    7.64    0.00    7.39    0.00    0.00    0.00    0.00    0.00    0.00   84.96
09:08:39 PM  all    7.78    0.00    7.28    0.00    0.00    0.00    0.00    0.00    0.00   84.94
09:08:40 PM  all    7.77    0.00    7.02    0.00    0.00    0.00    0.00    0.00    0.00   85.21
09:08:41 PM  all    7.40    0.00    7.28    0.00    0.00    0.00    0.00    0.00    0.00   85.32
09:08:42 PM  all    8.26    0.00    6.38    0.00    0.00    0.00    0.13    0.00    0.00   85.23
09:08:43 PM  all    7.66    0.00    7.41    0.00    0.00    0.00    0.00    0.00    0.00   84.92
09:08:44 PM  all    6.88    0.00    6.88    0.00    0.00    0.00    0.12    0.00    0.00   86.12

09:08:44 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
09:08:45 PM  all    7.15    0.00    7.40    0.00    0.00    0.00    0.00    0.00    0.00   85.45
09:08:46 PM  all    7.76    0.00    7.38    0.00    0.00    0.00    0.00    0.00    0.00   84.86
09:08:47 PM  all    7.78    0.00    7.15    0.00    0.00    0.00    0.00    0.00    0.00   85.07
09:08:48 PM  all    7.63    0.00    7.26    0.00    0.00    0.00    0.00    0.00    0.00   85.11
09:08:49 PM  all    7.79    0.00    7.04    0.00    0.00    0.00    0.00    0.00    0.00   85.18
09:08:50 PM  all    7.54    0.00    7.54    0.00    0.00    0.00    0.00    0.00    0.00   84.92
09:08:51 PM  all    7.27    0.00    7.52    0.00    0.00    0.00    0.00    0.00    0.00   85.21
09:08:52 PM  all    7.39    0.00    7.39    0.00    0.00    0.00    0.00    0.00    0.00   85.21
09:08:53 PM  all    6.40    0.00    7.15    0.00    0.00    0.00    0.00    0.00    0.00   86.45
09:08:54 PM  all    7.77    0.00    6.89    0.00    0.00    0.00    0.00    0.00    0.00   85.34
09:08:55 PM  all    7.90    0.00    7.28    0.00    0.00    0.00    0.00    0.00    0.00   84.82
09:08:56 PM  all    8.77    0.00    6.39    0.00    0.00    0.00    0.00    0.00    0.00   84.84
09:08:57 PM  all    7.77    0.00    7.02    0.00    0.00    0.00    0.00    0.00    0.00   85.21
09:08:58 PM  all    8.02    0.00    7.14    0.00    0.00    0.00    0.13    0.00    0.00   84.71
09:08:59 PM  all    7.90    0.00    7.40    0.00    0.00    0.00    0.00    0.00    0.00   84.69
09:09:00 PM  all    7.65    0.00    7.28    0.00    0.00    0.00    0.00    0.00    0.00   85.07
09:09:01 PM  all    7.14    0.00    7.27    0.00    0.00    0.00    0.00    0.00    0.00   85.59
09:09:02 PM  all   73.93    0.00    9.90    0.13    0.00    0.00    0.00    0.00    0.00   16.04
09:09:03 PM  all   98.75    0.00    1.12    0.00    0.00    0.00    0.00    0.00    0.00    0.12
09:09:04 PM  all   98.50    0.00    1.50    0.00    0.00    0.00    0.00    0.00    0.00    0.00
09:09:05 PM  all   99.00    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    0.25
09:09:06 PM  all   98.87    0.00    1.13    0.00    0.00    0.00    0.00    0.00    0.00    0.00
09:09:07 PM  all   99.50    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.12
09:09:08 PM  all   99.38    0.00    0.62    0.00    0.00    0.00    0.00    0.00    0.00    0.00
09:09:09 PM  all   99.00    0.00    0.62    0.00    0.00    0.00    0.00    0.00    0.00    0.37
09:09:10 PM  all   98.50    0.00    1.50    0.00    0.00    0.00    0.00    0.00    0.00    0.00
09:09:11 PM  all   99.25    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    0.00
09:09:12 PM  all   99.13    0.00    0.62    0.00    0.00    0.00    0.00    0.00    0.00    0.25
09:09:13 PM  all   99.12    0.00    0.88    0.00    0.00    0.00    0.00    0.00    0.00    0.00
09:09:14 PM  all   99.25    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    0.00
09:09:15 PM  all   48.31    0.00    6.76    0.00    0.00    0.00    0.00    0.00    0.00   44.93
09:09:16 PM  all   31.99    0.00   10.79    0.00    0.00    0.00    0.00    0.00    0.00   57.21
09:09:17 PM  all    5.92    0.00    2.14    0.13    0.00    0.00    0.13    0.00    0.00   91.69
09:09:18 PM  all    0.88    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00   98.50

Screen Shot 2019-05-01 at 12 00 46 PM

Screen Shot 2019-05-01 at 12 01 05 PM

@szilard
Copy link
Owner Author

szilard commented May 1, 2019 •
edited
Loading

lightgbm

1M

[0] Tesla V100-SXM2-16GB | 26'C,   0 % |     0 / 16160 MB |
[0] Tesla V100-SXM2-16GB | 27'C,   5 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,   7 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,   9 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,  10 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,  12 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,  12 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,   2 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,   3 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,   5 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,   6 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,   8 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,  10 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,  11 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,   2 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % |     0 / 16160 MB |

Screen Shot 2019-05-01 at 7 09 54 AM

02:07:04 PM  all    2.13    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00   97.62
02:07:05 PM  all   45.54    0.00    1.66    0.00    0.00    0.00    0.00    0.00    0.00   52.81
02:07:06 PM  all   95.22    0.00    1.76    0.00    0.00    0.00    0.00    0.00    0.00    3.02
02:07:07 PM  all   97.87    0.00    1.38    0.00    0.00    0.00    0.13    0.00    0.00    0.63
02:07:08 PM  all   97.37    0.00    1.00    0.00    0.00    0.00    0.00    0.00    0.00    1.63
02:07:09 PM  all   97.62    0.00    1.25    0.00    0.00    0.00    0.00    0.00    0.00    1.13
02:07:10 PM  all   97.62    0.00    1.38    0.00    0.00    0.00    0.00    0.00    0.00    1.00
02:07:11 PM  all   97.62    0.00    1.38    0.00    0.00    0.00    0.00    0.00    0.00    1.00
02:07:12 PM  all   97.61    0.00    0.75    0.00    0.00    0.00    0.13    0.00    0.00    1.51
02:07:13 PM  all   97.87    0.00    1.51    0.00    0.00    0.00    0.00    0.00    0.00    0.63
02:07:14 PM  all   98.49    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    0.75
02:07:15 PM  all   97.22    0.00    1.77    0.00    0.00    0.00    0.00    0.00    0.00    1.01
02:07:16 PM  all   98.24    0.00    1.13    0.00    0.00    0.00    0.00    0.00    0.00    0.63
02:07:17 PM  all   97.99    0.00    1.50    0.00    0.00    0.00    0.00    0.00    0.00    0.50
02:07:18 PM  all   98.22    0.00    1.02    0.00    0.00    0.00    0.00    0.00    0.00    0.76
02:07:19 PM  all   98.12    0.00    1.13    0.00    0.00    0.00    0.00    0.00    0.00    0.75
02:07:20 PM  all   97.60    0.00    1.26    0.00    0.00    0.00    0.13    0.00    0.00    1.01
02:07:21 PM  all   97.37    0.00    1.75    0.00    0.00    0.00    0.00    0.00    0.00    0.88
02:07:22 PM  all   59.32    0.00    1.00    0.00    0.00    0.00    0.00    0.00    0.00   39.67
02:07:23 PM  all    2.02    0.00    1.26    0.25    0.00    0.00    0.00    0.00    0.00   96.47

Screen Shot 2019-05-01 at 7 09 35 AM

10M:

[0] Tesla V100-SXM2-16GB | 29'C,   0 % |     0 / 16160 MB |
[0] Tesla V100-SXM2-16GB | 30'C,   5 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 30'C,  10 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 30'C,   6 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 30'C,   8 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 30'C,   6 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 30'C,   6 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 30'C,   7 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 30'C,   8 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 30'C,   6 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 30'C,   1 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   2 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 30'C,   2 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   2 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 30'C,   4 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   3 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   8 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   5 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   6 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   7 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   6 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,  11 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   8 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   8 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   1 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   6 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   1 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   1 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   2 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   3 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   4 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   4 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   4 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   4 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   6 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   7 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   3 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   6 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   7 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   1 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   0 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   2 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   2 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   6 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   5 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   6 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   7 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   7 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   7 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   7 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   7 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   9 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   9 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   0 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   0 % |     0 / 16160 MB |

06:54:58 PM  all   90.21    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00    9.54
06:54:59 PM  all   93.39    0.00    0.62    0.00    0.00    0.00    0.00    0.00    0.00    5.99
06:55:00 PM  all   93.61    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    5.76
06:55:01 PM  all   88.71    0.00    1.25    0.00    0.00    0.00    0.00    0.00    0.00   10.04
06:55:02 PM  all   92.67    0.00    0.87    0.00    0.00    0.00    0.00    0.00    0.00    6.46
06:55:03 PM  all   94.49    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    4.76
06:55:04 PM  all   93.73    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    5.52
06:55:05 PM  all   92.60    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    6.90
06:55:06 PM  all   93.94    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    5.43
06:55:07 PM  all   88.57    0.00    1.01    0.00    0.00    0.00    0.00    0.00    0.00   10.43
06:55:08 PM  all   95.21    0.00    0.88    0.00    0.00    0.00    0.00    0.00    0.00    3.91
06:55:09 PM  all   91.59    0.00    0.63    0.00    0.00    0.00    0.13    0.00    0.00    7.65
06:55:10 PM  all   91.48    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    7.77
06:55:11 PM  all   95.64    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    3.86
06:55:12 PM  all   91.91    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    7.46
06:55:13 PM  all   94.70    0.00    0.51    0.00    0.00    0.00    0.00    0.00    0.00    4.80
06:55:14 PM  all   89.56    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    9.69
06:55:15 PM  all   94.10    0.00    0.88    0.00    0.00    0.00    0.00    0.00    0.00    5.02
06:55:16 PM  all   91.01    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    8.24

06:55:16 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
06:55:17 PM  all   95.21    0.00    1.01    0.00    0.00    0.00    0.00    0.00    0.00    3.78
06:55:18 PM  all   91.88    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    7.62
06:55:19 PM  all   95.38    0.00    0.62    0.00    0.00    0.00    0.00    0.00    0.00    4.00
06:55:20 PM  all   94.60    0.00    1.01    0.00    0.00    0.00    0.00    0.00    0.00    4.40
06:55:21 PM  all   94.94    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    4.43
06:55:22 PM  all   95.68    0.00    0.89    0.00    0.00    0.00    0.00    0.00    0.00    3.43
06:55:23 PM  all   92.89    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    6.47
06:55:24 PM  all   94.95    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    4.42
06:55:25 PM  all   95.11    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    4.39
06:55:26 PM  all   96.62    0.00    0.62    0.00    0.00    0.00    0.12    0.00    0.00    2.62
06:55:27 PM  all   93.24    0.00    0.88    0.00    0.00    0.00    0.00    0.00    0.00    5.88
06:55:28 PM  all   94.97    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    4.27
06:55:29 PM  all   93.07    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    6.30
06:55:30 PM  all   94.59    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    4.65
06:55:31 PM  all   94.82    0.00    0.51    0.00    0.00    0.00    0.00    0.00    0.00    4.68
06:55:32 PM  all   94.37    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    4.88
06:55:33 PM  all   95.96    0.00    0.76    0.00    0.00    0.00    0.00    0.00    0.00    3.28
06:55:34 PM  all   93.72    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    5.65
06:55:35 PM  all   92.93    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    6.44
06:55:36 PM  all   94.24    0.00    1.25    0.00    0.00    0.00    0.00    0.00    0.00    4.51
06:55:37 PM  all   95.08    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    4.29
06:55:38 PM  all   92.99    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    6.26
06:55:39 PM  all   94.62    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    4.76
06:55:40 PM  all   94.93    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00    4.82
06:55:41 PM  all   93.46    0.00    1.01    0.00    0.00    0.00    0.00    0.00    0.00    5.53
06:55:42 PM  all   92.29    0.00    0.88    0.00    0.00    0.00    0.00    0.00    0.00    6.83
06:55:43 PM  all   94.00    0.00    0.88    0.00    0.00    0.00    0.12    0.00    0.00    5.00
06:55:44 PM  all   54.76    0.00    2.76    0.00    0.00    0.00    0.00    0.00    0.00   42.48
06:55:45 PM  all    1.64    0.00    1.51    0.13    0.00    0.00    0.00    0.00    0.00   96.73

@szilard
Copy link
Owner Author

szilard commented May 12, 2019

lightgbm 1M again:

15.48 0.7660041

Screen Shot 2019-05-12 at 7 56 16 AM

nthread = 1

21.901 0.7661341

Screen Shot 2019-05-12 at 7 59 02 AM

@szilard
Copy link
Owner Author

szilard commented May 12, 2019 •
edited
Loading

Force matrix construct before/outside training with:

lgb.Dataset.construct(dlgb_train)

though this crashes, unless this hack #12 (comment)

dlgb_train <- lgb.Dataset( ... , device = "gpu")

@szilard
Copy link
Owner Author

szilard commented May 12, 2019

default threads (all cores):

construct:
user system elapsed
1.904 0.008 0.472

16.11 0.7660714

nthread = 1

construct:
user system elapsed
1.956 0.004 0.486

20.178 0.7661254

@StrikerRUS StrikerRUS mentioned this issue Jun 7, 2019
Closed
@StrikerRUS StrikerRUS mentioned this issue Aug 11, 2019
Closed
@szilard szilard mentioned this issue Sep 13, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
analysis in-readme
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@szilard