Data transfer stats: NUMA 0 -> CUDA 0 831.0734 GB 532.4227 MB/s (transfers : 14264 - avg 59.6620 MB) CUDA 0 -> NUMA 0 611.4485 GB 391.7205 MB/s (transfers : 9994 - avg 62.6499 MB) NUMA 0 -> CUDA 1 2506.1521 GB 1605.5461 MB/s (transfers : 28106 - avg 91.3079 MB) CUDA 1 -> NUMA 0 800.3239 GB 512.7197 MB/s (transfers : 7097 - avg 115.4758 MB) CUDA 0 -> CUDA 1 700.8801 GB 449.0111 MB/s (transfers : 12122 - avg 59.2065 MB) CUDA 1 -> CUDA 0 44.2218 GB 28.3302 MB/s (transfers : 2066 - avg 21.9183 MB) NUMA 0 -> CUDA 2 3029.9631 GB 1941.1108 MB/s (transfers : 28368 - avg 109.3726 MB) CUDA 2 -> NUMA 0 513.4538 GB 328.9381 MB/s (transfers : 5337 - avg 98.5154 MB) CUDA 0 -> CUDA 2 397.7839 GB 254.8354 MB/s (transfers : 7912 - avg 51.4826 MB) CUDA 2 -> CUDA 0 65.0442 GB 41.6697 MB/s (transfers : 1970 - avg 33.8098 MB) CUDA 1 -> CUDA 2 548.7111 GB 351.5248 MB/s (transfers : 5972 - avg 94.0858 MB) CUDA 2 -> CUDA 1 763.4075 GB 489.0671 MB/s (transfers : 8176 - avg 95.6127 MB) NUMA 0 -> CUDA 3 2734.7544 GB 1751.9841 MB/s (transfers : 36522 - avg 76.6768 MB) CUDA 3 -> NUMA 0 623.7239 GB 399.5803 MB/s (transfers : 12012 - avg 53.1713 MB) CUDA 0 -> CUDA 3 500.3915 GB 320.5689 MB/s (transfers : 8620 - avg 59.4433 MB) CUDA 3 -> CUDA 0 79.9590 GB 51.2246 MB/s (transfers : 2532 - avg 32.3373 MB) CUDA 1 -> CUDA 3 587.3207 GB 376.2585 MB/s (transfers : 8591 - avg 70.0054 MB) CUDA 3 -> CUDA 1 797.0267 GB 510.6034 MB/s (transfers : 10162 - avg 80.3144 MB) CUDA 2 -> CUDA 3 417.8017 GB 267.6584 MB/s (transfers : 6734 - avg 63.5327 MB) CUDA 3 -> CUDA 2 652.2828 GB 417.8749 MB/s (transfers : 8444 - avg 79.1020 MB) NUMA 0 -> CUDA 4 135.5900 GB 86.8636 MB/s (transfers : 14178 - avg 9.7929 MB) CUDA 4 -> NUMA 0 1533.6919 GB 982.5351 MB/s (transfers : 27964 - avg 56.1615 MB) CUDA 0 -> CUDA 4 549.8950 GB 352.2812 MB/s (transfers : 13437 - avg 41.9061 MB) CUDA 4 -> CUDA 0 91.6971 GB 58.7442 MB/s (transfers : 4174 - avg 22.4959 MB) CUDA 1 -> CUDA 4 912.7166 GB 584.7163 MB/s (transfers : 13679 - avg 68.3253 MB) CUDA 4 -> CUDA 1 685.7457 GB 439.3111 MB/s (transfers : 9283 - avg 75.6440 MB) CUDA 2 -> CUDA 4 841.8708 GB 539.3297 MB/s (transfers : 11860 - avg 72.6877 MB) CUDA 4 -> CUDA 2 738.7199 GB 473.2476 MB/s (transfers : 10009 - avg 75.5769 MB) CUDA 3 -> CUDA 4 1078.5719 GB 690.9673 MB/s (transfers : 15409 - avg 71.6761 MB) CUDA 4 -> CUDA 3 969.2849 GB 620.9543 MB/s (transfers : 14213 - avg 69.8338 MB) NUMA 0 -> CUDA 5 1738.8773 GB 1113.9787 MB/s (transfers : 27540 - avg 64.6554 MB) CUDA 5 -> NUMA 0 932.9167 GB 597.6550 MB/s (transfers : 13563 - avg 70.4348 MB) CUDA 0 -> CUDA 5 413.5740 GB 264.9480 MB/s (transfers : 9491 - avg 44.6212 MB) CUDA 5 -> CUDA 0 226.8103 GB 145.3014 MB/s (transfers : 4915 - avg 47.2541 MB) CUDA 1 -> CUDA 5 631.8674 GB 404.7930 MB/s (transfers : 9975 - avg 64.8654 MB) CUDA 5 -> CUDA 1 604.2794 GB 387.1191 MB/s (transfers : 8889 - avg 69.6121 MB) CUDA 2 -> CUDA 5 674.2890 GB 431.9691 MB/s (transfers : 9736 - avg 70.9195 MB) CUDA 5 -> CUDA 2 561.2480 GB 359.5515 MB/s (transfers : 8361 - avg 68.7379 MB) CUDA 3 -> CUDA 5 705.0195 GB 451.6548 MB/s (transfers : 10544 - avg 68.4693 MB) CUDA 5 -> CUDA 3 683.6862 GB 437.9873 MB/s (transfers : 10739 - avg 65.1918 MB) CUDA 4 -> CUDA 5 683.6790 GB 437.9821 MB/s (transfers : 10865 - avg 64.4351 MB) CUDA 5 -> CUDA 4 1443.5536 GB 924.7765 MB/s (transfers : 23592 - avg 62.6568 MB) NUMA 0 -> CUDA 6 1782.6906 GB 1142.0355 MB/s (transfers : 27757 - avg 65.7663 MB) CUDA 6 -> NUMA 0 842.0036 GB 539.4079 MB/s (transfers : 15129 - avg 56.9907 MB) CUDA 0 -> CUDA 6 544.2794 GB 348.6784 MB/s (transfers : 10810 - avg 51.5580 MB) CUDA 6 -> CUDA 0 204.1459 GB 130.7807 MB/s (transfers : 4349 - avg 48.0675 MB) CUDA 1 -> CUDA 6 724.4305 GB 464.0870 MB/s (transfers : 10095 - avg 73.4836 MB) CUDA 6 -> CUDA 1 598.4980 GB 383.4114 MB/s (transfers : 8360 - avg 73.3089 MB) CUDA 2 -> CUDA 6 675.3506 GB 432.6440 MB/s (transfers : 9060 - avg 76.3310 MB) CUDA 6 -> CUDA 2 462.3806 GB 296.2103 MB/s (transfers : 6054 - avg 78.2091 MB) CUDA 3 -> CUDA 6 673.0132 GB 431.1455 MB/s (transfers : 9840 - avg 70.0372 MB) CUDA 6 -> CUDA 3 811.6522 GB 519.9601 MB/s (transfers : 12500 - avg 66.4905 MB) CUDA 4 -> CUDA 6 853.1138 GB 546.5209 MB/s (transfers : 12967 - avg 67.3701 MB) CUDA 6 -> CUDA 4 1002.4733 GB 642.2031 MB/s (transfers : 15115 - avg 67.9148 MB) CUDA 5 -> CUDA 6 593.9438 GB 380.4913 MB/s (transfers : 9632 - avg 63.1435 MB) CUDA 6 -> CUDA 5 1007.6204 GB 645.4997 MB/s (transfers : 14974 - avg 68.9063 MB) NUMA 0 -> CUDA 7 1912.8256 GB 1225.3898 MB/s (transfers : 28613 - avg 68.4561 MB) CUDA 7 -> NUMA 0 712.2228 GB 456.2622 MB/s (transfers : 10320 - avg 70.6702 MB) CUDA 0 -> CUDA 7 500.0460 GB 320.3379 MB/s (transfers : 10015 - avg 51.1280 MB) CUDA 7 -> CUDA 0 213.1522 GB 136.5488 MB/s (transfers : 4596 - avg 47.4908 MB) CUDA 1 -> CUDA 7 533.1670 GB 341.5555 MB/s (transfers : 8871 - avg 61.5447 MB) CUDA 7 -> CUDA 1 537.7964 GB 344.5210 MB/s (transfers : 7405 - avg 74.3691 MB) CUDA 2 -> CUDA 7 725.8231 GB 464.9736 MB/s (transfers : 9258 - avg 80.2811 MB) CUDA 7 -> CUDA 2 505.6527 GB 323.9288 MB/s (transfers : 6900 - avg 75.0418 MB) CUDA 3 -> CUDA 7 579.8309 GB 371.4483 MB/s (transfers : 8834 - avg 67.2115 MB) CUDA 7 -> CUDA 3 738.1367 GB 472.8612 MB/s (transfers : 11272 - avg 67.0557 MB) CUDA 4 -> CUDA 7 767.3229 GB 491.5580 MB/s (transfers : 11935 - avg 65.8348 MB) CUDA 7 -> CUDA 4 1093.9052 GB 700.7709 MB/s (transfers : 16736 - avg 66.9311 MB) CUDA 5 -> CUDA 7 745.2955 GB 477.4464 MB/s (transfers : 11748 - avg 64.9628 MB) CUDA 7 -> CUDA 5 716.8555 GB 459.2264 MB/s (transfers : 11337 - avg 64.7491 MB) CUDA 6 -> CUDA 7 749.9323 GB 480.4148 MB/s (transfers : 12297 - avg 62.4486 MB) CUDA 7 -> CUDA 6 932.1804 GB 597.1641 MB/s (transfers : 14015 - avg 68.1094 MB) Total transfers: 56783.0508 GB #--------------------- Real time: 27:25.60