EPYC 7002 series optimization issue

[zhenglcc]

I have several dual socket EPYC platforms：
(1)2XEPYC 7542（32c） with 512(2933Mhz)GB ram
(2)2XEPYC 7R32（48c） with 512(2933Mhz)GB ram
same motherboard、SSD、BIOS version（configuration）、OS（centos linux 8.5）；

Test 1
EPYC 7542
./bladebit -t 124 -w ...
Finished plotting in 216.51 seconds (3.61 minutes).

Test 2
EPYC 7R32
./bladebit -t 192 -w ...
Finished plotting in 335.97 seconds (5.60 minutes).

Test 3
EPYC 7R32（only open 4CCDs simulation 7542（4CCD 32c））
./bladebit -t 124 -w ...
Finished plotting in 218.10 seconds (3.63 minutes)

as result,there is a very strange issue, when the CCDs morethan 4, the plotting speed will be greatly slowed down.why?i have check numa status。same issue with other EPYC CPU with 6-8 CCDs，like 7642 7742 7F52。

logs is here

Test1

Running Phase 1
Generating F1...
Finished F1 generation in 0.74 seconds.
Sorting F1...
Finished F1 sort in 3.95 seconds.
Forward propagating to table 2...
Pairing L/R groups...
Finished pairing L/R groups in 4.5650 seconds. Created 4294967296 pairs.
Average of 236.1406 pairs per group.
Computing Fx...
Finished computing Fx in 5.1670 seconds.
Sorting entries...
Finished sorting in 7.76 seconds.
Finished forward propagating table 2 in 17.73 seconds.
Forward propagating to table 3...
Pairing L/R groups...
Finished pairing L/R groups in 4.5520 seconds. Created 4294967296 pairs.
Average of 236.1406 pairs per group.
Computing Fx...
Finished computing Fx in 5.2080 seconds.
Sorting entries...
Finished sorting in 7.90 seconds.
Finished forward propagating table 3 in 17.89 seconds.
Forward propagating to table 4...
Pairing L/R groups...
Finished pairing L/R groups in 4.5570 seconds. Created 4294967296 pairs.
Average of 236.1406 pairs per group.
Computing Fx...
Finished computing Fx in 5.3700 seconds.
Sorting entries...
Finished sorting in 7.88 seconds.
Finished forward propagating table 4 in 18.04 seconds.
Forward propagating to table 5...
Pairing L/R groups...
Finished pairing L/R groups in 4.5640 seconds. Created 4294962145 pairs.
Average of 236.1403 pairs per group.
Computing Fx...
Finished computing Fx in 5.3720 seconds.
Sorting entries...
Finished sorting in 7.90 seconds.
Finished forward propagating table 5 in 18.07 seconds.
Forward propagating to table 6...
Pairing L/R groups...
Finished pairing L/R groups in 4.5640 seconds. Created 4294919162 pairs.
Average of 236.1380 pairs per group.
Computing Fx...
Finished computing Fx in 5.2750 seconds.
Sorting entries...
Finished sorting in 7.78 seconds.
Finished forward propagating table 6 in 17.84 seconds.
Forward propagating to table 7...
Pairing L/R groups...
Finished pairing L/R groups in 4.5650 seconds. Created 4294794128 pairs.
Average of 236.1311 pairs per group.
Computing Fx...
Finished computing Fx in 5.1330 seconds.
Finished forward propagating table 7 in 9.91 seconds.
Finished Phase 1 in 104.20 seconds.
Running Phase 2
Prunning table 6...
Finished prunning table 6 in 0.27 seconds.
Prunning table 5...
Finished prunning table 5 in 8.20 seconds.
Prunning table 4...
Finished prunning table 4 in 7.79 seconds.
Prunning table 3...
Finished prunning table 3 in 7.65 seconds.
Prunning table 2...
Finished prunning table 2 in 7.60 seconds.
Finished Phase 2 in 31.70 seconds.
Running Phase 3
Compressing tables 1 and 2...
Finished compressing tables 1 and 2 in 12.46 seconds
Table 1 now has 3429410941 / 4294967296 entries ( 79.85% ).
Compressing tables 2 and 3...
Finished compressing tables 2 and 3 in 12.53 seconds
Table 2 now has 3439877690 / 4294967296 entries ( 80.09% ).
Compressing tables 3 and 4...
Finished compressing tables 3 and 4 in 12.63 seconds
Table 3 now has 3466106784 / 4294967296 entries ( 80.70% ).
Compressing tables 4 and 5...
Finished compressing tables 4 and 5 in 12.94 seconds
Table 4 now has 3533019526 / 4294962145 entries ( 82.26% ).
Compressing tables 5 and 6...
Finished compressing tables 5 and 6 in 13.66 seconds
Table 5 now has 3713615497 / 4294919162 entries ( 86.47% ).
Compressing tables 6 and 7...
Finished compressing tables 6 and 7 in 15.96 seconds
Table 6 now has 4294794128 / 4294794128 entries ( 100.00% ).
Finished Phase 3 in 80.18 seconds.
Running Phase 4
Writing P7.
Finished writing P7 in 0.27 seconds.
Writing C1 table.
Finished writing C1 table in 0.00 seconds.
Writing C2 table.
Finished writing C2 table in 0.00 seconds.
Writing C3 table.
Finished writing C3 table in 0.16 seconds.
Finished Phase 4 in 0.43 seconds.
Finished plotting in 216.51 seconds (3.61 minutes).

Test2

Running Phase 1
Generating F1...
Finished F1 generation in 0.69 seconds.
Sorting F1...
Finished F1 sort in 4.71 seconds.
Forward propagating to table 2...
Pairing L/R groups...
Finished pairing L/R groups in 3.7340 seconds. Created 4294967296 pairs.
Average of 236.1406 pairs per group.
Computing Fx...
Finished computing Fx in 4.1580 seconds.
Sorting entries...
Finished sorting in 19.94 seconds.
Finished forward propagating table 2 in 28.01 seconds.
Forward propagating to table 3...
Pairing L/R groups...
Finished pairing L/R groups in 3.6840 seconds. Created 4294956986 pairs.
Average of 236.1401 pairs per group.
Computing Fx...
Finished computing Fx in 4.2390 seconds.
Sorting entries...
Finished sorting in 18.93 seconds.
Finished forward propagating table 3 in 27.03 seconds.
Forward propagating to table 4...
Pairing L/R groups...
Finished pairing L/R groups in 3.6940 seconds. Created 4294967296 pairs.
Average of 236.1406 pairs per group.
Computing Fx...
Finished computing Fx in 4.3760 seconds.
Sorting entries...
Finished sorting in 19.06 seconds.
Finished forward propagating table 4 in 27.31 seconds.
Forward propagating to table 5...
Pairing L/R groups...
Finished pairing L/R groups in 3.7080 seconds. Created 4294844925 pairs.
Average of 236.1339 pairs per group.
Computing Fx...
Finished computing Fx in 4.4680 seconds.
Sorting entries...
Finished sorting in 18.97 seconds.
Finished forward propagating table 5 in 27.32 seconds.
Forward propagating to table 6...
Pairing L/R groups...
Finished pairing L/R groups in 3.7120 seconds. Created 4294689248 pairs.
Average of 236.1253 pairs per group.
Computing Fx...
Finished computing Fx in 4.2880 seconds.
Sorting entries...
Finished sorting in 19.79 seconds.
Finished forward propagating table 6 in 27.96 seconds.
Forward propagating to table 7...
Pairing L/R groups...
Finished pairing L/R groups in 3.7320 seconds. Created 4294324297 pairs.
Average of 236.1053 pairs per group.
Computing Fx...
Finished computing Fx in 4.1790 seconds.
Finished forward propagating table 7 in 8.09 seconds.
Finished Phase 1 in 151.14 seconds.
Running Phase 2
Prunning table 6...
Finished prunning table 6 in 0.22 seconds.
Prunning table 5...
Finished prunning table 5 in 9.33 seconds.
Prunning table 4...
Finished prunning table 4 in 8.83 seconds.
Prunning table 3...
Finished prunning table 3 in 8.72 seconds.
Prunning table 2...
Finished prunning table 2 in 8.65 seconds.
Finished Phase 2 in 35.92 seconds.
Running Phase 3
Compressing tables 1 and 2...
Finished compressing tables 1 and 2 in 25.56 seconds
Table 1 now has 3429416169 / 4294967296 entries ( 79.85% ).
Compressing tables 2 and 3...
Finished compressing tables 2 and 3 in 25.45 seconds
Table 2 now has 3439880033 / 4294956986 entries ( 80.09% ).
Compressing tables 3 and 4...
Finished compressing tables 3 and 4 in 25.73 seconds
Table 3 now has 3466079431 / 4294967296 entries ( 80.70% ).
Compressing tables 4 and 5...
Finished compressing tables 4 and 5 in 26.33 seconds
Table 4 now has 3532879443 / 4294844925 entries ( 82.26% ).
Compressing tables 5 and 6...
Finished compressing tables 5 and 6 in 27.50 seconds
Table 5 now has 3713392423 / 4294689248 entries ( 86.46% ).
Compressing tables 6 and 7...
Finished compressing tables 6 and 7 in 17.94 seconds
Table 6 now has 4294324297 / 4294324297 entries ( 100.00% ).
Finished Phase 3 in 148.52 seconds.
Running Phase 4
Writing P7.
Finished writing P7 in 0.26 seconds.
Writing C1 table.
Finished writing C1 table in 0.00 seconds.
Writing C2 table.
Finished writing C2 table in 0.00 seconds.
Writing C3 table.
Finished writing C3 table in 0.14 seconds.
Finished Phase 4 in 0.40 seconds.
Finished plotting in 335.97 seconds (5.60 minutes).

Test3

Running Phase 1
Generating F1...
Finished F1 generation in 0.76 seconds.
Sorting F1...
Finished F1 sort in 4.12 seconds.
Forward propagating to table 2...
Pairing L/R groups...
Finished pairing L/R groups in 4.8520 seconds. Created 4294967296 pairs.
Average of 236.1406 pairs per group.
Computing Fx...
Finished computing Fx in 5.4970 seconds.
Sorting entries...
Finished sorting in 7.88 seconds.
Finished forward propagating table 2 in 18.44 seconds.
Forward propagating to table 3...
Pairing L/R groups...
Finished pairing L/R groups in 4.8370 seconds. Created 4294967296 pairs.
Average of 236.1406 pairs per group.
Computing Fx...
Finished computing Fx in 5.5230 seconds.
Sorting entries...
Finished sorting in 7.95 seconds.
Finished forward propagating table 3 in 18.52 seconds.
Forward propagating to table 4...
Pairing L/R groups...
Finished pairing L/R groups in 4.8530 seconds. Created 4294934585 pairs.
Average of 236.1388 pairs per group.
Computing Fx...
Finished computing Fx in 5.7140 seconds.
Sorting entries...
Finished sorting in 7.87 seconds.
Finished forward propagating table 4 in 18.66 seconds.
Forward propagating to table 5...
Pairing L/R groups...
Finished pairing L/R groups in 4.8340 seconds. Created 4294895120 pairs.
Average of 236.1367 pairs per group.
Computing Fx...
Finished computing Fx in 5.7130 seconds.
Sorting entries...
Finished sorting in 7.81 seconds.
Finished forward propagating table 5 in 18.56 seconds.
Forward propagating to table 6...
Pairing L/R groups...
Finished pairing L/R groups in 4.8350 seconds. Created 4294945441 pairs.
Average of 236.1394 pairs per group.
Computing Fx...
Finished computing Fx in 5.6260 seconds.
Sorting entries...
Finished sorting in 7.79 seconds.
Finished forward propagating table 6 in 18.45 seconds.
Forward propagating to table 7...
Pairing L/R groups...
Finished pairing L/R groups in 4.8480 seconds. Created 4294905632 pairs.
Average of 236.1372 pairs per group.
Computing Fx...
Finished computing Fx in 5.4630 seconds.
Finished forward propagating table 7 in 10.52 seconds.
Finished Phase 1 in 108.05 seconds.
Running Phase 2
Prunning table 6...
Finished prunning table 6 in 0.26 seconds.
Prunning table 5...
Finished prunning table 5 in 7.57 seconds.
Prunning table 4...
Finished prunning table 4 in 7.18 seconds.
Prunning table 3...
Finished prunning table 3 in 7.05 seconds.
Prunning table 2...
Finished prunning table 2 in 7.01 seconds.
Finished Phase 2 in 29.26 seconds.
Running Phase 3
Compressing tables 1 and 2...
Finished compressing tables 1 and 2 in 12.32 seconds
Table 1 now has 3429390227 / 4294967296 entries ( 79.85% ).
Compressing tables 2 and 3...
Finished compressing tables 2 and 3 in 12.48 seconds
Table 2 now has 3439904327 / 4294967296 entries ( 80.09% ).
Compressing tables 3 and 4...
Finished compressing tables 3 and 4 in 12.62 seconds
Table 3 now has 3466102262 / 4294934585 entries ( 80.70% ).
Compressing tables 4 and 5...
Finished compressing tables 4 and 5 in 12.80 seconds
Table 4 now has 3532943840 / 4294895120 entries ( 82.26% ).
Compressing tables 5 and 6...
Finished compressing tables 5 and 6 in 13.61 seconds
Table 5 now has 3713661136 / 4294945441 entries ( 86.47% ).
Compressing tables 6 and 7...
Finished compressing tables 6 and 7 in 16.06 seconds
Table 6 now has 4294905632 / 4294905632 entries ( 100.00% ).
Finished Phase 3 in 79.90 seconds.
Running Phase 4
Writing P7.
Finished writing P7 in 0.27 seconds.
Writing C1 table.
Finished writing C1 table in 0.00 seconds.
Writing C2 table.
Finished writing C2 table in 0.00 seconds.
Writing C3 table.
Finished writing C3 table in 0.17 seconds.
Finished Phase 4 in 0.45 seconds.
Finished plotting in 217.65 seconds (3.63 minutes).

[paspy]

Hey,
I talked with someone who works at AMD, and he said the issue is that Zen2 has a CCD architecture issue with latency between them. It appears AMD has fixed it on the Zen 3 by giving a larger L3 Cache.

[zhenglcc]

thanks ，I have considered the difference between the architecture of zen2 and zen3，maybe I should try epyc 7003 series

[xorinox]

Hello, I find the same issue applies to dual 64 core zen3 CPU. (128 cores, 256 threads)

1. 256 threads, bios defaults but NPS4
   Finished plotting in 358.26 seconds (5.97 minutes).

2. 196 threads, bios defaults but NPS4
   Finished plotting in 316.14 seconds (5.27 minutes).

3. 128 threads, bios defaults but NPS4
   Finished plotting in 352.90 seconds (5.88 minutes).

4. 128 threads, bios defaults but NPS4, CCD4 (basically disabled 1/2 of all CPU cores!)
   Finished plotting in 220.05 seconds (3.67 minutes).

H, I have the log files, if you need so more details.

Possible some of this is going away with: https://www.phoronix.com/news/x86-urgent-For-Linux-6.1-rc2

[xorinox]

256 threads, bios defaults but NPS4

`Creating 1 plots:
Output path : /chia/scratch/disk01/
Thread count : 256
Warm start enabled : true
Farmer public key :
Pool contract address :

System Memory: 494/503 GiB.
Memory required: 416 GiB.
Allocating buffers.
Generating plot 1 / 1: 8830eb8d072003a726319776d24a8f6f0d1545d5f177317c3f4d31ecdd6a8f46

Running Phase 1
Generating F1...
Finished F1 generation in 1.03 seconds.
Sorting F1...
Finished F1 sort in 4.28 seconds.
Forward propagating to table 2...
Pairing L/R groups...
Finished pairing L/R groups in 2.9190 seconds. Created 4294959550 pairs.
Average of 236.1402 pairs per group.
Computing Fx...
Finished computing Fx in 3.0490 seconds.
Sorting entries...
Finished sorting in 18.26 seconds.
Finished forward propagating table 2 in 24.53 seconds.
Forward propagating to table 3...
Pairing L/R groups...
Finished pairing L/R groups in 2.8880 seconds. Created 4294939098 pairs.
Average of 236.1391 pairs per group.
Computing Fx...
Finished computing Fx in 3.2660 seconds.
Sorting entries...
Finished sorting in 18.20 seconds.
Finished forward propagating table 3 in 24.66 seconds.
Forward propagating to table 4...
Pairing L/R groups...
Finished pairing L/R groups in 2.9680 seconds. Created 4294865647 pairs.
Average of 236.1350 pairs per group.
Computing Fx...
Finished computing Fx in 3.3240 seconds.
Sorting entries...
Finished sorting in 17.89 seconds.
Finished forward propagating table 4 in 24.49 seconds.
Forward propagating to table 5...
Pairing L/R groups...
Finished pairing L/R groups in 2.9280 seconds. Created 4294842649 pairs.
Average of 236.1338 pairs per group.
Computing Fx...
Finished computing Fx in 3.3350 seconds.
Sorting entries...
Finished sorting in 17.98 seconds.
Finished forward propagating table 5 in 24.54 seconds.
Forward propagating to table 6...
Pairing L/R groups...
Finished pairing L/R groups in 2.9460 seconds. Created 4294651814 pairs.
Average of 236.1233 pairs per group.
Computing Fx...
Finished computing Fx in 3.2810 seconds.
Sorting entries...
Finished sorting in 18.44 seconds.
Finished forward propagating table 6 in 24.99 seconds.
Forward propagating to table 7...
Pairing L/R groups...
Finished pairing L/R groups in 2.8720 seconds. Created 4294358861 pairs.
Average of 236.1072 pairs per group.
Computing Fx...
Finished computing Fx in 2.7260 seconds.
Finished forward propagating table 7 in 5.88 seconds.
Finished Phase 1 in 134.41 seconds.
Running Phase 2
Prunning table 6...
Finished prunning table 6 in 0.34 seconds.
Prunning table 5...
Finished prunning table 5 in 9.68 seconds.
Prunning table 4...
Finished prunning table 4 in 9.26 seconds.
Prunning table 3...
Finished prunning table 3 in 10.97 seconds.
Prunning table 2...
Finished prunning table 2 in 9.99 seconds.
Finished Phase 2 in 40.51 seconds.
Running Phase 3
Compressing tables 1 and 2...
Finished compressing tables 1 and 2 in 29.30 seconds
Table 1 now has 3429389223 / 4294959550 entries ( 79.85% ).
Compressing tables 2 and 3...
Finished compressing tables 2 and 3 in 29.79 seconds
Table 2 now has 3439878279 / 4294939098 entries ( 80.09% ).
Compressing tables 3 and 4...
Finished compressing tables 3 and 4 in 29.89 seconds
Table 3 now has 3466016715 / 4294865647 entries ( 80.70% ).
Compressing tables 4 and 5...
Finished compressing tables 4 and 5 in 30.51 seconds
Table 4 now has 3532827105 / 4294842649 entries ( 82.26% ).
Compressing tables 5 and 6...
Finished compressing tables 5 and 6 in 32.16 seconds
Table 5 now has 3713324365 / 4294651814 entries ( 86.46% ).
Compressing tables 6 and 7...
Finished compressing tables 6 and 7 in 22.46 seconds
Table 6 now has 4294358859 / 4294358861 entries ( 100.00% ).
Finished Phase 3 in 174.10 seconds.
Running Phase 4
Writing P7.
Finished writing P7 in 0.32 seconds.
Writing C1 table.
Finished writing C1 table in 0.00 seconds.
Writing C2 table.
Finished writing C2 table in 0.00 seconds.
Writing C3 table.
Finished writing C3 table in 0.23 seconds.
Finished Phase 4 in 0.55 seconds.
Writing final plot tables to disk

Plot /chia/scratch/disk01/plot-k32-2022-10-27-22-40-8830eb8d072003a726319776d24a8f6f0d1545d5f177317c3f4d31ecdd6a8f46.plot finished writing to disk:
Table 1 pointer : 4096 ( 0x0000000000001000 )
Table 2 pointer : 14839488512 ( 0x000000037480a000 )
Table 3 pointer : 28822401024 ( 0x00000006b5f2f000 )
Table 4 pointer : 42911559680 ( 0x00000009fdba7000 )
Table 5 pointer : 57272303616 ( 0x0000000d55b20000 )
Table 6 pointer : 72366755840 ( 0x00000010d9651000 )
Table 7 pointer : 89823076352 ( 0x00000014e9df6000 )
C1 table pointer : 107537309696 ( 0x0000001909b92000 )
C2 table pointer : 107539030016 ( 0x0000001909d36000 )
C3 table pointer : 107539034112 ( 0x0000001909d37000 )

Finished writing tables to disk in 8.69 seconds.
Finished plotting in 358.26 seconds (5.97 minutes).`