[Bug] chia plotters bladebit cudaplot command produces invalid plots

[BrandtH22]

What happened?

Reported by Madderhat in discord (https://discord.com/channels/1034523881404370984/1034870571864948777/1145892307858624514)

User has tested both the binary command /opt/chia/bladebit/./bladebit_cuda and the integrated command chia plotters bladebit cudaplot for bladebit finding that the integrated command is producing invalid plots.

This user tested with a clean install of 2.0.0 on a clean install of Ubuntu 22.04 and used C7 compression.

This is the result when running a plot check on plots created with the integrated command:

Checking 100 random proofs with seed 0x934585ebd4b70db0f44d0da9d511e6b8a77dd5089199669ed4175646d7c56cd3...
Plot compression level: 7
 10%...
 20%...
 30%...
 40%...
 50%...
 60%...
 70%...
 80%...
 90%...
0 / 100 (0.00%) valid proofs found.
double free or corruption (!prev)
Aborted (core dumped)

Additional information from MadderHat:

I stopped my plotter and did a check on the 1100 plots still there. All 1100 of them failed the check with either "Invalid size for deltas" or "Destination buffer is too small"
Those 1100 plots were created with chia plotters bladebit cudaplot ...
For s&g's I created another plot using /opt/chia/bladebit/./bladebit_cuda ... with all other options the same.
The check on that plot successfully returned 81/100 proofs, so I created 9 more with similar check results. I moved those 10 plots to my farmer and unmounted the drives that were already connected. The harvester didn't have any issue with those 10 plots. No errors in the log. Plot check returned successful results.
Unless I'm missing something, the only conclusion I can draw is that chia plotters bladebit cudaplot ...  doesn't produce valid plots

also, if it's useful information, using the binary directly is ~3 seconds faster to complete a plot (100s vs. 103s on average)

Version

2.0.0

What platform are you using?

Linux

What ui mode are you using?

CLI

Relevant log output

na

[wjblanke]

Can you reach out to this guy and see exactly what command line he is running to generate and check these plots and if it is always reproducible on his system. Also see what his system configuration is GPU etc.

[dkackman]

I think I am seeing the same. (see #16233)

Using chia 2.0.0 released version:

plotting command:

chia plotters bladebit cudaplot -d /mnt/plots --compress 4 -f ae59fa02a1b3400e2ba224227d36c39f9a811d9cda682c9f5f13b9e4cd164111bac2478db9bc0e2a3ac6ea074be5f7b3 -c xch1q69typkg2y6yts02klctknuepkc8nuthert6gltx94qwtc0sratss7yg3q -n 4

plotter is
Lenovo S30 Xeon E5-2650L 256GB RAM DDR3
NVIDIA RTX 3060
Driver Version: 535.86.05 CUDA Version: 12.2
Ubuntu 22.04

plot check returns:

2023-08-30T20:21:33.250  chia.plotting.check_plots        : ERROR    <class 'RuntimeError'>: Destination buffer is too small error in getting challenge qualities for plot /mnt/plots04/plot-k32-c05-2023-08-30-20-54-5e39d051ff4a0642b43d5265d55baa803e0b94cb05301c4ffb86271666592a29.plot
2023-08-30T20:21:33.250  chia.plotting.check_plots        : ERROR       Proofs 0 / 10, 0.0 Filepath: /mnt/plots04/plot-k32-c05-2023-08-30-20-54-5e39d051ff4a0642b43d5265d55baa803e0b94cb05301c4ffb86271666592a29.plot
2023-08-30T20:21:33.251  chia.plotting.check_plots        : INFO
2023-08-30T20:21:33.251  chia.plotting.check_plots        : INFO
2023-08-30T20:21:33.251  chia.plotting.check_plots        : INFO     Summary
2023-08-30T20:21:33.251  chia.plotting.check_plots        : INFO     Found 0 valid plots, total size 0.00000 TiB
2023-08-30T20:21:33.251  chia.plotting.check_plots        : WARNING  1 invalid plots found:
2023-08-30T20:21:33.251  chia.plotting.check_plots        : WARNING      1 bad plots:
2023-08-30T20:21:33.251  chia.plotting.check_plots        : WARNING  /mnt/plots04/plot-k32-c05-2023-08-30-20-54-5e39d051ff4a0642b43d5265d55baa803e0b94cb05301c4ffb86271666592a29.plot

[ales]

Having the same issue. Using Bladebit CUDA through GUI.

Fresh install

Debian 12
X299X AORUS MASTER i9-10980XE 256GB RAM
GTX 1070 8GB

[BrandtH22]

Can you reach out to this guy and see exactly what command line he is running to generate and check these plots and if it is always reproducible on his system. Also see what his system configuration is GPU etc.

Here is MadderHats response:

Reproducable every time

Plotting Commands:
chia plotters bladebit cudaplot -f <farmer> -c <plotnft contract> -r 24 -n 1 -v --compress 7 -d /mnt/md0/
/opt/chia/bladebit/./bladebit_cuda -f <farmer> -c <plotnft contract> -t 24 -n 1 -z 7 cudaplot /mnt/md0/

Plot Check Commands:
chia plots check -n 100 -g c07-2023-08-29
/opt/chia/bladebit/./bladebit_cuda check /mnt/md0/plot-k32-c07-2023-08-29 ...

System:
  Host: madderhat-ubu Kernel: 6.2.0-31-generic x86_64 bits: 64 Console: pty pts/6
    Distro: Ubuntu 22.04.3 LTS (Jammy Jellyfish)
Machine:
  Type: Desktop Mobo: ASUSTeK model: PRIME TRX40-PRO S v: Rev 1.xx serial: <superuser required>
    UEFI: American Megatrends v: 1303 date: 11/11/2020
CPU:
  Info: 24-core AMD Ryzen Threadripper 3960X [MT MCP] speed (MHz): avg: 2375 min/max: 2200/3800
Graphics:
  Device-1: NVIDIA GA102GL [RTX A4500] driver: nvidia v: 535.86.05
  Display: server: X.org v: 1.21.1.4 with: Xwayland v: 22.1.1 driver: X: loaded: nvidia
    unloaded: fbdev,modesetting,nouveau,vesa gpu: nvidia tty: 256x107
  Message: GL data unavailable in console. Try -G --display
Network:
  Device-1: Intel I211 Gigabit Network driver: igb
Drives:
  Local Storage: total: 185.54 TiB used: 41.98 TiB (22.6%)
Info:
  Processes: 709 Uptime: 10h 13m Memory: 251.54 GiB used: 224.72 GiB (89.3%) Init: systemd
  runlevel: 5 Shell: Bash inxi: 3.3.13

[ales]

Workaround found! The problem for me is plotting through GUI. From CLI I could make a valid plot, yay!

GUI calls this command, that produces invalid plot

/opt/chia/resources/app.asar.unpacked/daemon/bladebit/bladebit_cuda --threads 0 --count 1 --farmer-key {} --pool-contract {} --compress 1 cudaplot --device 0 --no-direct-downloads /mnt/nvme

command that made 2 valid plots just fine

/opt/chia/resources/app.asar.unpacked/daemon/bladebit/bladebit_cuda -f {} -c {} -n 2 --compress 1 cudaplot /mnt/nvme/

Debian 12
256GB cudaplot

P.S. on Windows 11 it works through GUI fine, except its 2-4 times slower :)

[mattdallaire]

/opt/chia/resources/app.asar.unpacked/daemon/bladebit/bladebit_cuda -f {} -c {} -n 2 --compress 1 cudaplot /mnt/nvme/

Thanks @ales! Can confirm this workaround works on Ubuntu 22 with --compress 7

[ChrisGute]

Just validated workaround by @ales as well by calling /opt/chia/resources/app.asar.unpacked/daemon/bladebit/bladebit_cuda directly. It now generating plots that pass chia plots check.

/opt/chia/resources/app.asar.unpacked/daemon/bladebit/bladebit_cuda \
--threads 0 \
--count 1 \
--farmer-key ${KEY_FARMER} \
--pool-contract ${ID_CONTRACT} \
--verbose \
--compress 7 \
cudaplot \
${DIR_TEMP_PLOT}

Original command that was generating plots that where failing chia plots check

chia plotters bladebit cudaplot \
    --threads 0 \
    --count 1 \
    --farmerkey ${KEY_FARMER} \
    --contract ${ID_CONTRACT} \
    --verbose \
    --final_dir ${DIR_TEMP_PLOT} \
    --compress 7 \
    --device 0

[emlowe]

This is fixed in 2.0.1