finetune: fix no KV cache

[ngxson]

Overview

llama-finetune is broken on master because training code doesn't use KV cache, but the graph still try to access it. This results in this error:

ggml.c:7287: GGML_ASSERT(!node->view_src || node->op == GGML_OP_CPY || ...) failed

This change makes ubatch to hold the entire seq and apply a mask over it.

llama-finetune -m stories260K-f32.gguf --file README.md -c 128 -b 128 -ub 128 -ngl 0 -epochs 1 -val-split 0.0 -o /tmp/ft.gguf 

....

0.01.207.298 I epoch 0 lr=1e-050000082 loss=7.87451±0.15058 acc=6.57±0.26% t=00:00:01 ETA=00:00:00 
0.01.222.294 I epoch 0 lr=1e-050000082 loss=7.86201±0.14906 acc=6.60±0.26% t=00:00:01 ETA=00:00:00 
0.01.237.362 I epoch 0 lr=1e-050000082 loss=7.84961±0.14758 acc=6.62±0.25% t=00:00:01 ETA=00:00:00 
0.01.252.818 I epoch 0 lr=1e-050000082 loss=7.81554±0.14956 acc=6.73±0.25% t=00:00:01 ETA=00:00:00 
0.01.269.276 I epoch 0 lr=1e-050000082 loss=7.78222±0.15132 acc=6.85±0.25% t=00:00:01 ETA=00:00:00 
0.01.284.785 I epoch 0 lr=1e-050000082 loss=7.75408±0.15199 acc=7.00±0.26% t=00:00:01 ETA=00:00:00 
0.01.300.457 I epoch 0 lr=1e-050000082 loss=7.72652±0.15257 acc=7.15±0.26% t=00:00:01 ETA=00:00:00 
0.01.316.689 I epoch 0 lr=1e-050000082 loss=7.71430±0.15114 acc=7.14±0.25% t=00:00:01 ETA=00:00:00 
0.01.332.688 I epoch 0 lr=1e-050000082 loss=7.70219±0.14976 acc=7.13±0.25% t=00:00:01 ETA=00:00:00 
train: [███████▉] data=0000082/0000082 loss=7.68039±0.14952 acc=7.10±0.25% t=00:00:01 ETA=00:00:00

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: code is written by AI, manually validated and clean up

[ngxson]

@ggerganov could you have a quick look? I'm planning to play a bit with finetuning and potentially add back the LoRA training example

[ggerganov]

because training code doesn't use KV cache, but the graph still try to access it.

Hm, this does not sound right. When the example was working, we basically were creating the cache just to be able to reuse all the machinery of the KQ mask without changes. Basically the changes related to cparams.training and kq_mask_train should not be needed.

[ggerganov]

If the cache is not created now for some reason, it's simpler to just create it. Then you don't need the extra logic for if (training).

[ggerganov]

Seems like the code hasn't been updated since the introduction of GGML_OP_SET_ROWS. This change on master should fix it:

diff --git a/ggml/src/ggml.c b/ggml/src/ggml.c
index 8dc0945084..2d5fdb7c10 100644
--- a/ggml/src/ggml.c
+++ b/ggml/src/ggml.c
@@ -7335,7 +7335,7 @@ void ggml_build_backward_expand(
         }
 
         // inplace operations are currently not supported
-        GGML_ASSERT(!node->view_src || node->op == GGML_OP_CPY || node->op == GGML_OP_VIEW ||
+        GGML_ASSERT(!node->view_src || node->op == GGML_OP_CPY || node->op == GGML_OP_SET_ROWS || node->op == GGML_OP_VIEW ||
             node->op == GGML_OP_RESHAPE || node->op == GGML_OP_PERMUTE || node->op == GGML_OP_TRANSPOSE);
 
         const size_t ihash = ggml_hash_find(&cgraph->visited_hash_set, node);

[ngxson]

thanks, confirm that it works now