Bugfix: missing hparam type_vocab_size
#32
Conversation
|
Here is some of the model upon bert DO NOT use
|
|
Hi, This seems like a good change. But surely bert.cpp:358 also needs to be changed? Where the hparams are read from the model file |
sure, I will add it. |
|
|
||
| model.layers.resize(n_layer); | ||
|
|
||
| model.word_embeddings = ggml_new_tensor_2d(ctx, wtype, n_embd, n_vocab); | ||
| model.token_type_embeddings = ggml_new_tensor_2d(ctx, wtype, n_embd, 2); | ||
| model.token_type_embeddings = ggml_new_tensor_2d(ctx, wtype, n_embd, n_vocab_size); |
There was a problem hiding this comment.
@skeskinen
here is a change since the tensor is related to n_vocab_size.
for many case, it is 2, but not a const.
There was a problem hiding this comment.
| @@ -23,6 +23,7 @@ struct bert_hparams | |||
| int32_t n_intermediate = 1536; | |||
| int32_t n_head = 12; | |||
| int32_t n_layer = 6; | |||
| int32_t n_vocab_size = 2; | |||
There was a problem hiding this comment.
Here is a break change since new field.
I believe some others will happen in future.
So we may try to shift into GGUF.
| @@ -364,6 +365,7 @@ struct bert_ctx * bert_load_from_file(const char *fname) | |||
| fin.read((char *)&hparams.n_intermediate, sizeof(hparams.n_intermediate)); | |||
| fin.read((char *)&hparams.n_head, sizeof(hparams.n_head)); | |||
| fin.read((char *)&hparams.n_layer, sizeof(hparams.n_layer)); | |||
| fin.read((char *)&hparams.n_vocab_size, sizeof(hparams.n_vocab_size)); | |||
type_vocab_sizeis also a hparam (can not use const as 2).