Ca not convert model to use it with spago

[AlexSnet]

Hello, i'm trying to convert a model as it described in README.md in Import pretrained model section.

Step 1 (building):

Just build binary as usual: go build -o hugging_face_importer ./cmd/huggingfaceimporter/main.go

Step 2 (converting models):

Yes, directory ~/.spago is already created.

I found a model at hugginface - DeepPavlov/rubert-base-cased-conversational and then I try to run huggingface importer like this: ./hugging_face_importer --model=DeepPavlov/rubert-base-cased-conversational --repo=~/.spago

After many lines of logging that everything is OK I've got a crash:

Reading encoder.layer.10.attention.output.LayerNorm.bias.... ok
2020/06/14 14:51:06 Convert word/positional/type embeddings...
panic: runtime error: slice bounds out of range [:768] with capacity 0

goroutine 1 [running]:
github.com/nlpodyssey/spago/pkg/nlp/transformers/bert.assignToParamsList(0x0, 0x0, 0x0, 0xc000172000, 0x200, 0x200, 0x200, 0x300)
        /Users/alex/Downloads/spago-master/pkg/nlp/transformers/bert/converter.go:225 +0x112
github.com/nlpodyssey/spago/pkg/nlp/transformers/bert.(*huggingFacePreTrainedConverter).convertEmbeddings(0xc080285dd8, 0xc000114600)
        /Users/alex/Downloads/spago-master/pkg/nlp/transformers/bert/converter.go:203 +0xaf
github.com/nlpodyssey/spago/pkg/nlp/transformers/bert.(*huggingFacePreTrainedConverter).convert(0xc080285dd8, 0x2, 0x2)
        /Users/alex/Downloads/spago-master/pkg/nlp/transformers/bert/converter.go:101 +0x17e
github.com/nlpodyssey/spago/pkg/nlp/transformers/bert.ConvertHuggingFacePreTrained(0xc00002e580, 0x3e, 0x2, 0xc00002e580)
        /Users/alex/Downloads/spago-master/pkg/nlp/transformers/bert/converter.go:65 +0x5b9
main.main()
        /Users/alex/Downloads/spago-master/cmd/huggingfaceimporter/main.go:48 +0x385

Can I help with anything else?

[matteo-grella]

Thank @AlexSnet for your report.

Fixed. The general problem is that there is no guarantee of uniformity in the names of parameters in various PyTorch serialized models.

For instance, in the DeepPavlov/rubert-base-cased-conversational model, instead of the expected parameter bert.embeddings.word_embeddings.weight we only find embeddings.word_embeddings.weight without the prefix bert., thus causing the crash.

Since this won't be the only case, this issue would require more attention and effort to keep the import code clean, probably using regular expressions instead of a trivial normalization as I have now set out.

I also noticed that while the conversion process is now going well, it reports some WARNINGS: for the model you'd like to use and all the other DeepPavalov models stored by Hugging Face, only the BERT encoder has been serialized. It appears that with these Russian models alone, masked tokens cannot be predicted; You can't predict the "correctness" of the next sentence; You can't do question-answering, etc.

However, there is a pooler layer, which is positive. I need to get into DeepPavalov's code and documentation to figure out how to use these models. In the meantime, did you have anything particular in mind you wanted to try on Russian?