replacing the python utils with rust code?

[rustrust]

Do you mind going ahead with replacing the python utilities with rust code?

The python utilis scripts are segfaulting (!!) on my system (possibly due to out of memory). Even the first import alone causes a segfault. I've been careful to create a fresh venv using python 3.6 and installed torch torchvision and transformers only in the venv on an up-to-date linux system running cuda 10.1 (several different versions of python actually including 3.7 and 3.8).

This line alone causes a segfault:

from transformers import BERT_PRETRAINED_CONFIG_ARCHIVE_MAP

[guillaume-be]

Hello,

I had a look at the serde-pickle crate (https://docs.rs/serde-pickle/0.6.0/serde_pickle/) and I am afraid a Rust native loading of the Pytorch model files would require a significant amount of work. Here are the issues I have identified following a very brief investigation:

1. The toch.load() method does far more than just pickle.load(), as can be seen at https://pytorch.org/docs/stable/_modules/torch/serialization.html#load.
2. The serde-pickle crate only supports Python built-in types. I would have to look more in details but I believe Torch pickles the model parameter tensors directly

This conversion requirement is linked to the tch-rs crate. I will share this requirement with the crate author to understand if a Rust-native conversion would be possible (see LaurentMazare/tch-rs#171).

However, I would like to support in getting you started using the library. Please note that the download scripts have all been tested and are running in a controlled CI Linux environment. The CI setup can be seen at https://travis-ci.com/guillaume-be/rust-bert/jobs/295652126

Regarding the segfaults, could you post the error message you are getting? The BERT_PRETRAINED_CONFIG_ARCHIVE_MAP is just a small Python dictionary with about 20 string key/value pairs. This should not cause any system to Segfault (see
https://github.com/huggingface/transformers/blob/master/src/transformers/configuration_bert.py). The Transformers library is used by thousands and the scripts proposed should be working fine - I am happy to support wherever I can to get you started.

[guillaume-be]

Hello,

Sorry for the delayed response on this as I was working on a more convenient alternative for a pure Rust utilization. I have worked with the Transformers' authors (Thank you to @julien-c and the Hugging Face team for the support), to offer Rust-compatible model weights for direct download in the library. This means that users no longer need to use Python if they want to load a set of pre-trained weights.

Please refer to the updated documentation for examples on how to use the library.

[rustrust]

it works brilliantly thanks. for other people who come across this bug, you need to run (and make sure that you have git pull origin master if this hasn't shipped yet) :

cargo run --example download_all_dependencies