-
Notifications
You must be signed in to change notification settings - Fork 211
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BERT support #331
Comments
This looks really exciting. Running models from https://github.com/huggingface/transformers without the TF hassle would be neat. Happy to help with a test case if needed. |
@bminixhofer I tried making a test-case for this one https://github.com/onnx/models/tree/master/text/machine_comprehension/bert-squad, but I have little patience for debugging python notebooks that got broken months ago... If your offer of helping with test-cases still stand, please do :) |
[x] NonZeroWe now have a working test case in incorporated mode. Performance is of course awful. |
You can use the following script to generate a BERT model in ONNX format that uses the ConstantOfShape operator.
Running the script will produce 2 files:
Both use the ConstantOfShape operator. Let me know if you have trouble with this. I could upload the ONNX model somewhere but they are > 400 MB. |
Hi, just a heads up: pytorch-pretrained-bert has long been deprecated in favor of huggingface/transformers. Maybe you can try updating to huggingface/transformers? The networks I exported with it did apparently not have this operator. @kali You could also try changing the |
Wondering if there is any progress here. Perhaps it is limited by how well (or easily) huggingface is exporting to ONNX. e.g. |
There have been progress.
Then it should be a matter of just fixing a handful of operators. We'll get there. |
Hello, I was trying to load an ONNX file of a FlauBERT model (which is similar to BERT). and I ran into the following error:
|
Politely ping... |
@Yevgnen thanks for the ping. Let's try to revive this, and see where we are. Keeping up with huggingface models releases is a bit of challenge for me :) Does anybody knows the current state of affairs? What is their latest release format? Does it take the same form as what @bminixhofer put in the tests in 2020 in onnx/transformer-mlm ? What are we ultimately trying to achieve here ? My thinking right now is we want a working rust example to be put in examples alongside the others. I'm just not sure which model, and what kind of workflow (fully known input sizes before optim ? partially known input sizes ?). Could anyone help me define this ? @llenotre can you provide me or tell me where to find an instance of Flaubert so i can have a look at the missing op ? |
Hi, thanks for your quick response. I'm quite new to Rust and ONNX. It would be great to have examples related to the BERT models. At least related early model architectures does not change for last two years. What are we missing to get it work? I got similar error like @llenotre when trying to load an ONNX file which uses I think models like BERT or Albert would be the simple and good to start with. The typical inputs are |
I'm a bit surprised that this is not covered by the |
I tried slightly adapted code from the Codeimport torch
import numpy as np
from transformers import AutoTokenizer, AutoModelForMaskedLM
import onnxruntime
model_name = "distilbert-base-uncased"
model_path = "model.onnx"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForMaskedLM.from_pretrained(model_name)
text = "tract is a machine " + tokenizer.mask_token + " library."
print("MODEL:", model_name)
print("TEXT:", text)
encoded = tokenizer.encode_plus(text)
mask_idx = encoded["input_ids"].index(tokenizer.mask_token_id)
input_ids = torch.tensor([encoded["input_ids"]], dtype=torch.long)
attention_mask = torch.tensor([encoded["attention_mask"]], dtype=torch.long)
torch.onnx.export(
model,
(input_ids, attention_mask),
model_path,
input_names=["input_ids", "attention_mask"],
output_names=["output"],
dynamic_axes={
"input_ids": {0: "batch", 1: "seq"},
"attention_mask": {0: "batch", 1: "seq"},
"output": {0: "batch", 1: "seq"},
},
opset_version=13,
)
sess = onnxruntime.InferenceSession(model_path)
outputs = sess.run(
None, {"input_ids": input_ids.numpy(), "attention_mask": attention_mask.numpy()}
)[0]
np.savez_compressed(
open("io.npz", "wb"),
input_ids=input_ids.numpy(),
attention_mask=attention_mask.numpy(),
output=outputs,
) (note that I had to add and: This works with I hope this helps in the discussion, from my POV the models which were mentioned here work. Maybe it would be good to add an example with code similar to the one above + inference in Rust. |
Thanks for you example. I tried to get the model run in Rust code, but the result did not seem to equal to the Python output. Do you have any suggestions? Python code # -*- coding: utf-8 -*-
import torch
import numpy as np
from transformers import AutoTokenizer, AutoModel
import onnxruntime
model_name = "voidful/albert_chinese_tiny"
model_path = "model.onnx"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)
model.eval()
text = "hey there!"
print("MODEL:", model_name)
print("TEXT:", text)
encoded = tokenizer(text, return_tensors="pt")
print(encoded)
input_ids = encoded["input_ids"]
attention_mask = encoded["attention_mask"]
token_type_ids = encoded["token_type_ids"]
torch.onnx.export(
model,
(input_ids, attention_mask, token_type_ids),
model_path,
input_names=["input_ids", "attention_mask", "token_type_ids"],
output_names=["output"],
dynamic_axes={
"input_ids": {0: "batch", 1: "seq"},
"attention_mask": {0: "batch", 1: "seq"},
"token_type_ids": {0: "batch", 1: "seq"},
"output": {0: "batch", 1: "seq"},
},
opset_version=13,
)
sess = onnxruntime.InferenceSession(model_path)
outputs = sess.run(
None,
{
"input_ids": input_ids.numpy(),
"attention_mask": attention_mask.numpy(),
"token_type_ids": token_type_ids.numpy(),
},
)
print(outputs[0])
np.savez_compressed(
open("io.npz", "wb"),
input_ids=input_ids.numpy(),
attention_mask=attention_mask.numpy(),
token_type_ids=token_type_ids.numpy(),
output=outputs[0],
) Rust Code use std::{
path::{Path, PathBuf},
str::FromStr,
};
use tokenizers::tokenizer::{Result, Tokenizer};
use tract_onnx::prelude::*;
fn main() -> Result<()> {
// model here: https://huggingface.co/voidful/albert_chinese_tiny/tree/main
// the model does have a tokenizer.json so I save the downloaded tokenizer manually in Python
let model_dir = PathBuf::from_str("./albert_chinese_tiny/")?;
let tokenizer = Tokenizer::from_file(Path::join(&model_dir, "tokenizer.json"))?;
let text = "hey there!";
let tokenizer_output = tokenizer.encode(text, true)?;
println!("{:?}", tokenizer_output);
let input_ids = tokenizer_output.get_ids();
let attention_mask = tokenizer_output.get_attention_mask();
let token_type_ids = tokenizer_output.get_type_ids();
let length = input_ids.len();
let input_ids: Tensor = tract_ndarray::Array2::from_shape_vec(
(1, length),
input_ids.iter().map(|&x| x as i64).collect(),
)?
.into();
let attention_mask: Tensor = tract_ndarray::Array2::from_shape_vec(
(1, length),
attention_mask.iter().map(|&x| x as i64).collect(),
)?
.into();
let token_type_ids: Tensor = tract_ndarray::Array2::from_shape_vec(
(1, length),
token_type_ids.iter().map(|&x| x as i64).collect(),
)?
.into();
let model = tract_onnx::onnx()
.model_for_path(Path::join(&model_dir, "model.onnx"))?
.with_input_fact(
0,
InferenceFact::dt_shape(i64::datum_type(), tvec!(1, length)),
)?
.with_input_fact(
1,
InferenceFact::dt_shape(i64::datum_type(), tvec!(1, length)),
)?
.with_input_fact(
2,
InferenceFact::dt_shape(i64::datum_type(), tvec!(1, length)),
)?
.into_optimized()?
.into_runnable()?;
let result = model.run(tvec!(input_ids, attention_mask, token_type_ids))?;
println!("{:?}", result);
Ok(())
} Python input:
Python output:
Rust input:
Rust output:
|
tract output looks really wrong: it does not even have the correct shape. As you do not set the model output in your tract code, tract makes a guess, and is probably just wrong. You can setup the output node for your model with setOutputName() |
@kali Thanks for the hint. I'm new to Rust and try to follow |
Ha sorry, no, you were right, forget my comment. I misread the python output, did not see it was truncated. So I don't know what's wrong. Can you bundle the model.onnx and io.npz you are using in one single tidy piece I can download ? I don't want to sound like a diva, but the less I have to touch any python, the more likely I'll get motivated to look into the problem. |
@Yevgnen Good news, this is what I get with tract top of tree:
So I guess you can just pull the main branch for now. Anyway, I'm gonna do a release very soon (I was actually waiting for rust 1.59, it was release a few days ago). |
I'm also going to close this issue (as soon as the release is done). It would be nice to have an example, maybe something one of you nice folks would be nice enough to contribute (created #634 to track this). But as there is nothing structural in either tract or BERT models that is compatible, we can safely say tract supports BERT. If we have remaining issues with on or the other BERT incarnation around, we will deal with them as bugs. |
@kali Hi, that's really a good news. Thanks for you work! I confirm using the main branch I can get the correct output. I also made a small example and hope that help! |
@Yevgnen It does help ! thanks you. |
Not too sure what specific operators the BERT architecture will require but:
The text was updated successfully, but these errors were encountered: