Transformer-based language model for text generation.
GPT-2 is a large transformer-based language model with a simple objective: predict the next word, given all of the previous words within some text.
| Model | Download | Download (with sample test data) | ONNX version | Opset version | Accuracy |
|---|---|---|---|---|---|
| GPT-2 | 522.81 MB | 438.3 MB | 1.6 | 10 | mAP of 0.024 |
The script for ONNX model conversion and ONNX Runtime inference is here.
Sequence of words as a string. Example: "Here is some text to encode : Hello World", tokenized by Byte-Pair-Encoding. input_ids: Indices of input tokens in the vocabulary. It's a long tensor of dynamic shape (batch_size, sequence_length).
Use tokenizer.encode() to encode the input text:
text = "Here is some text to encode : Hello World"
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
tokens_tensor = torch.tensor([torch.tensor(tokenizer.encode(text))])last_hidden_state: Sequence of hidden-states at the last layer of the model. It's a float tensor of size (batch_size, sequence_length, hidden_size). past: pre-computed hidden-states. It's a list of tensors (key and values in the attention blocks) of size (batch_size, num_heads, sequence_length, sequence_length), one per each layer.
Output of this model is the tuple (last_hidden_state, past)
Note that output_hidden_states=False and output_attentions=False in the PretrainedConfig configs.
outputs = model(input_ids)
last_hidden_states = outputs[0]The original model from OpenAI is pretrained on a dataset of 8 million web pages. The pretrained model is referenced in huggingface/transformers repository as a causal (unidirectional) transformer pre-trained using language modeling on a very large corpus of ~40 GB of text data. https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-pytorch_model.bin
Metric and benchmarking details are provided by HuggingFace in this post.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, andIlya Sutskever. Language Models are Unsupervised Multitask Learners. 2019.
This model is converted directly from huggingface/transformers.
Negin Raoof
Apache 2.0 License