[Question] Failed to fetch onnx model when to use AutoModel.from_pretrained

[peter-up]

The code here:

import { AutoModel, AutoTokenizer } from '@xenova/transformers';

const modelPath = 'Xenova/distilgpt2'
let tokenizer = await AutoTokenizer.from_pretrained(modelPath); // **successful to fetch model**

let model = await AutoModel.from_pretrained(modelPath);  // **failed to fetch model**
let inputs = await tokenizer('I love transformers!');
let { logits } = await model(inputs);

Error information:
file:///Users/xxx/Documents/github/transformers.js/examples/node/esm/node_modules/@xenova/transformers/src/utils/hub.js:223
throw Error(Could not locate file: "${remoteURL}".)
^

Error: Could not locate file: "https://huggingface.co/Xenova/distilgpt2/resolve/main/onnx/model_quantized.onnx".
at handleError (file:///Users/xxx/Documents/github/transformers.js/examples/node/esm/node_modules/@xenova/transformers/src/utils/hub.js:223:19)
at getModelFile (file:///Users/xxx/Documents/github/transformers.js/examples/node/esm/node_modules/@xenova/transformers/src/utils/hub.js:412:24)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async constructSession (file:///Users/xxx/Documents/github/transformers.js/examples/node/esm/node_modules/@xenova/transformers/src/models.js:88:18)

transformers.js version: 2.1.1

[xenova]

Hi there. The reason for this is because we have not exported the model without its LM head. Is feature extraction your intended use-case?

You are most likely looking to use AutoModelForCausalLM, right?

[peter-up]

Thanks xenova. I want to compute perplexity with transformers.js. The origin python code is here:
https://huggingface.co/docs/transformers/perplexity#example-calculating-perplexity-with-gpt2-in-transformers.

I tried AutoModelForCausalLM and other model like gpt2, but it also got error.

[xenova]

Okay I see what you're looking for now 👍 . Although using AutoModelForCausalLM is able to find the files, it produces another issue since it assumes it is run within the .generate function.

I'll see what I can do.

[peter-up]

Thank you @xenova . I'm looking forward to it 👍 👍.

[xenova]

Will hopefully get this done this week! 👍 In the meantime, can you provide the equivalent python code (and its expected output) for doing this with the original transformers library, that would seriously help me out!

[peter-up]

Sure, all the code is from https://huggingface.co/docs/transformers/perplexity#example-calculating-perplexity-with-gpt2-in-transformers, and pasted below:

  from transformers import GPT2LMHeadModel, GPT2TokenizerFast
  import torch
  from tqdm import tqdm
  
  device = "cpu"
  model_id = "gpt2"
  model = GPT2LMHeadModel.from_pretrained(model_id).to(device)
  tokenizer = GPT2TokenizerFast.from_pretrained(model_id)
  
  
  test = 'I love transformer'
  encodings = tokenizer(test, return_tensors="pt")
  max_length = model.config.n_positions
  stride = 512
  seq_len = encodings.input_ids.size(1)
  
  nlls = []
  prev_end_loc = 0
  for begin_loc in tqdm(range(0, seq_len, stride)):
      end_loc = min(begin_loc + max_length, seq_len)
      trg_len = end_loc - prev_end_loc  # may be different from stride on last loop
      input_ids = encodings.input_ids[:, begin_loc:end_loc].to(device)
      target_ids = input_ids.clone()
      target_ids[:, :-trg_len] = -100
  
      with torch.no_grad():
        outputs = model(input_ids, labels=target_ids)
  
        # loss is calculated using CrossEntropyLoss which averages over valid labels
        # N.B. the model only calculates loss over trg_len - 1 labels, because it internally shifts the labels
        # to the left by 1.
        neg_log_likelihood = outputs.loss
  
  
      nlls.append(neg_log_likelihood)
  
      prev_end_loc = end_loc
      if end_loc == seq_len:
          break
  
  ppl = torch.exp(torch.stack(nlls).mean())
  print('ppl result:', ppl)

Output:
ppl result: tensor(83531.9844)

[xenova]

Busy with this here. Should be ready soon 👍

[xenova]

Okay I think I've got what you needed. These changes are currently present on this branch, and I would appreciate it if you tested it out before I merged into main.

Here's the example python and javascript code I used for testing:

Python

Adapted from https://huggingface.co/docs/transformers/perplexity#example-calculating-perplexity-with-gpt2-in-transformers

from transformers import GPT2LMHeadModel, GPT2TokenizerFast
import torch
from tqdm import tqdm

device = "cpu"
model_id = "gpt2"
model = GPT2LMHeadModel.from_pretrained(model_id).to(device)
tokenizer = GPT2TokenizerFast.from_pretrained(model_id)


test = 'Once upon a time, in a land shrouded by mist and mystery, there existed a small village nestled within a dense forest.'
encodings = tokenizer(test, return_tensors="pt")
max_length = 8 # model.config.n_positions
stride = 2 # 512
seq_len = encodings.input_ids.size(1)

nlls = []
prev_end_loc = 0
for begin_loc in range(0, seq_len, stride):
    end_loc = min(begin_loc + max_length, seq_len)
    trg_len = end_loc - prev_end_loc  # may be different from stride on last loop
    input_ids = encodings.input_ids[:, begin_loc:end_loc].to(device)
    target_ids = input_ids.clone()
    target_ids[:, :-trg_len] = -100

    with torch.no_grad():
      outputs = model(input_ids, labels=target_ids)

      # loss is calculated using CrossEntropyLoss which averages over valid labels
      # N.B. the model only calculates loss over trg_len - 1 labels, because it internally shifts the labels
      # to the left by 1.
      neg_log_likelihood = outputs.loss

    nlls.append(neg_log_likelihood)

    prev_end_loc = end_loc
    if end_loc == seq_len:
        break

ppl = torch.exp(torch.stack(nlls).mean())
print('ppl result:', ppl)

output:

ppl result: tensor(42.8545)

JavaScript

Above tutorial translated into JS.

import { AutoModel, AutoTokenizer, log_softmax } from '@xenova/transformers'; // NOTE: You may need to change this import if running locally

// This code is normally handled inside the forward method of the model, but since
// we can't pass target_ids, we must calculate this separately using the logits.
const loss = (logits, labels, ignore_index = -100) => {
    let [batchSize, seqLength, vocabSize] = logits.dims;

    // Shift so that tokens < n predict n
    let shiftLogits = logits.slice(null, [0, seqLength - 1], null);
    let shiftLabels = labels.slice(null, [1, seqLength], null);

    // Flatten
    shiftLogits = shiftLogits.view(-1, shiftLogits.dims.at(-1));
    shiftLabels = shiftLabels.view(-1);

    // Actually calculate cross entropy loss
    let sum = 0;
    let count = 0;
    for (let i = 0; i < shiftLogits.dims[0]; ++i) {
        let label = shiftLabels[i].data;
        if (label == ignore_index) { // == to match int with BigInt
            continue;
        }

        let logLogits = log_softmax(shiftLogits[i].data);
        sum += logLogits[label];
        ++count;
    }
    return - sum / count;
};

// Choose model to use
let model_id = "Xenova/gpt2"

// Load model and tokenizer
let model = await AutoModel.from_pretrained(model_id, { quantized: false });
let tokenizer = await AutoTokenizer.from_pretrained(model_id);

// Tokenize input text
let text = 'Once upon a time, in a land shrouded by mist and mystery, there existed a small village nestled within a dense forest.';
let encodings = await tokenizer(text);

// Set parameters. NOTE: these values are just for testing, and should be replaced with the commented out values.
let max_length = 8; // model.config.n_positions;
let stride = 2; // 512;
let seq_len = encodings.input_ids.dims[1];

let nlls = [];
let prev_end_loc = 0;
for (let begin_loc = 0; begin_loc < seq_len; begin_loc += stride) {
    // Perform same indexing as the tutorial
    let end_loc = Math.min(begin_loc + max_length, seq_len);
    let trg_len = end_loc - prev_end_loc;

    let input_ids = encodings.input_ids.slice(null, [begin_loc, end_loc]);
    let target_ids = input_ids.clone();
    for (let tg of target_ids) { // target_ids[:, :-trg_len] = -100
        tg.data.fill(-100n, 0, -trg_len);
    }

    // Run model
    let output = await model({ input_ids });

    // Compute loss
    let neg_log_likelihood = loss(output.logits, target_ids);
    nlls.push(neg_log_likelihood);

    prev_end_loc = end_loc;
    if (end_loc === seq_len) {
        break;
    }
}

// Compute perplexity
const mean = nlls.reduce((acc, val) => acc + val, 0) / nlls.length;
const ppl = Math.exp(mean);
console.log(`ppl result: ${ppl}`);

output:

ppl result: 42.855085073203895

---

The JS code is a little longer, but that's mainly because I had to reimplement the missing torch functions (e.g., cross entropy loss calculation). Let me know if there's something missing, otherwise I'll merge into main and make a release. 👍

[peter-up]

I tested it and it works now. Thank you so much for your professional and kind help. 👍👍