add context_length to model configs that us HF models (and as param to HFTokenizer)

[iejMac]

No description provided.

[usuyama]

How about something like this?

open_clip/src/open_clip/factory.py

Line 73 in 3e3e8a0

def get_tokenizer(model_name):

def get_tokenizer(model_name):
    config = get_model_config(model_name)
    context_length = config['text_cfg'].get("context_length", 77)
    if 'hf_tokenizer_name' in config['text_cfg']:
       tokenizer = HFTokenizer(config['text_cfg']['hf_tokenizer_name'], context_length)
    else:
       tokenizer = lambda texts: tokenize(texts, context_length)
    return tokenizer

[rom1504]

sure why not

[iejMac]

btw, I'm not sure if this is urgent so I've been putting it off. If people need this soon I can push it up the priority queue

[rom1504]

we noticed that some models that use relative positional embeddings (eg mt5) do not have a limited context length.

That's an edge case to take into account here