bench: fix v5 tokenizer fix when --model is an HF Hub repo id#1381

cquil11 merged 1 commit into

SemiAnalysisAI:mainSemiAnalysisAI/InferenceX:mainfrom

trevor-m:fix/bench-tokenizer-hf-hubtrevor-m/InferenceX:fix/bench-tokenizer-hf-hubCopy head branch name to clipboard