Motivation
For a remote server, the served model name may be different from its tokenizer name/path. The tokenizer is not loaded in such case, and thus tokens-level stats may be incomplete at the end of the run.
Proposed Solution
Add an optional tokenizer_path parameter to ModelParams and use when the model's name parameter isn't a valid path (either locally or on huggingface)
Alternatives Considered
No response
Additional Context
No response
Motivation
For a remote server, the served model name may be different from its tokenizer name/path. The tokenizer is not loaded in such case, and thus tokens-level stats may be incomplete at the end of the run.
Proposed Solution
Add an optional
tokenizer_pathparameter toModelParamsand use when the model'snameparameter isn't a valid path (either locally or on huggingface)Alternatives Considered
No response
Additional Context
No response