dehardcode test string

[JimmyZhang12]

What does this PR do ?

By default to_word_list_format uses '<extra_id_1>' as its test string to map a string to its tokens, but using '<extra_id_1>' may not map correctly, so add option to specifiy the test string.

Collection: [Note which collection this PR will affect]

Changelog

• Add specific line by line info of high level changes in this PR.

Usage

• You can potentially add a usage example below

# Add a code snippet demonstrating how to use this 

Jenkins CI

To run Jenkins, a NeMo User with write access must comment jenkins on the PR.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

• New Feature
• Bugfix
• Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

• Related to # (issue)

[odelalleau]

Just a minor request to update the comment.

Besides this lgtm, but I just want to point out that using <extra_id_1> as stop word with the Llama2 tokenizer is a bad idea since as you mentioned it may be tokenized in various ways when merged with what comes before. When using the text generation from NeMo at least we fall back to the (slower) string matching (here), but with TRT-LLM I don't know if such a string matching exists (?) => one solution could be to postprocess the output to be sure we stop on the first stop word (this has nothing to do with this PR though -- just mentioning it in case this would be relevant to your work)

[oyilmaz-nvidia]

LGTM.

[JimmyZhang12]

Can this be merged @oyilmaz-nvidia ?

[oyilmaz-nvidia]

LGTM, thanks!

[oyilmaz-nvidia]

jenkins