Avoid dummy token in PLD to optimize performance

[ofirzaf]

What does this PR do?

Current PLD implementation adds a dummy token in case there is no match found in the prompt to avoid issues. In previous PR #28508 the code was fixed to handle zero candidate tokens so there is no need for the dummy token in PLD now. Not inserting this additional token can accelerate PDL on weaker HW where doing inference with the target with 2 tokens is significantly slower than doing inference with 1 token.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

@gante

[gante]

@ofirzaf Thank you for the optimization!

For future reference, could you share your test script (ideally paired up with the output numbers in your hardware)?

[ArthurZucker]

Looks good thanks for following up on this

[ofirzaf]

@gante I am unable to share my measurements since I am working on HW that wasn't released yet. Is it blocking the merge?

[gante]

@ofirzaf no :) I'm going to merge the PR

In any case, if you have the script to reproduce your measurements, it would be a nice addition