PaSS methodology.

[jivanph]

Thank you so much for your work in establishing this benchmark!

I wanted to ask if you would include PaSS (https://arxiv.org/pdf/2311.13581.pdf) to your benchmark analysis, since it is included in your references, and it allows nucleus sampling, greedy sampling, but it doe require the training of extra token embeddings.

[hemingkx]

Thank you for your suggestion!

Unfortunately, we have not been able to find an open-source implementation of PaSS, nor have we found the pretrained weights for the token embeddings required to reproduce this method. We are keen to include PaSS in Spec-Bench if these resources become available.

[jivanph]

Thank you for your response. This makes a lot of sense.