Starcoder repeat penalty

[the-crypt-keeper]

The santacoder model doesn't do well without repeat-penalty, failing to emit EOS for many basic prompts. For example:

main: prompt: 'def fib('
main: number of tokens in prompt = 3
main: token[0] =    563, def
main: token[1] =  24240,  fib
main: token[2] =      7, (


def fib(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fib(n-1) + fib(n-2)

def fib_iter(n):
    a, b = 0, 1
    for i in range(n):
        a, b = b, a + b
    return a

def fib_iter_2(n):
    a, b = 0, 1
    for i in range(n):
        a, b = b, a + b
    return a

def fib_iter_3(n):
    a, b = 0, 1
    for i in range(n):
        a, b = b, a + b
    return a

def fib_iter_4(n):
    a, b = 0, 1
    for i in range(n):
        a, b =

This change heavily borrows from examples/mpt/main.cpp to use gpt_sample_top_k_top_p_repeat and connect the necessary parameters. In an effort to not affect other examples using the same code, I have left the default repeat penalty at 1.0 so it wont be enabled by surprise.

Applying a --repeat-penalty 1.176 to the example above yields:

main: temp           = 0.900
main: top_k          = 1
main: top_p          = 0.900
main: repeat_last_n  = 64
main: repeat_penalty = 1.176
main: prompt: 'def fib('
main: number of tokens in prompt = 3
main: token[0] =    563, def
main: token[1] =  24240,  fib
main: token[2] =      7, (


def fib(n):
    if n == 0:
        return 1

    elif n == 1 or n == 2:
        return 1

    else:
        return fib(n - 1) + fib(n - 2)


print("fibonacci series:")
for i in range(5, 36):
    print("{0} = {1}".format(i, fib(i)))<|endoftext|>

Which is a much more useful result.