Is there somewhere more detailed insight on arguments and their influence on the output?

[mirozahorak]

Is there any more in depth info on how to adjust the arguments for best results? More specifically i mean these arguments, for us, who are not at home with ML, please?

  -mc N,     --max-context N     [-1     ] maximum number of text context tokens to store
  -ml N,     --max-len N         [0      ] maximum segment length in characters
  -bo N,     --best-of N         [5      ] number of best candidates to keep
  -bs N,     --beam-size N       [-1     ] beam size for beam search
  -wt N,     --word-thold N      [0.01   ] word timestamp probability threshold
  -et N,     --entropy-thold N   [2.40   ] entropy threshold for decoder fail
  -lpt N,    --logprob-thold N   [-1.00  ] log probability threshold for decoder fail
  -nf,       --no-fallback       [false  ] do not use temperature fallback while decoding

I was playing with some random adjustments but see no effect in the output.

[rianbrob]

found this

#291

[joliss]

I did some digging on the --enthropy-thold option, which can help suppress hallucinated repetitions, and found the algorithm that computes it.

Notably, this entropy is not derived from the softmaxed token probabilities -- that's what --logprob-thold is for, though I've found it less useful -- but rather it looks at how often there are repeated tokens in the last 32 tokens. Specifically, it's calculated as

$$H=-\sum _i{p}_i\ln (p_i)$$

(the Shannon entropy), where

$$p_i=\frac{\text{count of token }i\text{ among last 32 tokens}}{32}.$$

The behavior is that if the last 32 tokens fall below the given entropy_thold (2.40 by default), it tries decoding again (continuing this loop) with the temperature increased by 0.2 (--temperature-inc), up through a maximum temperature of 1.0.

To get a sense of the scale, I used ChatGPT to calculate the entropy you get if the last 32 tokens are a sentence of $n$ distinct tokens, repeated indefinitely. For example, for $n=3$, we calculate the entropy for the token sequence [0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1]. (This is to mimic the hallucinated repetitions I tend to see. Note however that this does not take into account timing tokens, which I assume are always distinct.)

As you can see from the plot, the highest entropy you can get with all-distinct tokens is 3.47. To suppress repetitions more aggressively, you can increase the threshold from the default of 2.40, but you probably don't want to get too close to the maximum of 3.47, or you'll get a lot of retries and unnecessarily increase the temperature. In some very brief testing I found that 3.0 seems to do a good job getting rid of repetitions, and looking at the transcript diff, I can't make out much of a decrease in quality.