Confusion about the prompt size

[viraptor]

I'm trying to play with rwkv's long context capabilities. As far as I understand, the linked https://huggingface.co/cgisky/RWKV-x070-Ai00/tree/main/world_v3/3b model should be enough for the RWKV capabilities of really long prompts, right? When I cross a few thousand characters (using the oapi endpoint and the test python script from this repo), I start getting silly responses. This happens even after bumping up max_tokens.

Am I misunderstanding what's possible to do here? Or doing something wrong?

[cryscan]

The model's architecture supports infinite context length, in the sense of consuming only a finite amount of VRAM. The performance, on the other hand, depends on a lot of things:

1. Is your prompt properly formatted?
2. Is your prompt close to the model's training data?
3. Is the sampling parameters suitable?

You could paste your prompt and your sampling parameters here if you feel it comfortable and we can help check if everything is right.

[viraptor]

So when I said silly responses, I mean actual corrupted garbage.
Going up in size:
"testing" -> reasonable response.
"Summarize the following messages: " + 200 tokens -> reasonable summary.
"Summarize the following messages: " + 1200 tokens -> noise like "When we are finished", or random Chinese characters, or "When we think about this. ", or string of 👉 emoji.

I used the parameters from the test client file:

ai00.set_params(
    max_tokens = 10000,
    top_p = 0.55,
    temperature = 2,
    presence_penalty = 0.3,
    frequency_penalty = 0.8,
    half_life = 400,
    stop = ['\x00','\n\n']
)

[shoumenchougou]

maybe can set temperature = 1.3 and follow RWKV Prompting Format Guidelines