decoder may have some bug

[lichen914]

this is something wrong when using GPTBigCodeForCausalLM model in decoder，while using greedy method.

basemodel

starcoderbase-15b huggingface

prompt：

/**\n * Write typescript function to find the minimum cost path to reach (m, n) from (0, 0) for the given cost matrix cost[][] and a position (m, n) in cost[][].\n * \n * Examples:\n * >>> min_cost([[1, 2, 3], [4, 8, 2], [1, 5, 3]], 2, 2)\n * >>> 8\n * >>> min_cost([[2, 3, 4], [5, 9, 3], [2, 6, 4]], 2, 2)\n * >>> 12\n * >>> min_cost([[3, 4, 5], [6, 10, 4], [3, 7, 5]], 2, 2)\n * >>> 16\n */\nconst min_cost = function (cost: Array<Array<number>>, m: number, n: number) : number {\n

vllm generate result:

\t>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

greedy config

sampling_params = SamplingParams( temperature=0, max_tokens=512 )

huggingface generate result：

function minCost(cost: number[][], m: number, n: number): number { let min = cost[0][0]; for (let i = 0; i < m; i++) { for (let j = 0; j < n; j++) { if (i === 0 && j === 0) { continue; } if (cost[i][j] < min) { min = cost[i][j]; } } } return min; }

the greedy config

generation_config = GenerationConfig( temperature=0, max_new_tokens=512, pad_token_id=0 )

[anttttti]

I came across likely the same problem. Decoding for at least all starcoder models was broken in vllm==0.1.3
With vllm==0.1.2:
def sum_numbers(x, y): -> "\n return x + y\n\ndef sum_numbers_2(x, y):\n return"
With vllm==0.1.3:
def sum_numbers(x, y): -> "\n def test_id=None,\n def test_id='''')\n self-\n"

0.1.2 response is near identical to Transformers model response. The syntax is broken for the 0.1.3 response. If you test it with other prompts, you get various types of garbled output with 0.1.3

This should be a high priority bug to fix

[aranku]

Should be fixed already #676. Try building from source