Where do I set my temperature

[shatealaboxiaowang]

Hi ,dear:
I set temperature on the client as shown below, but it does not take effect:

curl -X POST localhost:8019/v2/models/ensemble/generate -d '{"text_input": "import numpy", "max_tokens": 128, "bad_words": "", "stop_words": "pandas", "temperature": 0.7}'
every curl, Same input and same output

thank you!

[byshiue]

If you don't set topk/topp, the default setting is greedy search and there is no effect by setting temperature.

[shatealaboxiaowang]

If you don't set topk/topp, the default setting is greedy search and there is no effect by setting temperature.

thank you for your replay!
But it doesn't work when I set top_k/p. Here's what I did:
curl -X POST localhost:8019/v2/models/ensemble/generate -d '{"text_input": "import numpy", "max_tokens": 50, "bad_words": "", "stop_words":"", "temperature":1.0, "runtime_top_k":100, "runtime_top_p":100}'

[byshiue]

topp is a number between 0.0 ~ 1.0.
Beside, if you don't change the random seed, it will generate same output under same random seed.

[shatealaboxiaowang]

topp is a number between 0.0 ~ 1.0. Beside, if you don't change the random seed, it will generate same output under same random seed.

when i set topp as 0.0 ~ 1.0, the error is: {"error":"parameter 'runtime_top_p' has invalid type. It should be either 'int', 'bool', or 'string'."}. And I set random_seed, and random_seed is different for each request, and the output is the same.
And I have another question, temperature/topp/topk three parameters should be the parameters to control the output diversification, random_seed parameter in the inference process is used in which calculation process?

[moseshu]

topp is a number between 0.0 ~ 1.0. Beside, if you don't change the random seed, it will generate same output under same random seed.
in end_to_end_grpc_client.py, I add random_seed .but it does't work,why?
grpc protocol,streaming mode

    random_seed = random.randint(1,1000)
    print(f"random_seed:{random_seed}")
    random_seed_data = np.array([[random_seed]], dtype=np.uint64)
    inputs = [
        prepare_tensor("text_input", input0_data, FLAGS.protocol),
        prepare_tensor("max_tokens", output0_len, FLAGS.protocol),
        prepare_tensor("stream", streaming_data, FLAGS.protocol),
        prepare_tensor("beam_width", beam_width_data, FLAGS.protocol),
        prepare_tensor("temperature", temperature_data, FLAGS.protocol),
        prepare_tensor("random_seed", random_seed_data, FLAGS.protocol),
    ]

[kisseternity]

topp is a number between 0.0 ~ 1.0. Beside, if you don't change the random seed, it will generate same output under same random seed.

when i set topp as 0.0 ~ 1.0, the error is: {"error":"parameter 'runtime_top_p' has invalid type. It should be either 'int', 'bool', or 'string'."}. And I set random_seed, and random_seed is different for each request, and the output is the same. And I have another question, temperature/topp/topk three parameters should be the parameters to control the output diversification, random_seed parameter in the inference process is used in which calculation process?

Confused too. In huggingface's generate func, random_seed is not needed. Only with top_p, top_k and temperature, it will generate diverse outputs.

[byshiue]

A gentle remind. Please make sure you quote the content you want to refer. Don't put your question in "quote", it is hard to read the question and reply.

[byshiue]

topp is a number between 0.0 ~ 1.0. Beside, if you don't change the random seed, it will generate same output under same random seed.

when i set topp as 0.0 ~ 1.0, the error is: {"error":"parameter 'runtime_top_p' has invalid type. It should be either 'int', 'bool', or 'string'."}. And I set random_seed, and random_seed is different for each request, and the output is the same. And I have another question, temperature/topp/topk three parameters should be the parameters to control the output diversification, random_seed parameter in the inference process is used in which calculation process?

Please check your config.pbtxt and example codes. From definition in config.pbtxt https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/all_models/inflight_batcher_llm/tensorrt_llm/config.pbtxt#L121-L127, it should be float.

If you only set the random seed, then the sampling is ran on topk=1 and hence the resutls are always same.

random_seed is used in sampling.

[byshiue]

topp is a number between 0.0 ~ 1.0. Beside, if you don't change the random seed, it will generate same output under same random seed.
in end_to_end_grpc_client.py, I add random_seed .but it does't work,why?
grpc protocol,streaming mode

    random_seed = random.randint(1,1000)
    print(f"random_seed:{random_seed}")
    random_seed_data = np.array([[random_seed]], dtype=np.uint64)
    inputs = [
        prepare_tensor("text_input", input0_data, FLAGS.protocol),
        prepare_tensor("max_tokens", output0_len, FLAGS.protocol),
        prepare_tensor("stream", streaming_data, FLAGS.protocol),
        prepare_tensor("beam_width", beam_width_data, FLAGS.protocol),
        prepare_tensor("temperature", temperature_data, FLAGS.protocol),
        prepare_tensor("random_seed", random_seed_data, FLAGS.protocol),
    ]

Please see #221 (comment).

[byshiue]

topp is a number between 0.0 ~ 1.0. Beside, if you don't change the random seed, it will generate same output under same random seed.

when i set topp as 0.0 ~ 1.0, the error is: {"error":"parameter 'runtime_top_p' has invalid type. It should be either 'int', 'bool', or 'string'."}. And I set random_seed, and random_seed is different for each request, and the output is the same. And I have another question, temperature/topp/topk three parameters should be the parameters to control the output diversification, random_seed parameter in the inference process is used in which calculation process?

Confused too. In huggingface's generate func, random_seed is not needed. Only with top_p, top_k and temperature, it will generate diverse outputs.

If we don't expose the random seed parameter, then users cannot get same results under fully same inputs, and hard to reproduce the results. So, we think exposing the random seed is a more general way.

[byshiue]

Related issue #237

[kisseternity]

Confused too. In huggingface's generate func, random_seed is not needed. Only with top_p, top_k and temperature, it will generate diverse outputs.

Thanks for the explanation.
Indeed I check the huggingface's generate func and find there is an optional random_seed parameter. It will be a random number if not set. It's just that the code in client demo https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/inflight_batcher_llm/client/inflight_batcher_llm_client.py doesn't have a parameter to set the random_seed param so I get confused.

[shatealaboxiaowang]

topp is a number between 0.0 ~ 1.0. Beside, if you don't change the random seed, it will generate same output under same random seed.

when i set topp as 0.0 ~ 1.0, the error is: {"error":"parameter 'runtime_top_p' has invalid type. It should be either 'int', 'bool', or 'string'."}. And I set random_seed, and random_seed is different for each request, and the output is the same. And I have another question, temperature/topp/topk three parameters should be the parameters to control the output diversification, random_seed parameter in the inference process is used in which calculation process?

Please check your config.pbtxt and example codes. From definition in config.pbtxt https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/all_models/inflight_batcher_llm/tensorrt_llm/config.pbtxt#L121-L127, it should be float.

If you only set the random seed, then the sampling is ran on topk=1 and hence the resutls are always same.

random_seed is used in sampling.

thank you!