Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

default values of max_generated_tokens, top_k, top_p, and temperature? #11033

Open
JamieVC opened this issue May 15, 2024 · 1 comment
Open

Comments

@JamieVC
Copy link

JamieVC commented May 15, 2024

What are default values of max_generated_tokens, top_k, top_p, and temperature?
If user doesn't set all parameters in generate_kwargs such as the example below, it should use default values. How do we get them in which source file?

    # Use custom LLM in BigDL
    from ipex_llm.llamaindex.llms import IpexLLM
    llm = IpexLLM.from_model_id(
        model_name=args.model_path,
        tokenizer_name=args.tokenizer_path,
        context_window=512,
        max_new_tokens=args.n_predict,
        generate_kwargs={"temperature": 0.7, "do_sample": False},
        model_kwargs={},
        messages_to_prompt=messages_to_prompt,
        completion_to_prompt=completion_to_prompt,
        device_map="xpu",
    )

refer to:
https://github.com/intel-analytics/ipex-llm/blob/main/python/llm/example/GPU/LlamaIndex/rag.py

@ivy-lv11
Copy link
Contributor

The default values are determined by the model configuration. For instance, if you use the Llama-2-chat-hf model, the default settings are temperature=0.9 and top_p=0.6. You can modify these settings in the generation_config.json file located in the model folder.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants