bug: Max context length automatically defaults to 4096 tokens

[hantran-co]

Jan version

0.5.8

Describe the Bug

https://discord.com/channels/1107178041848909847/1204617960615055401/1301375861282312235

For imported models only, Jan did try to fall back to hub model.json, so it uses the maximum number instead:

We need to find a better solution for this.

Steps to Reproduce

No response

Screenshots / Logs

No response

What is your OS?

• MacOS
• Windows
• Linux

[louis-jan]

Current:
The current default value = min(8192, max_model_context_length)

Where it should be:
Differentiate between model metadata and run settings:

1. Users can define model load parameters (both CLI and API, currently API is supported), default is the formula above.
2. Model.yaml should maintain GGUF metadata.

Session settings:
-> Stored in threads/assistants accordingly - client’s perspective
-> Run arguments - CLI perspective