I can not save the LLM's config (e.g. MAX_TOKEN) in workflow after publish 【v0.6.3】

[sapipoZZZ]

Self Checks

• This is only for bug report, if you would like to ask a quesion, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• Pleas do not modify this template :) and fill in all the required fields.

Dify version

0.6.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

I can not save the LLM's config (e.g. MAX_TOKEN) in workflow after publish 【v0.6.3】

✔️ Expected Behavior

fix bug

❌ Actual Behavior

No response

[dosubot]

To address the issue of not being able to save the LLM's MAX_TOKEN configuration in Dify version 0.6.3, it's important to first ensure that the configuration changes are being made correctly within the system. The process involves navigating to Settings > Model Providers > OpenLLM in your Dify setup and entering the model details, including the MAX_TOKEN setting. After entering these details, clicking "Save" should apply the configuration.

The MAX_TOKEN setting is defined within the Dify configuration as follows, allowing a range from 1 to 8192:

  - name: max_tokens
    use_template: max_tokens
    default: 2048
    min: 1
    max: 8192

If you're encountering issues saving this setting, it could be related to changes in how Dify handles JSON and XML response formats in version 0.6.3. These changes might affect configuration saving if your setup relies on specific output formats for processing or storage.

Unfortunately, without a specific error message or more detailed description of the behavior you're experiencing, it's challenging to provide a precise solution. Could you provide more details about the error you encounter when attempting to save the MAX_TOKEN setting? This information will be crucial for identifying the root cause and guiding you towards a resolution.

Sources

• Relate to Custom > Max Token
• api/core/model_runtime/model_providers/openai/llm/llm.py
• en/tutorials/model-configuration/openllm.md
• api/core/model_runtime/model_providers/tongyi/llm/qwen-turbo.yaml
• api/core/model_runtime/model_providers/tongyi/llm/qwen-max.yaml
• api/core/model_runtime/entities/defaults.py
• api/core/model_runtime/model_providers/tongyi/llm/qwen-max-0403.yaml
• api/core/model_runtime/model_providers/spark/llm/spark-3.5.yaml

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[crazywoola]

Please provide more information regarding this issue? Which LLM do you use for that node. A screenshot is perferred.

[sapipoZZZ]

Please provide more information regarding this issue? Which LLM do you use for that node. A screenshot is perferred.
Whether using GPT-4 or Qwen1.5, the setting for max tokens will be lost.

[upseem]

The default max-token of command-r is 128k, but the display is 4096. I modified - api/core/model_runtime/model_providers/cohere/llm/command-r.yaml and mapped it to docker. Dify can operate, but messages cannot be sent. Report an error.

[upseem]

[Yeuoly]

Max tokens is different from context_size, the first one take care of "at most how many tokens the model could output", and the second one means "how many tokens the model could accept as messages", the max_tokens is actually 4000, that's why you got this error.

[Yeuoly]

Please provide more information regarding this issue? Which LLM do you use for that node. A screenshot is perferred.
Whether using GPT-4 or Qwen1.5, the setting for max tokens will be lost.

I just cannot reproduce this issue, configs were saved correctly, are there any more detailed steps?

[upseem]

Gpt-4 turbo is a 128k context, and it is the same situation. I can only set it to 4096. This is restricted by Dify or OpenAI.

@Yeuoly Please help me answer it, thank you very much.

[crazywoola]

Max tokens is different from context_size, the first one take care of "at most how many tokens the model could output", and the second one means "how many tokens the model could accept as messages", the max_tokens is actually 4000, that's why you got this error.

@upseem Already answered

[sapipoZZZ]

Please provide more information regarding this issue? Which LLM do you use for that node. A screenshot is perferred.
Whether using GPT-4 or Qwen1.5, the setting for max tokens will be lost.

In my workflow, I have 3 LLM nodes,
you can try:
step1: LLM1-->model--->change MAX_TOKENS
step2: publish--> update
step3: click LLM2's model, and switch to LLM1. LLM1's model setting will be lost

[crazywoola]

Please provide more information regarding this issue? Which LLM do you use for that node. A screenshot is perferred.
Whether using GPT-4 or Qwen1.5, the setting for max tokens will be lost.

In my workflow, I have 3 LLM nodes, you can try: step1: LLM1-->model--->change MAX_TOKENS step2: publish--> update step3: click LLM2's model, and switch to LLM1. LLM1's model setting will be lost

Step 2

Step 3

Cant repro

[sapipoZZZ]

@crazywoola I have recorded a video for you. It seems to always restore to the default LLM setting.

2024-04-18.09-20-34.mp4

[tangxqa]

@crazywoola I have recorded a video for you. It seems to always restore to the default LLM setting.

2024-04-18.09-20-34.mp4

I encountered the same problem today