Model load balancing usage issues

[yunfeng13]

Self Checks

• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

Dify version

0.8.3

Cloud or Self Hosted

Cloud

Steps to reproduce

After load balancing is enabled, the key used is always the latest one added. Why is the default key not used? Shouldn't it be used one by one? I want to know when the default key will be used, when load balancing will take effect, and when the current is limited.

✔️ Expected Behavior

After load balancing is enabled, the key used is always the latest one added. Why is the default key not used? Shouldn't it be used one by one? I want to know when the default key will be used, when load balancing will take effect, and when the current is limited.

❌ Actual Behavior

After load balancing is enabled, the key used is always the latest one added. Why is the default key not used? Shouldn't it be used one by one? I want to know when the default key will be used, when load balancing will take effect, and when the current is limited.

[dosubot[bot]]

The default key is not used in load balancing because load balancing requires multiple API keys to distribute the requests. The default key is the credential added when initially configuring the model provider. Load balancing will take effect and the current will be limited when at least one additional model credential is added to the configuration ^[1].

For more details, you might find this similar discussion helpful: What will happen to the API when load balancing is enabled and rate limiting is enabled? ^[2].

To continue talking to Dosu, mention @dosu.

[yunfeng13]

@StudyingLover @dosu

I set up two additional keys, but I still use one key and don’t use both together.

[dosubot[bot]]

To ensure that load balancing works correctly with multiple API keys, follow these steps:

1. Add Configuration: Make sure you have added the additional API keys by clicking the Add Configuration button in the load balancing settings.
2. Save Changes: Ensure that you save the changes after adding the new keys.
3. Enable Load Balancing: Verify that the load balancing feature is enabled.

Dify uses the Round-robin strategy by default for load balancing, which means it should distribute requests evenly across all configured keys ^[1]. If you still see only one key being used, double-check that all keys are correctly added and saved in the configuration.

If the issue persists, there might be a problem with how the keys are being recognized or saved. In that case, reviewing the configuration steps or checking for any known issues in the documentation might help.

To continue talking to Dosu, mention @dosu.

[yunfeng13]

@StudyingLover @dosu
But it was indeed saved successfully, but there was no load balancing, but the first key was always used.

[dosubot[bot]]

The first key is always used instead of load balancing across multiple keys because at least one additional model credential must be added to save and enable load balancing. If only the default API key is configured without adding additional keys, load balancing will not be activated, and the system will continue to use the first key. Additionally, it is important to note that model load balancing is a paid feature and requires a subscription to a SaaS paid service or the purchase of an enterprise version to enable this functionality ^[1].

If you have already added additional keys and saved the configuration successfully, but still see only the first key being used, it might be worth checking the following:

1. Verify Configuration: Double-check that all additional keys are correctly added and saved in the configuration.
2. Subscription: Ensure that your subscription includes the load balancing feature.
3. Logs and Debugging: Use the logging and debugging features in Dify to diagnose the issue. The Logs feature records all interaction processes, and integrating with Langfuse can provide detailed trace information ^[2][3].

If everything seems correct and the issue persists, there might be a deeper problem with how the keys are being recognized or saved.

To continue talking to Dosu, mention @dosu.

[yunfeng13]

@StudyingLover @dosu
Can you tell me which part of the load balancing code is in, that is, use the key saved by the load balancing to request

[dosubot[bot]]

The part of the load balancing code responsible for using the key saved by the load balancing to make requests is:

provider_model_credentials_cache = ProviderCredentialsCache(
    tenant_id=load_balancing_model_config.tenant_id,
    identity_id=load_balancing_model_config.id,
    cache_type=ProviderCredentialsCacheType.LOAD_BALANCING_MODEL,
)

# Get cached provider model credentials
cached_provider_model_credentials = provider_model_credentials_cache.get()

This code snippet retrieves the cached provider model credentials using the key saved by the load balancing configuration. The ProviderCredentialsCache class is used to manage the cache, and the get method fetches the credentials based on the tenant_id and identity_id from the load_balancing_model_config ^[1][2].

To continue talking to Dosu, mention @dosu.

[yunfeng13]

@StudyingLover @dosu
So it means that if I don’t have it in redis, will there be a situation where only one key is used?

[dosubot[bot]]

Yes, if you don’t have the load balancing configuration in Redis, there will be a situation where only one key is used. This is because the ModelInstance class relies on the LBModelManager for load balancing. If the load_balancing_manager is not set (i.e., None), the _round_robin_invoke method will directly call the function with the provided credentials without any load balancing.

Here is the relevant code snippet:

def _round_robin_invoke(self, function: Callable, *args, **kwargs):
    """
    Round-robin invoke
    :param function: function to invoke
    :param args: function args
    :param kwargs: function kwargs
    :return:
    """
    if not self.load_balancing_manager:
        return function(*args, **kwargs)

If self.load_balancing_manager is None, the function is invoked directly, meaning only one set of credentials (one key) will be used ^[1].

To continue talking to Dosu, mention @dosu.

[yunfeng13]

@StudyingLover @dosu The saved load balancing configuration is now in the table load_balancing_model_configs, but the load balancing fails and the default key is always used. What may be the reason?

[dosubot[bot]]

The load balancing fails and the default key is always used because at least one additional model credential must be added to save and enable load balancing. If no additional credentials are added, the system will default to using the initial API key.

Additionally, the system checks for the existence of an "inherit" configuration, which represents the provider or model custom credentials. If this "inherit" configuration does not exist, it initializes one. This ensures that the load balancing configuration is properly set up.

Moreover, if the encrypted_config field in the load_balancing_model_config is not populated, the system will skip that configuration unless it is the "inherit" configuration, which is added with empty credentials ^[1][2][3].

To continue talking to Dosu, mention @dosu.