fix: implement dynamic top_p parameter adjustment for LLM API calls

[Airmomo]

• Add _top_p_adjusted flag to track parameter adjustments
• Implement _fix_top_p_param to adjust invalid top_p values
• Enhance _call_llm_with_retry with parameter adjustment retry mechanism
• Apply consistent implementation across all generator classes
• Improve error logging for API error debugging

---

Issue

When I request LLM API with top_p=0, some model providers （such as DeepSeek）reject the request with a 400 error, indicating that the valid range for top_p is (0, 1.0]. This causes API calls to fail in various generators(L1), including L1Generator, TopicsGenerator, ShadeGenerator, and StatusBioGenerator.

Cause analysis

1. The original code had these issues:

# In all generator classes - hardcoded top_p=0
self.model_params = {
    "temperature": 0,
    "top_p": 0,  # This value is rejected by some LLM providers
    # other parameters...
}

# No error handling for invalid top_p parameter

2. The problems with this approach:

• Compatibility issues: Different LLM providers have different requirements for valid parameter ranges
• Rigid configuration: No flexibility to adjust parameters when API calls fail
• Poor error handling: API errors related to parameter validation were not properly handled
• No retry mechanism: Failed API calls were not retried with adjusted parameters

Fix

1. Implemented a flag to track parameter adjustments:

self._top_p_adjusted = False  # Flag to track if top_p has been adjusted

2. Added a function to fix top_p parameter when errors occur:

def _fix_top_p_param(self, error_message: str) -> bool:
    if not self._top_p_adjusted and "top_p" in error_message.lower():
        logger.warning(f"API Error: {error_message}")
        logger.info("Fixing top_p parameter from 0 to 0.001 to comply with model API requirements")
        self.bio_model_params["top_p"] = 0.001
        self._top_p_adjusted = True
        return True
    return False

3. Enhanced the API call method with retry mechanism:

def _call_llm_with_retry(self, messages: List[Dict[str, str]], **kwargs) -> Any:
    try:
        return self.client.chat.completions.create(
            model=self.model_name,
            messages=messages,
            **self.bio_model_params,
            **kwargs
        )
    except Exception as e:
        error_msg = str(e)
        logger.error(f"API Error: {error_msg}")
        
        # Try to fix top_p parameter if needed
        if hasattr(e, 'response') and hasattr(e.response, 'status_code') and e.response.status_code == 400:
            if self._fix_top_p_param(error_msg):
                logger.info("Retrying LLM API call with adjusted top_p parameter")
                return self.client.chat.completions.create(
                    model=self.model_name,
                    messages=messages,
                    **self.bio_model_params,
                    **kwargs
                )
        
        # Re-raise the exception
        raise

Currently, only the top_p parameter for error handling and dynamic adjustment. I think: If other parameter error issues arise in the future, error handling can be added on this basis.

4. Applied these changes consistently across all four generator classes:
   • L1Generator
   • TopicsGenerator
   • ShadeGenerator
   • ShadeMerger
   • StatusBioGenerator

Each generator has implemented the same logic, which may be a bit redundant, but in order not to compromise the independence of each generator, I think I should do it this way.

Improvements

1. Increased Compatibility:

   • System now works with LLM providers that don't accept top_p=0
   • Uses a value close to 0 (0.001) to maintain near-deterministic behavior

2. Robust Error Handling:

   • Properly logs API errors for debugging
   • Provides clear information about parameter adjustments

3. Automatic Recovery:

   • Implements a retry mechanism to recover from parameter validation errors

4. Consistent Implementation:

   • Same fix applied uniformly across all (L1) generators
   • Maintains code consistency throughout the codebase

Share my thinking

At first, I wanted to simply add a configuration item in .env to uniformly set top_p and temperature, but after reviewing the code of the generator, I realized that there may be different parameter sets for each individual generator in the future development of the project. Therefore, it is not necessary to unify the parameter configuration here.

In addition, considering the concept of the project, I believe that the generated results should be as close to the data itself as possible. Therefore, it is not necessary to treat it as a customized configuration item here. I only need to dynamically update top_p to a value as close to 0 as possible, which will achieve a very close approximation; and temperature is a value that almost all mainstream models support setting to 0, so I will not make dynamic adjustments to it for now.

[kevin-mindverse]

@yingapple plz take a look at this.

[yingapple]

great job!

[ScarletttMoon]

Love the work!

[ScarletttMoon]

Hi @Airmomo 👋,

Thank you so much for your contribution to this PR! Your work is really appreciated. If you haven’t already, feel free to join our Discord community here: Discord Invite Link — it's a great place to connect with our team and other contributors, share ideas, and stay up to date with the project!

Also, feel free to add me directly on WeChat (id: Neon_MoonS) if you’d like to chat more or have any questions!

Looking forward to connecting! 😊