fix: supports_function_calling works with llm_proxy models

[pazevedo-hyland]

This PR enhances the LiteLLM proxy function calling capabilities with comprehensive testing and improved model info retrieval, addressing the challenge of determining function calling support for custom proxy model names.

🔍 Problem Statement

The LiteLLM proxy allows routing requests through custom model names that map to underlying LLM providers. However, the `supports_function_calling()` function faces a challenge: it needs to determine if a custom proxy model name supports function calling, but without access to the proxy server's configuration, it cannot resolve custom names to underlying models.

Examples of the Challenge:

• `'bedrock-claude-3-haiku'` might map to `'bedrock/converse/anthropic.claude-3-haiku-20240307-v1:0'`
• `'prod-gpt4'` might map to `'azure/gpt-4'`
• `'my-custom-claude'` might map to `'anthropic/claude-3-sonnet-20240229'`

🛠️ How It Works

1. Direct Model Resolution ✅

When possible, LiteLLM tries to infer the underlying model from the proxy model name using fallback logic:

# These work with current fallback logic:
assert supports_function_calling(\"litellm_proxy/gpt-4\") == True
assert supports_function_calling(\"litellm_proxy/claude-3-sonnet-20240229\") == True
assert supports_function_calling(\"litellm_proxy/gemini/gemini-1.5-pro\") == True

2. Custom Model Names ⚠️

When proxy model names are custom and cannot be resolved, the function returns `False` as a safe default:

# These return False without proxy config (expected behavior):
assert supports_function_calling(\"litellm_proxy/bedrock-claude-3-haiku\") == False
assert supports_function_calling(\"litellm_proxy/my-custom-gpt4\") == False
assert supports_function_calling(\"litellm_proxy/prod-claude-haiku\") == False

3. Proxy Server Context 🎯

In a real proxy server deployment, the server has access to model mapping configuration:

# proxy_server_config.yaml
model_list:
  - model_name: bedrock-claude-3-haiku
    litellm_params:
      model: bedrock/converse/anthropic.claude-3-haiku-20240307-v1:0
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: us-east-1
      
  - model_name: prod-claude-haiku
    litellm_params:
      model: bedrock/converse/anthropic.claude-3-haiku-20240307-v1:0
      aws_access_key_id: os.environ/PROD_AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/PROD_AWS_SECRET_ACCESS_KEY
      aws_region_name: us-west-2

🧪 Testing Strategy

This comprehensive test suite covers:

✅ Consistency Tests

Verify that resolvable proxy models behave the same as their direct counterparts:

@pytest.mark.parametrize(\"direct_model,proxy_model,expected_result\", [
    (\"gpt-4\", \"litellm_proxy/gpt-4\", True),
    (\"claude-3-sonnet-20240229\", \"litellm_proxy/claude-3-sonnet-20240229\", True),
])
def test_proxy_function_calling_support_consistency(direct_model, proxy_model, expected_result):
    direct_result = supports_function_calling(direct_model)
    proxy_result = supports_function_calling(proxy_model)
    assert direct_result == proxy_result == expected_result

⚠️ Custom Model Names

Document the expected behavior for unresolvable custom names:

@pytest.mark.parametrize(\"proxy_model_name,underlying_model,expected_proxy_result\", [
    (\"litellm_proxy/bedrock-claude-3-haiku\", \"bedrock/anthropic.claude-3-haiku-20240307-v1:0\", False),
    (\"litellm_proxy/my-gpt4-deployment\", \"azure/gpt-4\", False),
])
def test_proxy_custom_model_names_without_config(proxy_model_name, underlying_model, expected_proxy_result):
    # Without proxy config context, these return False
    proxy_result = supports_function_calling(proxy_model_name)
    assert proxy_result == expected_proxy_result

🏗️ Real-World Bedrock Scenarios

Test actual Bedrock Converse API mappings used in production:

@pytest.mark.parametrize(\"proxy_model_name,underlying_bedrock_model,description\", [
    (\"litellm_proxy/bedrock-claude-3-haiku\", \"bedrock/converse/anthropic.claude-3-haiku-20240307-v1:0\", \"Bedrock Claude 3 Haiku via Converse API\"),
    (\"litellm_proxy/prod-claude-haiku\", \"bedrock/converse/anthropic.claude-3-haiku-20240307-v1:0\", \"Production Claude Haiku\"),
])
def test_bedrock_converse_api_proxy_mappings(proxy_model_name, underlying_bedrock_model, description):
    # Test underlying model works correctly
    underlying_result = supports_function_calling(underlying_bedrock_model)
    assert underlying_result is True  # Claude 3 supports function calling
    
    # Test proxy model returns False without config
    proxy_result = supports_function_calling(proxy_model_name)
    assert proxy_result is False  # Cannot resolve without proxy config

🔧 Changes Made

1. Enhanced Function Calling Tests (`tests/litellm_utils_tests/test_proxy_function_calling.py`)

• Parameterized tests for consistency between direct and proxied models
• Edge case validation and import verification
• Documentation of expected vs. current behavior

2. Improved Model Info Retrieval (`litellm/utils.py`)

• +53 lines of fallback handling for `litellm_proxy` models
• Enhanced support for custom model name handling
• Improved proxy model resolution logic

3. Comprehensive Documentation

• Detailed explanations of proxy model resolution behavior
• Real-world configuration examples
• Usage patterns and limitations
• Future enhancement opportunities

🚀 How to Test

Run all proxy function calling tests:

pytest tests/litellm_utils_tests/test_proxy_function_calling.py -v

Run specific test categories:

# Test consistency between direct and proxy models
pytest tests/litellm_utils_tests/test_proxy_function_calling.py::TestProxyFunctionCalling::test_proxy_function_calling_support_consistency -v

# Test custom model names without config
pytest tests/litellm_utils_tests/test_proxy_function_calling.py::TestProxyFunctionCalling::test_proxy_custom_model_names_without_config -v

# View real-world configuration documentation
pytest tests/litellm_utils_tests/test_proxy_function_calling.py::TestProxyFunctionCalling::test_real_world_proxy_config_documentation -v -s

Quick functional test:

from litellm.utils import supports_function_calling

# ✅ These work (resolvable proxy models):
print(supports_function_calling(\"litellm_proxy/gpt-4\"))  # True
print(supports_function_calling(\"litellm_proxy/claude-3-sonnet-20240229\"))  # True

# ⚠️ These return False (custom names without config):
print(supports_function_calling(\"litellm_proxy/bedrock-claude-3-haiku\"))  # False
print(supports_function_calling(\"litellm_proxy/my-custom-gpt4\"))  # False

# ✅ Direct underlying models work as expected:
print(supports_function_calling(\"bedrock/converse/anthropic.claude-3-haiku-20240307-v1:0\"))  # True
print(supports_function_calling(\"gpt-4\"))  # True

💡 Key Benefits

• Comprehensive test coverage for proxy function calling scenarios
• Fallback handling for proxy models that can be resolved
• Clear documentation of limitations and expected behavior
• Future-proof testing framework for enhancements

🎯 Impact

This enhancement improves the reliability and testability of proxy function calling in LiteLLM while clearly documenting current limitations and providing a foundation for future improvements.

Files Changed:

• `tests/litellm_utils_tests/test_proxy_function_calling.py` (+593 lines)
• `litellm/utils.py` (+53 lines)
• `test_proxy_function_calling.py` (demonstration script)"

[CLAassistant]

All committers have signed the CLA.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 6, 2025 0:39am

[krrishdholakia]

Hey @pazevedo-hyland it's hard to analyze the change due to the formatting changes.

Can you please revert those?

[pazevedo-hyland]

Yes of course @krrishdholakia , check now

[krrishdholakia]

we should avoid adding bloat to this function, as it's quite large already.

I think this can be achieved by updating _get_potential_model_names instead

[pazevedo-hyland]

@krrishdholakia , Okay, ive simplified things a bit and used that method instead. Let me know if it aligns with your idea