Bug:  AWS Bedrock Integration Issues with browser-use

### Browser Use Version

0.8.1

### Bug Description, Steps to Reproduce, Screenshots

# AWS Bedrock Integration Issues with browser-use

## Summary

AWS Bedrock's `converse()` API has fundamental incompatibilities with browser-use's structured output approach, making it unusable for complex agent workflows despite being a critical enterprise LLM provider.

## Environment

- **browser-use version**: 0.8.1
- **Python version**: 3.12
- **LLM Provider**: AWS Bedrock
- **Region**: us-east-1
- **Models Tested**: 
  - `anthropic.claude-3-5-sonnet-20240620-v1:0` (ON_DEMAND)
  - `anthropic.claude-sonnet-4-20250514-v1:0` (INFERENCE_PROFILE only - incompatible)
  - `amazon.nova-pro-v1:0`
  - `mistral.mistral-large-2402-v1:0`
  - `meta.llama3-70b-instruct-v1:0`

## Problem Description

### Issue 1: Schema Format Incompatibility

AWS Bedrock's `converse()` API does not properly handle browser-use's complex union-type action schemas.

**Expected Behavior**: 
Agent should be able to call actions like `click`, `input`, `navigate`, etc. with proper field names.

**Actual Behavior**:
- AWS Bedrock returns incorrect field names: `{"click": {"element": 9}}` instead of `{"click": {"index": 9}}`
- Only the first action type (`done`) appears in the schema sent to Bedrock
- Union types (`anyOf`, `oneOf`) are not properly converted

**Error Output**:
```
ERROR [Agent] ❌ Result failed 1/9 times:
 Failed to validate structured output: 41 validation errors for AgentOutput
action.0.ClickActionModel.click.index
  Field required [type=missing, input_value={'element': 9}, input_type=dict]
action.0.ClickActionModel.click.element
  Extra inputs are not permitted [type=extra_forbidden, input_value={'element': 9}, input_type=dict]
```

**Root Cause**:
The `_format_tools_for_request()` method in `chat_bedrock.py` only processes top-level schema properties and doesn't:
1. Resolve `$ref` pointers to `$defs`
2. Handle complex union types properly
3. Recursively process nested action schemas

**Debug Evidence**:
```json
// Schema sent to AWS Bedrock (INCOMPLETE)
{
  "action": {
    "type": "array",
    "items": {
      "type": "object",
      "properties": {
        "done": { ... }  // ❌ Only 'done' action, missing 'click', 'input', etc.
      },
      "required": ["done"]
    }
  }
}
```

### Issue 2: Extra Fields Validation

AWS Bedrock models (particularly Claude) return reasoning fields alongside structured output:

**Fields returned by AWS Bedrock Claude**:
- `thinking` - Model's reasoning process
- `evaluation_previous_goal` - Assessment of previous step
- `memory` - Context retention
- `next_goal` - Next step planning

**Error**:
```
ERROR [Agent] ❌ Result failed 1/9 times:
 Failed to validate structured output: 165 validation errors for AgentOutput
action.0.DoneActionModel.thinking
  Extra inputs are not permitted [type=extra_forbidden, input_value="<think>...", input_type=str]
```

**Note**: Setting `extra='ignore'` in `AgentOutput.model_config` helps, but doesn't solve the core schema issue.

### Issue 3: toolChoice Configuration

AWS Bedrock requires explicit `toolChoice` parameter to force tool usage:

**Required Configuration**:
```python
body['toolConfig'] = {
    'tools': tools,
    'toolChoice': {'any': {}}  # ❌ Missing in current implementation
}
```

Without this, models return plain text instead of using tools:
```
ERROR [Agent] ❌ Expected structured output but no tool use found in response
```

## Attempted Solutions

### ✅ Solution 1: Add toolChoice (Partial Success)
```python
# In chat_bedrock.py line ~201
body['toolConfig'] = {
    'tools': tools,
    'toolChoice': {'any': {}}  # Force tool usage
}
```
**Result**: Models now attempt to use tools, but schema incompatibility remains.

### ❌ Solution 2: Recursive Schema Converter (Failed)
Attempted to recursively convert Pydantic schemas with `$ref` resolution:
```python
def convert_schema_recursive(schema_obj):
    # Handle $ref, properties, items, anyOf, etc.
    ...
```
**Result**: Still doesn't capture all action types in the schema.

### ⚠️ Solution 3: Response Transformation (Workaround)
Post-process AWS Bedrock responses to fix field names:
```python
def fix_bedrock_response(obj):
    if key == 'click' and 'element' in value:
        return {'index': value['element']}
    # Filter extra fields
    if key in ['thinking', 'evaluation_previous_goal', 'memory', 'next_goal']:
        continue
```
**Result**: Reduces errors but doesn't solve root cause. Schema is still incomplete.

### ❌ Solution 4: ChatAnthropicBedrock (Incompatible)
Tried using Anthropic's native SDK via Bedrock:
```python
llm = ChatAnthropicBedrock(model="anthropic.claude-3-5-sonnet-20240620-v1:0")
```
**Error**: 
```
Error code: 400 - {'message': 'You invoked an unsupported model or your request did not allow prompt caching.'}
```
**Result**: Anthropic SDK tries to use prompt caching which isn't supported by AWS Bedrock.

## Alternative Providers Tested

### ✅ Groq (Partially Working)
- **Working Models**: `meta-llama/llama-4-maverick-17b-128e-instruct`, `openai/gpt-oss-120b`
- **Issue**: JSON parsing errors with some responses (EOF while parsing)
- **Token Limits**: Some models have low TPM limits (6K for Llama 4 Maverick)

### ✅ OpenAI/Gemini (Known Working)
These providers work well with browser-use as documented.

## Impact

This blocks enterprise users who:
- Must use AWS Bedrock for compliance/security
- Have AWS credits but no OpenAI/Anthropic access
- Need to use Claude models through AWS infrastructure

## Proposed Solutions

### Option 1: Enhanced AWS Bedrock Schema Converter (High Priority)

Implement a proper schema converter that:
1. **Resolves all `$ref` pointers** from `$defs`
2. **Handles union types** by including ALL action schemas, not just the first
3. **Recursively processes** nested structures
4. **Maps Pydantic types** to AWS Bedrock's expected format

**Example Implementation**:
```python
def _format_tools_for_request(self, output_format: type[BaseModel]) -> list[dict[str, Any]]:
    schema = output_format.model_json_schema()
    definitions = schema.get('$defs', {})
    
    def resolve_and_flatten_unions(schema_obj):
        """Handle anyOf/oneOf by including all variants"""
        if 'anyOf' in schema_obj:
            # Create a merged schema with all union members
            all_properties = {}
            for variant in schema_obj['anyOf']:
                resolved = resolve_ref(variant) if '$ref' in variant else variant
                if 'properties' in resolved:
                    all_properties.update(resolved['properties'])
            return {'type': 'object', 'properties': all_properties}
        # ... handle other cases
    
    # Apply to full schema
    ...
```

### Option 2: Response Normalization Layer (Medium Priority)

Add a middleware layer that:
1. **Renames fields** (`element` → `index`)
2. **Filters reasoning fields** before validation
3. **Validates against expected schema** and provides clear error messages

### Option 3: Dedicated AWS Bedrock Integration (Ideal)

Create a separate `ChatBedrockConverse` class that:
- Uses simplified schemas compatible with Bedrock
- Maps responses to browser-use's expected format
- Handles provider-specific quirks
- Documents limitations clearly

### Option 4: Documentation (Immediate)

Add clear documentation stating:
```markdown
## AWS Bedrock Limitations

⚠️ **AWS Bedrock's `converse()` API has known compatibility issues with browser-use v0.8.1**

**Issues:**
- Complex action schemas not fully supported
- Field name mismatches (`element` vs `index`)
- Extra reasoning fields cause validation errors

**Recommended Alternatives:**
- Use Anthropic Direct API (not through Bedrock)
- Use Amazon Nova Pro (better Bedrock compatibility)
- Use Groq with Llama 4 models (free tier available)

**Status**: Fix in progress - see issue #XXX
```

## Reproduction Steps

1. **Prerequisites**:
   - AWS account with Bedrock access
   - Configure AWS credentials in `.env` or environment:
     ```bash
     AWS_ACCESS_KEY_ID=your_access_key
     AWS_SECRET_ACCESS_KEY=your_secret_key
     AWS_REGION=us-east-1
     ```
   - Enable Claude 3.5 Sonnet model in AWS Bedrock console

2. **Setup**:
```python
from browser_use import Agent
from browser_use.llm.aws.chat_bedrock import ChatAWSBedrock

llm = ChatAWSBedrock(
    model="anthropic.claude-3-5-sonnet-20240620-v1:0",
    aws_region="us-east-1"
)

agent = Agent(
    task="Navigate to http://localhost:3000 and click on any button",
    llm=llm
)
```

3. **Run**:
```bash
await agent.run()
```

4. **Observe**: Validation errors for field name mismatches

## Additional Context

### Related Issues
- [Link to any related issues if they exist]

### Workarounds Currently Used
1. Manual response transformation in `chat_bedrock.py`
2. Setting `extra='ignore'` in Pydantic models
3. Switching to Groq provider (requires separate API key)

### Files Affected
- `browser_use/llm/aws/chat_bedrock.py` (lines 117-175, 285-330)
- `browser_use/agent/views.py` (line 150 - `AgentOutput` model config)

## Expected Outcome

AWS Bedrock should work seamlessly with browser-use, allowing enterprise users to leverage their AWS infrastructure for browser automation 

<img width="2313" height="547" alt="Image" src="https://github.com/user-attachments/assets/7a2336c1-47c2-48c9-8900-b666d7a70ffd" />

### Failing Python Code

```python
from browser_use import Agent
from dotenv import load_dotenv
import asyncio
from browser_use.llm.aws.chat_bedrock import ChatAWSBedrock


load_dotenv()

async def main():
    llm = ChatAWSBedrock(
        model="anthropic.claude-3-5-sonnet-20240620-v1:0",
        aws_region="us-east-1"
    )

    agent = Agent(
        task="Navigate to http://localhost:3000 and click on any button",
        llm=llm
    )
    
    await agent.run()
    
if __name__ == "__main__":
    asyncio.run(main())
```

### LLM Model

Other (specify in description)

### Operating System & Browser Versions

Ubuntu 24.04.3 LTS (Noble Numbat) | Python 3.12.3 | Google Chrome 141.0.7390.54  | browser-use==0.8.1 | playwright==1.55.0

### Full DEBUG Log Output

```shell
 ⚡ashwith ❯❯ python main.py
INFO     [service] Using anonymized telemetry, see https://docs.browser-use.com/development/telemetry.
INFO     [Agent] 🔗 Found URL in task: http://localhost:3000, adding as initial action...

🚀 Starting agent execution...
============================================================
INFO     [Agent] 🎯 Task: Navigate to http://localhost:3000 and click on any button
INFO     [Agent] Starting a browser-use agent with version 0.8.1, with provider=aws_bedrock and model=anthropic.claude-3-5-sonnet-20240620-v1:0
INFO     [Agent]   ▶️   navigate: url: http://localhost:3000, new_tab: False
INFO     [tools] 🔗 Navigated to http://localhost:3000
INFO     [Agent] 

INFO     [Agent] 📍 Step 1:
ERROR    [Agent] ❌ Result failed 1/9 times:
 Expected structured output but no tool use found in response
INFO     [Agent] 

INFO     [Agent] 📍 Step 2:
ERROR    [Agent] ❌ Result failed 2/9 times:
 Expected structured output but no tool use found in response
INFO     [Agent] 

INFO     [Agent] 📍 Step 3:
ERROR    [Agent] ❌ Result failed 3/9 times:
 Failed to validate structured output: 41 validation errors for AgentOutput
action.0.DoneActionModel.done
  Field required [type=missing, input_value={'click': {'element': 3}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
action.0.DoneActionModel.click
  Extra inputs are not permitted [type=extra_forbidden, input_value={'element': 3}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
action.0.SearchActionModel.search
  Field required [type=missing, input_value={'click': {'element': 3}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
action.0.SearchActionModel.click
  Extra inputs are not permitted [type=extra_forbidden, input_value={'element': 3}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
action.0.NavigateActionModel.navigate
  Field required [type=missing, input_value={'click': {'element': 3}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
action.0.NavigateActionModel.click
  Extra inputs are not permitted [type=extra_forbidden, input_value={'element': 3}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
action.0.GoBackActionModel.go_back
  Field required [type=missing, input_value={'click': {'element': 3}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
action.0.GoBackActionModel.click
  Extra inputs are not permitted [type=extra_forbidden, input_value={'element': 3}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
action.0.WaitActionModel.wait
  Field required [type=missing, input_value={'click': {'element': 3}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
action.0.WaitActionModel.click
  Extra inputs are not permitted [type=extra_forbidden, input_value={'element': 3}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
action.0.ClickActionModel.click.index
  Field required [type=missing, input_value={'element': 3}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
action.0.InputActionModel.input
  Field required [type=missing, input_value={'click': {'element': 3}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
action.0.InputActionModel.click
  Extra inputs are not permitted [type=extra_forbidden, input_value={'element': 3}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
action.0.UploadFileActionModel.upload_file
  Field required [type=missing, input_value={'click': {'element': 3}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
action.0.UploadFileActionModel.click
  Extra inputs are not permitted [type=extra_forbidden, input_value={'element': 3}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
action.0.SwitchActionModel.switch
  Field required [type=missing, input_value={'click': {'element': 3}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
action.0.SwitchActionModel.click
  Extra inputs are not permitted [type=extra_forbidden, input_value={'element': 3}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
action.0.CloseActionModel.close
  Field required [type=missing, input_value={'click': {'element': 3}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
action.0.CloseActionModel.click
  Extra inputs are not permitted [type=extra_forbidden, input_value={'element': 3}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
action.0.ExtractActionModel.extract
  Field required [type=missing, input_value={'click': {'element': 3}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
action.0.ExtractActionModel.click
  Extra inputs are not permitted [type=extra_forbidden, input_value={'element': 3}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
action.0.ScrollActionModel.scroll
  Field required [type=missing, input_value={'click': {'element': 3}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
action.0.ScrollActionModel.click
  Extra inputs are not permitted [type=extra_forbidden, input_value={'element': 3}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
action.0.SendKeysActionModel.send_keys
  Field required [type=missing, input_value={'click': {'element': 3}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
action.0.SendKeysActionModel.click
  Extra inputs are not permitted [type=extra_forbidden, input_value={'element': 3}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
action.0.FindTextActionModel.find_text
  Field required [type=missing, input_value={'click': {'element': 3}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
action.0.FindTextActionModel.click
  Extra inputs are not permitted [type=extra_forbidden, input_value={'element': 3}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
action.0.ScreenshotActionModel.screenshot
  Field required [type=missing, input_value={'click': {'element': 3}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
action.0.ScreenshotActionModel.click
  Extra inputs are not permitted [type=extra_forbidden, input_value={'element': 3}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
action.0.DropdownOptionsActionModel.dropdown_options
  Field required [type=missing, input_value={'click': {'element': 3}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
action.0.DropdownOptionsActionModel.click
  Extra inputs are not permitted [type=extra_forbidden, input_value={'element': 3}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
action.0.SelectDropdownActionModel.select_dropdown
  Field required [type=missing, input_value={'click': {'element': 3}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
action.0.SelectDropdownActionModel.click
  Extra inputs are not permitted [type=extra_forbidden, input_value={'element': 3}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
action.0.WriteFileActionModel.write_file
  Field required [type=missing, input_value={'click': {'element': 3}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
action.0.WriteFileActionModel.click
  Extra inputs are not permitted [type=extra_forbidden, input_value={'element': 3}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
action.0.ReplaceFileActionModel.replace_file
  Field required [type=missing, input_value={'click': {'element': 3}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
action.0.ReplaceFileActionModel.click
  Extra inputs are not permitted [type=extra_forbidden, input_value={'element': 3}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
action.0.ReadFileActionModel.read_file
  Field required [type=missing, input_value={'click': {'element': 3}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
action.0.ReadFileActionModel.click
  Extra inputs are not permitted [type=extra_forbidden, input_value={'element': 3}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
action.0.EvaluateActionModel.evaluate
  Field required [type=missing, input_value={'click': {'element': 3}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
action.0.EvaluateActionModel.click
  Extra inputs are not permitted [type=extra_forbidden, input_value={'element': 3}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
INFO     [Agent]
```

Bug: AWS Bedrock Integration Issues with browser-use #3371

Description

Browser Use Version

Bug Description, Steps to Reproduce, Screenshots

AWS Bedrock Integration Issues with browser-use

Summary

Environment

Problem Description

Issue 1: Schema Format Incompatibility

Issue 2: Extra Fields Validation

Issue 3: toolChoice Configuration

Attempted Solutions

✅ Solution 1: Add toolChoice (Partial Success)

❌ Solution 2: Recursive Schema Converter (Failed)

⚠️ Solution 3: Response Transformation (Workaround)

❌ Solution 4: ChatAnthropicBedrock (Incompatible)

Alternative Providers Tested

✅ Groq (Partially Working)

✅ OpenAI/Gemini (Known Working)

Impact

Proposed Solutions

Option 1: Enhanced AWS Bedrock Schema Converter (High Priority)

Option 2: Response Normalization Layer (Medium Priority)

Option 3: Dedicated AWS Bedrock Integration (Ideal)

Option 4: Documentation (Immediate)

Reproduction Steps

Additional Context

Related Issues

Workarounds Currently Used

Files Affected

Expected Outcome

Failing Python Code

LLM Model

Operating System & Browser Versions

Full DEBUG Log Output

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions