Inconsistent behavior when sensitive data values are present in task but not in sensitive_data dict

[threecoffeeminimum]

Bug Description

When a task contains substitution strings for sensitive values, different behavior occurs depending on whether all the sensitive values are given, one is missing, or multiple are missing.

Consider a prompt that looks like the following:
Enter x_sens1, x_sens2, x_sens3, x_sens4, and x_sens5 into the search bar, all separated by a comma. There is no need to execute the search.

If x_sens2 is missing from the sensitive_data dict, the agent will substitute <secret>x_sens2</secret> along with the values from the sensitive_data dict for the respective substitution strings.

If multiple keys are missing from the sensitive_data dict (e.g. x_sens2 and x_sens4 are missing), the sensitive_data kwarg will be silently ignored, resulting in a task which contains all the substitution strings and not the sensitive values.

Ideally a warning or error should be thrown when substitution string keys are missing from the sensitive_data dict. At a minimum the dict should not be silently ignored.

Reproduction Steps

1. Create a task that contains substitution strings that correspond to the keys in the sensitive_data dict
2. Run the agent
3. Everything works as expected

INFO     [agent] 🚀 Starting task: Enter x_sens1, x_sens2, x_sens3, x_sens4, and x_sens5 into the search bar, all separated by a comma. There is no need to execute the search.
INFO     [browser] Reusing existing Chrome instance
INFO     [controller] 🔗  Opened new tab with https://www.google.com
INFO     [agent] 📍 Step 1
INFO     [agent] 👍 Eval: Success - A new tab was opened with Google search page.
INFO     [agent] 🧠 Memory: A new tab is open with Google.com. I need to enter the placeholders into the search bar.
INFO     [agent] 🎯 Next goal: Enter the placeholders 'x_sens1, x_sens2, x_sens3, x_sens4, x_sens5' into the search bar.
INFO     [agent] 🛠️  Action 1/1: {"input_text":{"index":9,"text":"<secret>x_sens1</secret>, <secret>x_sens2</secret>, <secret>x_sens3</secret>, <secret>x_sens4</secret>, <secret>x_sens5</secret>"}}
INFO     [controller] ⌨️  Input test1, test2, test3, test4, test5 into index 9

1. Use the same task from the first run
2. Delete one of the keys from the sensitive_data dict
3. Run the agent
4. One of the substitution strings is wrapped in tags and injected instead of a value

INFO     [agent] 🚀 Starting task: Enter x_sens1, x_sens2, x_sens3, x_sens4, and x_sens5 into the search bar, all separated by a comma. There is no need to execute the search.
INFO     [controller] 🔗  Opened new tab with https://www.google.com
INFO     [agent] 📍 Step 1
INFO     [agent] 👍 Eval: Success - A new tab with Google was opened successfully.
INFO     [agent] 🧠 Memory: Opened Google homepage. Ready to input search query.
INFO     [agent] 🎯 Next goal: Input the search terms into Google search bar.
INFO     [agent] 🛠️  Action 1/1: {"input_text":{"index":9,"text":"<secret>x_sens1</secret>, <secret>x_sens2</secret>, <secret>x_sens3</secret>, <secret>x_sens4</secret>, <secret>x_sens5</secret>","xpath":null}}
INFO     [controller] ⌨️  Input test1, <secret>x_sens2</secret>, test3, test4, test5 into index 9

1. Use the same task from the first run
2. Delete multiple keys from the sensitive_data dict
3. Run the agent
4. All of the sensitive_data dict is ignored

INFO     [browser_use] BrowserUse logging setup complete with level info
INFO     [root] Anonymized telemetry enabled. See https://docs.browser-use.com/development/telemetry for more information.
INFO     [agent] 🚀 Starting task: Enter x_sens1, x_sens2, x_sens3, x_sens4, and x_sens5 into the search bar, all separated by a comma. There is no need to execute the search.
INFO     [controller] 🔗  Opened new tab with https://www.google.com
INFO     [agent] 📍 Step 1
INFO     [agent] 👍 Eval: Success - Google search page is open and ready for input.
INFO     [agent] 🧠 Memory: Google search page is open. Need to enter 'x_sens1, x_sens2, x_sens3, x_sens4, x_sens5' into the search bar.
INFO     [agent] 🎯 Next goal: Input the given text into the search bar.
INFO     [agent] 🛠️  Action 1/1: {"input_text":{"index":9,"text":"x_sens1, x_sens2, x_sens3, x_sens4, x_sens5"}}
INFO     [controller] ⌨️  Input x_sens1, x_sens2, x_sens3, x_sens4, x_sens5 into index 9

Code Sample

task = 'Enter x_sens1, x_sens2, x_sens3, x_sens4, and x_sens5 into the search bar, all separated by a comma. There is no need to execute the search.'

sensitive_data = {'x_sens1': 'test1', 'x_sens3': 'test3', 'x_sens5': 'test5'}

async def main():

    browser = Browser(
        config=BrowserConfig(
            # NOTE: you need to close your chrome browser - so that this can open your browser in debug mode
            chrome_instance_path='/usr/bin/google-chrome',
        )
    )


    initial_actions = [
        {'open_tab': {'url': 'https://www.google.com'}},
    ]
    agent = Agent(
        initial_actions=initial_actions,
        task=task,
        llm=ChatOpenAI(model='gpt-4o'),
        browser=browser,
        sensitive_data=sensitive_data,
    )

    history = await agent.run()

Version

0.1.36

LLM Model

GPT-4o

Operating System

Pop!_OS 22.04 LTS

Relevant Log Output