[Bug]: Presidio guardrail doesn't scrub effective request and response

[btiernay]

What happened?

I'm noticing that when I run LiteLLM with Presidio and AWS Bedrock via Open WebUI, while the logs show that things are being redacted, when I check my cloud logs (CloudWatch) things are not.

Current settings:

# See https://docs.litellm.ai/docs/proxy/guardrails/pii_masking_v2
guardrails:
- guardrail_name: "presidio-pre-guard"
  litellm_params:
    guardrail: presidio
    mode: "pre_call"
    output_parse_pii: True

When I enter in the following prompt using Open WebUI:

My name is Bobby. What is my name backwards?

I get these logs from LiteLLM:

litellm-1              | 22:43:39 - LiteLLM Proxy:DEBUG: model_checks.py:85 - ALL KEY MODELS - 0
litellm-1              | 22:43:39 - LiteLLM Proxy:DEBUG: model_checks.py:112 - ALL TEAM MODELS - 0
litellm-1              | INFO:     172.18.0.5:53562 - "GET /models HTTP/1.1" 200 OK
litellm-1              | 22:43:39 - LiteLLM Proxy:DEBUG: proxy_server.py:3436 - Request received by LiteLLM:
litellm-1              | {
litellm-1              |     "stream": true,

litellm-1              | 22:43:39 - LiteLLM Proxy:DEBUG: model_checks.py:85 - ALL KEY MODELS - 0
litellm-1              | 22:43:39 - LiteLLM Proxy:DEBUG: model_checks.py:112 - ALL TEAM MODELS - 0
litellm-1              | INFO:     172.18.0.5:53562 - "GET /models HTTP/1.1" 200 OK
litellm-1              | 22:43:39 - LiteLLM Proxy:DEBUG: proxy_server.py:3436 - Request received by LiteLLM:
litellm-1              | {
litellm-1              |     "stream": true,
litellm-1              |     "model": "bedrock-claude-3.5-sonnet",
litellm-1              |     "messages": [
litellm-1              |         {
litellm-1              |             "role": "user",
litellm-1              |             "content": "."
litellm-1              |         },
litellm-1              |         {
litellm-1              |             "role": "assistant",
litellm-1              |             "content": "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind."
litellm-1              |         },
litellm-1              |         {
litellm-1              |             "role": "user",
litellm-1              |             "content": "My name is Bobby. What is my name backwards?"
litellm-1              |         }
litellm-1              |     ]
litellm-1              | }
litellm-1              | 22:43:39 - LiteLLM Proxy:DEBUG: litellm_pre_call_utils.py:450 - Request Headers: Headers({'host': 'litellm:4000', 'authorization': 'Bearer anything', 'content-type': 'application/json', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'user-agent': 'Python/3.11 aiohttp/3.11.8', 'content-length': '396'})
litellm-1              | 22:43:39 - LiteLLM Proxy:DEBUG: litellm_pre_call_utils.py:456 - receiving data: {'stream': True, 'model': 'bedrock-claude-3.5-sonnet', 'messages': [{'role': 'user', 'content': '.'}, {'role': 'assistant', 'content': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind."}, {'role': 'user', 'content': 'My name is Bobby. What is my name backwards?'}], 'proxy_server_request': {'url': 'http://litellm:4000/chat/completions', 'method': 'POST', 'headers': {'host': 'litellm:4000', 'content-type': 'application/json', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'user-agent': 'Python/3.11 aiohttp/3.11.8', 'content-length': '396'}, 'body': {'stream': True, 'model': 'bedrock-claude-3.5-sonnet', 'messages': [{'role': 'user', 'content': '.'}, {'role': 'assistant', 'content': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind."}, {'role': 'user', 'content': 'My name is Bobby. What is my name backwards?'}]}}}
litellm-1              | 22:43:39 - LiteLLM Proxy:DEBUG: litellm_pre_call_utils.py:607 - [PROXY] returned data from litellm_pre_call_utils: {'stream': True, 'model': 'bedrock-claude-3.5-sonnet', 'messages': [{'role': 'user', 'content': '.'}, {'role': 'assistant', 'content': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind."}, {'role': 'user', 'content': 'My name is Bobby. What is my name backwards?'}], 'proxy_server_request': {'url': 'http://litellm:4000/chat/completions', 'method': 'POST', 'headers': {'host': 'litellm:4000', 'content-type': 'application/json', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'user-agent': 'Python/3.11 aiohttp/3.11.8', 'content-length': '396'}, 'body': {'stream': True, 'model': 'bedrock-claude-3.5-sonnet', 'messages': [{'role': 'user', 'content': '.'}, {'role': 'assistant', 'content': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind."}, {'role': 'user', 'content': 'My name is Bobby. What is my name backwards?'}]}}, 'metadata': {'requester_metadata': {}, 'user_api_key_hash': 'anything', 'user_api_key_alias': None, 'user_api_key_team_id': None, 'user_api_key_user_id': None, 'user_api_key_org_id': None, 'user_api_key_team_alias': None, 'user_api_key_end_user_id': None, 'user_api_key': 'anything', 'user_api_end_user_max_budget': None, 'litellm_api_version': '1.59.7', 'global_max_parallel_requests': None, 'user_api_key_team_max_budget': None, 'user_api_key_team_spend': None, 'user_api_key_spend': 0.0, 'user_api_key_max_budget': None, 'user_api_key_model_max_budget': {}, 'user_api_key_metadata': {}, 'headers': {'host': 'litellm:4000', 'content-type': 'application/json', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'user-agent': 'Python/3.11 aiohttp/3.11.8', 'content-length': '396'}, 'endpoint': 'http://litellm:4000/chat/completions', 'litellm_parent_otel_span': None, 'requester_ip_address': ''}}
litellm-1              | 22:43:39 - LiteLLM Proxy:DEBUG: utils.py:476 - Inside Proxy Logging Pre-call hook!
litellm-1              | 22:43:39 - LiteLLM Proxy:DEBUG: parallel_request_limiter.py:48 - Inside Max Parallel Request Pre-Call Hook
litellm-1              | 22:43:39 - LiteLLM Proxy:DEBUG: parallel_request_limiter.py:48 - current: {'current_requests': 0, 'current_tpm': 464, 'current_rpm': 1}
litellm-1              | 22:43:39 - LiteLLM Proxy:DEBUG: max_budget_limiter.py:23 - Inside Max Budget Limiter Pre-Call Hook
litellm-1              | 22:43:39 - LiteLLM Proxy:DEBUG: cache_control_check.py:27 - Inside Cache Control Check Pre-Call Hook
litellm-1              | 22:43:39 - LiteLLM Router:DEBUG: router.py:3063 - Inside async function with retries: args - (); kwargs - {'proxy_server_request': {'url': 'http://litellm:4000/chat/completions', 'method': 'POST', 'headers': {'host': 'litellm:4000', 'content-type': 'application/json', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'user-agent': 'Python/3.11 aiohttp/3.11.8', 'content-length': '396'}, 'body': {'stream': True, 'model': 'bedrock-claude-3.5-sonnet', 'messages': [{'role': 'user', 'content': '.'}, {'role': 'assistant', 'content': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind."}, {'role': 'user', 'content': 'My name is Bobby. What is my name backwards?'}]}}, 'metadata': {'requester_metadata': {}, 'user_api_key_hash': 'anything', 'user_api_key_alias': None, 'user_api_key_team_id': None, 'user_api_key_user_id': None, 'user_api_key_org_id': None, 'user_api_key_team_alias': None, 'user_api_key_end_user_id': None, 'user_api_key': 'anything', 'user_api_end_user_max_budget': None, 'litellm_api_version': '1.59.7', 'global_max_parallel_requests': None, 'user_api_key_team_max_budget': None, 'user_api_key_team_spend': None, 'user_api_key_spend': 0.0, 'user_api_key_max_budget': None, 'user_api_key_model_max_budget': {}, 'user_api_key_metadata': {}, 'headers': {'host': 'litellm:4000', 'content-type': 'application/json', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'user-agent': 'Python/3.11 aiohttp/3.11.8', 'content-length': '396'}, 'endpoint': 'http://litellm:4000/chat/completions', 'litellm_parent_otel_span': None, 'requester_ip_address': '', 'model_group': 'bedrock-claude-3.5-sonnet'}, 'litellm_call_id': '9cc93ce7-0c76-4d1c-9b29-e1947307a4e7', 'litellm_logging_obj': <litellm.litellm_core_utils.litellm_logging.Logging object at 0xffff9917d350>, 'model': 'bedrock-claude-3.5-sonnet', 'messages': [{'role': 'user', 'content': '.'}, {'role': 'assistant', 'content': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind."}, {'role': 'user', 'content': 'My name is Bobby. What is my name backwards?'}], 'stream': True, 'original_function': <bound method Router._acompletion of <litellm.router.Router object at 0xffff9d65c980>>, 'num_retries': 2, 'litellm_trace_id': '74ad16cd-5c93-40d5-8868-4c9a628b3328'}
litellm-1              | 22:43:39 - LiteLLM Router:DEBUG: router.py:3085 - async function w/ retries: original_function - <bound method Router._acompletion of <litellm.router.Router object at 0xffff9d65c980>>, num_retries - 2
litellm-1              | 22:43:39 - LiteLLM Router:DEBUG: router.py:868 - Inside _acompletion()- model: bedrock-claude-3.5-sonnet; kwargs: {'proxy_server_request': {'url': 'http://litellm:4000/chat/completions', 'method': 'POST', 'headers': {'host': 'litellm:4000', 'content-type': 'application/json', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'user-agent': 'Python/3.11 aiohttp/3.11.8', 'content-length': '396'}, 'body': {'stream': True, 'model': 'bedrock-claude-3.5-sonnet', 'messages': [{'role': 'user', 'content': '.'}, {'role': 'assistant', 'content': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind."}, {'role': 'user', 'content': 'My name is Bobby. What is my name backwards?'}]}}, 'metadata': {'requester_metadata': {}, 'user_api_key_hash': 'anything', 'user_api_key_alias': None, 'user_api_key_team_id': None, 'user_api_key_user_id': None, 'user_api_key_org_id': None, 'user_api_key_team_alias': None, 'user_api_key_end_user_id': None, 'user_api_key': 'anything', 'user_api_end_user_max_budget': None, 'litellm_api_version': '1.59.7', 'global_max_parallel_requests': None, 'user_api_key_team_max_budget': None, 'user_api_key_team_spend': None, 'user_api_key_spend': 0.0, 'user_api_key_max_budget': None, 'user_api_key_model_max_budget': {}, 'user_api_key_metadata': {}, 'headers': {'host': 'litellm:4000', 'content-type': 'application/json', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'user-agent': 'Python/3.11 aiohttp/3.11.8', 'content-length': '396'}, 'endpoint': 'http://litellm:4000/chat/completions', 'litellm_parent_otel_span': None, 'requester_ip_address': '', 'model_group': 'bedrock-claude-3.5-sonnet', 'model_group_size': 1}, 'litellm_call_id': '9cc93ce7-0c76-4d1c-9b29-e1947307a4e7', 'litellm_logging_obj': <litellm.litellm_core_utils.litellm_logging.Logging object at 0xffff9917d350>, 'stream': True, 'litellm_trace_id': '74ad16cd-5c93-40d5-8868-4c9a628b3328'}
litellm-1              | 22:43:39 - LiteLLM Router:DEBUG: router.py:5359 - initial list of deployments: [{'model_name': 'bedrock-claude-3.5-sonnet', 'litellm_params': {'model': 'bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0'}, 'model_info': {'id': 'f2ba4839a7858dfafc0cbb29ebbdbb92d9f0817d4af578395aa2264022bf3412', 'db_model': False}}]
litellm-1              | 22:43:39 - LiteLLM Router:DEBUG: cooldown_handlers.py:310 - retrieve cooldown models: []
litellm-1              | 22:43:39 - LiteLLM Router:DEBUG: router.py:5420 - async cooldown deployments: []
litellm-1              | 22:43:39 - LiteLLM Router:DEBUG: router.py:5423 - cooldown_deployments: []
litellm-1              | 22:43:39 - LiteLLM Router:DEBUG: router.py:5702 - cooldown deployments: []
litellm-1              | 22:43:39 - LiteLLM Router:INFO: router.py:963 - litellm.acompletion(model=bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0) 200 OK
litellm-1              | 22:43:39 - LiteLLM Router:DEBUG: router.py:2828 - Async Response: <litellm.litellm_core_utils.streaming_handler.CustomStreamWrapper object at 0xffff9917f100>
litellm-1              | INFO:     172.18.0.5:53568 - "POST /chat/completions HTTP/1.1" 200 OK
litellm-1              | 22:43:39 - LiteLLM Proxy:DEBUG: proxy_server.py:2907 - inside generator
webui-1                | INFO:     172.18.0.1:64910 - "POST /api/chat/completions HTTP/1.1" 200 OK
webui-1                | INFO:     172.18.0.1:64910 - "GET /api/v1/chats/?page=1 HTTP/1.1" 200 OK
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: proxy_server.py:2911 - async_data_generator: received streaming chunk - ModelResponseStream(id='chatcmpl-7e2799e1-42f4-4df0-ba87-1fa831f15db1', created=1737931420, model='anthropic.claude-3-5-sonnet-20240620-v1:0', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields={}, content='Your name backwards is', role='assistant', function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options=None)
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': '.', 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: proxy_server.py:2911 - async_data_generator: received streaming chunk - ModelResponseStream(id='chatcmpl-7e2799e1-42f4-4df0-ba87-1fa831f15db1', created=1737931420, model='anthropic.claude-3-5-sonnet-20240620-v1:0', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields={}, content=':', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options=None)
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind.", 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': '.', 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: proxy_server.py:2911 - async_data_generator: received streaming chunk - ModelResponseStream(id='chatcmpl-7e2799e1-42f4-4df0-ba87-1fa831f15db1', created=1737931420, model='anthropic.claude-3-5-sonnet-20240620-v1:0', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields={}, content='\n\nybboB', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options=None)
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': 'My name is Bobby. What is my name backwards?', 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': '.', 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind.", 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: proxy_server.py:2911 - async_data_generator: received streaming chunk - ModelResponseStream(id='chatcmpl-7e2799e1-42f4-4df0-ba87-1fa831f15db1', created=1737931420, model='anthropic.claude-3-5-sonnet-20240620-v1:0', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason='stop', index=0, delta=Delta(provider_specific_fields={}, content=None, role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options=None)
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind.", 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': 'My name is Bobby. What is my name backwards?', 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': 'My name is Bobby. What is my name backwards?', 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': '.', 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind.", 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': 'My name is Bobby. What is my name backwards?', 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: model_max_budget_limiter.py:151 - in RouterBudgetLimiting.async_log_success_event
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: model_max_budget_limiter.py:167 - Not running _PROXY_VirtualKeyModelMaxBudgetLimiter.async_log_success_event because user_api_key_model_max_budget is None or empty. `user_api_key_model_max_budget`={}
presidio-analyzer-1    | 2025-01-26 22:43:40,204 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: parallel_request_limiter.py:48 - INSIDE parallel request limiter ASYNC SUCCESS LOGGING
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: parallel_request_limiter.py:48 - updated_value in success call: {'current_requests': 0, 'current_tpm': 539, 'current_rpm': 2}, precise_minute: 2025-01-26-22-43
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: parallel_request_limiter.py:48 - updated_value in success call: {'current_requests': 0, 'current_tpm': 539, 'current_rpm': 2}, precise_minute: 2025-01-26-22-43
webui-1                | INFO:     172.18.0.1:64910 - "POST /api/chat/completed HTTP/1.1" 200 OK
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
presidio-analyzer-1    | 2025-01-26 22:43:40,224 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind.", 'items': []}
webui-1                | INFO:     172.18.0.1:64910 - "POST /api/v1/chats/27a964a1-443f-4668-84ca-574bf3e39ccb HTTP/1.1" 200 OK
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
presidio-analyzer-1    | 2025-01-26 22:43:40,229 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': '.', 'items': []}
webui-1                | INFO:     172.18.0.1:64910 - "GET /api/v1/chats/?page=1 HTTP/1.1" 200 OK
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
presidio-analyzer-1    | 2025-01-26 22:43:40,236 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': 'My name is <PERSON>. What is my name backwards?', 'items': [{'start': 11, 'end': 19, 'entity_type': 'PERSON', 'text': '<PERSON>', 'operator': 'replace'}]}
litellm-1              | 22:43:40 - LiteLLM Proxy:INFO: presidio.py:335 - Presidio PII Masking: Redacted pii message: [{'role': 'user', 'content': '.'}, {'role': 'assistant', 'content': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind."}, {'role': 'user', 'content': 'My name is <PERSON>. What is my name backwards?'}]
litellm-1              | 22:43:40 - LiteLLM:WARNING: custom_guardrail.py:165 - Guardrail Tracing is only available for premium users. Skipping guardrail logging for guardrail=presidio-pre-guard event_hook=pre_call
litellm-1              | 22:43:40 - LiteLLM:WARNING: custom_guardrail.py:165 - Guardrail Tracing is only available for premium users. Skipping guardrail logging for guardrail=presidio-pre-guard event_hook=pre_call
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': '.', 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind.", 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': 'My name is <PERSON>. What is my name backwards?', 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
presidio-analyzer-1    | 2025-01-26 22:43:40,242 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': '.', 'items': []}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
presidio-analyzer-1    | 2025-01-26 22:43:40,253 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind.", 'items': []}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
presidio-analyzer-1    | 2025-01-26 22:43:40,264 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind.", 'items': []}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
presidio-analyzer-1    | 2025-01-26 22:43:40,269 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': '.', 'items': []}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
presidio-analyzer-1    | 2025-01-26 22:43:40,274 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': '.', 'items': []}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
presidio-analyzer-1    | 2025-01-26 22:43:40,280 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': 'My name is <PERSON>. What is my name backwards?', 'items': [{'start': 11, 'end': 19, 'entity_type': 'PERSON', 'text': '<PERSON>', 'operator': 'replace'}]}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
presidio-analyzer-1    | 2025-01-26 22:43:40,291 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind.", 'items': []}
litellm-1              | 22:43:40 - LiteLLM Proxy:INFO: presidio.py:335 - Presidio PII Masking: Redacted pii message: [{'role': 'user', 'content': '.'}, {'role': 'assistant', 'content': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind."}, {'role': 'user', 'content': 'My name is <PERSON>. What is my name backwards?'}]
litellm-1              | 22:43:40 - LiteLLM:WARNING: custom_guardrail.py:165 - Guardrail Tracing is only available for premium users. Skipping guardrail logging for guardrail=presidio-pre-guard event_hook=pre_call
litellm-1              | 22:43:40 - LiteLLM:WARNING: custom_guardrail.py:165 - Guardrail Tracing is only available for premium users. Skipping guardrail logging for guardrail=presidio-pre-guard event_hook=pre_call
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': '.', 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind.", 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': 'My name is <PERSON>. What is my name backwards?', 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
presidio-analyzer-1    | 2025-01-26 22:43:40,298 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': 'My name is <PERSON>. What is my name backwards?', 'items': [{'start': 11, 'end': 19, 'entity_type': 'PERSON', 'text': '<PERSON>', 'operator': 'replace'}]}
litellm-1              | 22:43:40 - LiteLLM Proxy:INFO: presidio.py:335 - Presidio PII Masking: Redacted pii message: [{'role': 'user', 'content': '.'}, {'role': 'assistant', 'content': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind."}, {'role': 'user', 'content': 'My name is <PERSON>. What is my name backwards?'}]
litellm-1              | 22:43:40 - LiteLLM:WARNING: custom_guardrail.py:165 - Guardrail Tracing is only available for premium users. Skipping guardrail logging for guardrail=presidio-pre-guard event_hook=pre_call
litellm-1              | 22:43:40 - LiteLLM:WARNING: custom_guardrail.py:165 - Guardrail Tracing is only available for premium users. Skipping guardrail logging for guardrail=presidio-pre-guard event_hook=pre_call
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': '.', 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind.", 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': 'My name is <PERSON>. What is my name backwards?', 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
presidio-analyzer-1    | 2025-01-26 22:43:40,304 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': 'My name is <PERSON>. What is my name backwards?', 'items': [{'start': 11, 'end': 19, 'entity_type': 'PERSON', 'text': '<PERSON>', 'operator': 'replace'}]}
litellm-1              | 22:43:40 - LiteLLM Proxy:INFO: presidio.py:335 - Presidio PII Masking: Redacted pii message: [{'role': 'user', 'content': '.'}, {'role': 'assistant', 'content': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind."}, {'role': 'user', 'content': 'My name is <PERSON>. What is my name backwards?'}]
litellm-1              | 22:43:40 - LiteLLM:WARNING: custom_guardrail.py:165 - Guardrail Tracing is only available for premium users. Skipping guardrail logging for guardrail=presidio-pre-guard event_hook=pre_call
litellm-1              | 22:43:40 - LiteLLM:WARNING: custom_guardrail.py:165 - Guardrail Tracing is only available for premium users. Skipping guardrail logging for guardrail=presidio-pre-guard event_hook=pre_call
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': '.', 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind.", 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:163 - Making request to: http://presidio-analyzer:5002/analyze with payload: {'text': 'My name is <PERSON>. What is my name backwards?', 'language': 'en'}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
presidio-analyzer-1    | 2025-01-26 22:43:40,315 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind.", 'items': []}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
presidio-analyzer-1    | 2025-01-26 22:43:40,337 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': '.', 'items': []}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
presidio-analyzer-1    | 2025-01-26 22:43:40,343 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': 'My name is <PERSON>. What is my name backwards?', 'items': []}
litellm-1              | 22:43:40 - LiteLLM Proxy:INFO: presidio.py:335 - Presidio PII Masking: Redacted pii message: [{'role': 'user', 'content': '.'}, {'role': 'assistant', 'content': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind."}, {'role': 'user', 'content': 'My name is <PERSON>. What is my name backwards?'}]
litellm-1              | 22:43:40 - LiteLLM:WARNING: custom_guardrail.py:165 - Guardrail Tracing is only available for premium users. Skipping guardrail logging for guardrail=presidio-pre-guard event_hook=post_call
litellm-1              | 22:43:40 - LiteLLM:WARNING: custom_guardrail.py:165 - Guardrail Tracing is only available for premium users. Skipping guardrail logging for guardrail=presidio-pre-guard event_hook=post_call
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
presidio-analyzer-1    | 2025-01-26 22:43:40,348 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': '.', 'items': []}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
presidio-analyzer-1    | 2025-01-26 22:43:40,358 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind.", 'items': []}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
presidio-analyzer-1    | 2025-01-26 22:43:40,365 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': 'My name is <PERSON>. What is my name backwards?', 'items': []}
litellm-1              | 22:43:40 - LiteLLM Proxy:INFO: presidio.py:335 - Presidio PII Masking: Redacted pii message: [{'role': 'user', 'content': '.'}, {'role': 'assistant', 'content': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind."}, {'role': 'user', 'content': 'My name is <PERSON>. What is my name backwards?'}]
litellm-1              | 22:43:40 - LiteLLM:WARNING: custom_guardrail.py:165 - Guardrail Tracing is only available for premium users. Skipping guardrail logging for guardrail=presidio-pre-guard event_hook=post_call
litellm-1              | 22:43:40 - LiteLLM:WARNING: custom_guardrail.py:165 - Guardrail Tracing is only available for premium users. Skipping guardrail logging for guardrail=presidio-pre-guard event_hook=post_call
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
presidio-analyzer-1    | 2025-01-26 22:43:40,375 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind.", 'items': []}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
presidio-analyzer-1    | 2025-01-26 22:43:40,380 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': '.', 'items': []}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
presidio-analyzer-1    | 2025-01-26 22:43:40,386 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': 'My name is <PERSON>. What is my name backwards?', 'items': []}
litellm-1              | 22:43:40 - LiteLLM Proxy:INFO: presidio.py:335 - Presidio PII Masking: Redacted pii message: [{'role': 'user', 'content': '.'}, {'role': 'assistant', 'content': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind."}, {'role': 'user', 'content': 'My name is <PERSON>. What is my name backwards?'}]
litellm-1              | 22:43:40 - LiteLLM:WARNING: custom_guardrail.py:165 - Guardrail Tracing is only available for premium users. Skipping guardrail logging for guardrail=presidio-pre-guard event_hook=post_call
litellm-1              | 22:43:40 - LiteLLM:WARNING: custom_guardrail.py:165 - Guardrail Tracing is only available for premium users. Skipping guardrail logging for guardrail=presidio-pre-guard event_hook=post_call
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
presidio-analyzer-1    | 2025-01-26 22:43:40,390 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': '.', 'items': []}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
presidio-analyzer-1    | 2025-01-26 22:43:40,400 - presidio-analyzer - INFO - Fetching all recognizers for language en
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind.", 'items': []}
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:176 - Making request to: http://presidio-anonymizer:5001/anonymize
litellm-1              | 22:43:40 - LiteLLM Proxy:DEBUG: presidio.py:189 - redacted_text: {'text': 'My name is <PERSON>. What is my name backwards?', 'items': []}
litellm-1              | 22:43:40 - LiteLLM Proxy:INFO: presidio.py:335 - Presidio PII Masking: Redacted pii message: [{'role': 'user', 'content': '.'}, {'role': 'assistant', 'content': "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind."}, {'role': 'user', 'content': 'My name is <PERSON>. What is my name backwards?'}]
litellm-1              | 22:43:40 - LiteLLM:WARNING: custom_guardrail.py:165 - Guardrail Tracing is only available for premium users. Skipping guardrail logging for guardrail=presidio-pre-guard event_hook=post_call
litellm-1              | 22:43:40 - LiteLLM:WARNING: custom_guardrail.py:165 - Guardrail Tracing is only available for premium users. Skipping guardrail logging for guardrail=presidio-pre-guard event_hook=post_call

And from AWS Bedrock's model invocation logs in CloudWatch in the backend:

{
    "schemaType": "ModelInvocationLog",
    "schemaVersion": "1.0",
    "timestamp": "2025-01-26T22:43:39Z",
    "accountId": "<snip>",
    "identity": {
        "arn": "arn:aws:sts::<snip>:assumed-role/user-admin/<snip>"
    },
    "region": "us-west-2",
    "requestId": "f8c97cd5-9009-46bf-9110-24b34d4283dd",
    "operation": "ConverseStream",
    "modelId": "anthropic.claude-3-5-sonnet-20240620-v1:0",
    "input": {
        "inputContentType": "application/json",
        "inputBodyJson": {
            "messages": [
                {
                    "role": "user",
                    "content": [
                        {
                            "text": "."
                        }
                    ]
                },
                {
                    "role": "assistant",
                    "content": [
                        {
                            "text": "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind."
                        }
                    ]
                },
                {
                    "role": "user",
                    "content": [
                        {
                            "text": "My name is Bobby. What is my name backwards?"
                        }
                    ]
                }
            ],
            "system": [],
            "inferenceConfig": {},
            "additionalModelRequestFields": {}
        },
        "inputTokenCount": 65
    },
    "output": {
        "outputContentType": "application/json",
        "outputBodyJson": {
            "output": {
                "message": {
                    "role": "assistant",
                    "content": [
                        {
                            "text": "Your name backwards is:\n\nybboB"
                        }
                    ]
                }
            },
            "stopReason": "end_turn",
            "metrics": {
                "latencyMs": 908
            },
            "usage": {
                "inputTokens": 65,
                "outputTokens": 13,
                "totalTokens": 78
            }
        },
        "outputTokenCount": 13
    }
}

Note that the text isn't redacted. The ultimate response is this:

Your name backwards is "ybboB"

This should not be possible.

See this Discord thread for initial discussion

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

ghcr.io/berriai/litellm@sha256:1d0c5293526b55569e0d98e20e395c239452f017562b3eee3a8ce1590e3fdfa7

Twitter / LinkedIn details

No response

[btiernay]

When analyzing with Claude Code:

Bug Description

The Presidio guardrail in LiteLLM successfully redacts PII (Personally Identifiable Information) in logs, but fails to redact PII in the actual request sent to the LLM provider (AWS Bedrock in this case).

Root Cause

The bug is in the presidio.py file in the _OPTIONAL_PresidioPIIMasking class. The async_pre_call_hook method (lines 211-261) properly redacts PII in the input messages by updating data["messages"], but this change isn't propagating to the actual request sent to the backend LLM API.

The issue occurs because:

1. The proxy correctly logs redacted content (showing My name is <PERSON>)
2. However, AWS CloudWatch logs from Bedrock show the original content with PII (My name is Bobby)

Technical Details

Looking at the request flow:

1. The async_pre_call_hook modifies the data object that's passed to it
2. This modification affects the copy used for logging purposes
3. But the actual request payload sent to Bedrock is using a separate data structure or copy
4. The CloudWatch logs show unredacted messages in the inputBodyJson field

Suggested Fix

Create a PR with the following changes:

@log_guardrail_information
async def async_pre_call_hook(
    self,
    user_api_key_dict: UserAPIKeyAuth,
    cache: DualCache,
    data: dict,
    call_type: str,
):
    """
    - Check if request turned off pii
        - Check if user allowed to turn off pii (key permissions -> 'allow_pii_controls')

    - Take the request data
    - Call /analyze -> get the results
    - Call /anonymize w/ the analyze results -> get the redacted text

    For multiple messages in /chat/completions, we'll need to call them in parallel.
    """

    try:
        content_safety = data.get("content_safety", None)
        verbose_proxy_logger.debug("content_safety: %s", content_safety)
        presidio_config = self.get_presidio_settings_from_request_data(data)

        if call_type == "completion":  # /chat/completions requests
            messages = data["messages"]
            tasks = []

            for m in messages:
                if isinstance(m["content"], str):
                    tasks.append(
                        self.check_pii(
                            text=m["content"],
                            output_parse_pii=self.output_parse_pii,
                            presidio_config=presidio_config,
                            request_data=data,
                        )
                    )
            responses = await asyncio.gather(*tasks)
            for index, r in enumerate(responses):
                if isinstance(messages[index]["content"], str):
                    messages[index]["content"] = r  # replace content with redacted string
            verbose_proxy_logger.info(
                f"Presidio PII Masking: Redacted pii message: {data['messages']}"
            )

            # FIX: Update the messages in the proxy_server_request body
            # This is the critical section that was missing. The proxy architecture
            # has two copies of the messages:
            # 1. data["messages"] - Used for logging and internal operations
            # 2. data["proxy_server_request"]["body"]["messages"] - The actual payload
            #    sent to the LLM provider (AWS Bedrock, OpenAI, etc.)
            #
            # We need to make sure both are updated with redacted content.
            # Using deepcopy to ensure we don't have any reference issues.
            if "proxy_server_request" in data and "body" in data["proxy_server_request"]:
                verbose_proxy_logger.debug("Updating proxy_server_request body with redacted messages")
                if "messages" in data["proxy_server_request"]["body"]:
                    # Update the payload that will be sent to the LLM provider
                    data["proxy_server_request"]["body"]["messages"] = copy.deepcopy(messages)
                    verbose_proxy_logger.debug(
                        f"Updated proxy_server_request body messages: {data['proxy_server_request']['body']['messages']}"
                    )
            else:
                verbose_proxy_logger.warning(
                    "proxy_server_request or body not found in data, PII may not be properly masked in the request to the LLM provider"
                )

            data["messages"] = messages
        return data
    except Exception as e:
        verbose_proxy_logger.exception(f"Error in Presidio PII masking: {str(e)}")

This change ensures that when Presidio redacts PII in the messages, it also updates the same messages in the proxy_server_request["body"] object, which is what's actually sent to the LLM provider.

Testing Recommendation

Add the following unit test to verify both message paths are updated:

  @pytest.mark.asyncio
  async def test_presidio_updates_effective_request_payload():
      """
      Test that Presidio PII masking updates both data["messages"] and
      data["proxy_server_request"]["body"]["messages"]
      """
      pii_masking = _OPTIONAL_PresidioPIIMasking(
          mock_testing=True, mock_redacted_text=input_b_anonymizer_results
      )

      _api_key = "sk-12345"
      user_api_key_dict = UserAPIKeyAuth(api_key=_api_key)
      local_cache = DualCache()

      sample_content = "My name is Jane Doe, who are you?"
      test_data = {
          "messages": [{"role": "user", "content": sample_content}],
          "proxy_server_request": {
              "body": {
                  "messages": [{"role": "user", "content": sample_content}]
              }
          }
      }

      result_data = await pii_masking.async_pre_call_hook(
          user_api_key_dict=user_api_key_dict,
          cache=local_cache,
          data=test_data,
          call_type="completion",
      )

      # Verify both places were updated with redacted content
      assert "<PERSON>" in result_data["messages"][0]["content"]
      assert "<PERSON>" in result_data["proxy_server_request"]["body"]["messages"][0]["content"]
      assert "Jane Doe" not in result_data["messages"][0]["content"]
      assert "Jane Doe" not in result_data["proxy_server_request"]["body"]["messages"][0]["content"]

This ensures that both the logged messages and the effective request contain properly redacted PII.

[btiernay]

👋 Hey @krrishdholakia

I wanted to follow up about this issue since it appears to be a significant privacy/security concern that hasn't received attention yet. While the guardrail correctly logs redacted PII in internal logs, it fails to redact the actual requests sent to LLM providers like AWS Bedrock.

In environments where PII protection is required (healthcare, finance, etc.), this creates a false sense of security where operators think PII is being scrubbed when it's actually being sent to third parties in full.

I've identified the likely issue above. The fix should be straightforward - ensuring redactions are also applied to the proxy_server_request["body"]["messages"] object.

Would appreciate if your team could take a look at this with some urgency. Happy to provide more details or help test if needed. 🙇

Hey @btiernay did you specify the guardrail in the request?

I don't see it here

22:43:39 - LiteLLM Proxy:DEBUG: proxy_server.py:3436 - Request received by LiteLLM:
litellm-1 | {
litellm-1 | "stream": true,
litellm-1 | "model": "bedrock-claude-3.5-sonnet",
litellm-1 | "messages": [
litellm-1 | {
litellm-1 | "role": "user",
litellm-1 | "content": "."
litellm-1 | },
litellm-1 | {
litellm-1 | "role": "assistant",
litellm-1 | "content": "I don't have any additional context or question to respond to. Is there something specific you'd like to ask or discuss? I'm happy to help if you have a question or topic in mind."
litellm-1 | },
litellm-1 | {
litellm-1 | "role": "user",
litellm-1 | "content": "My name is Bobby. What is my name backwards?"
litellm-1 | }
litellm-1 | ]
litellm-1 | }

The guardrail needs to be specified in your request, to have it run on the request

You can also turn it on by default - https://docs.litellm.ai/docs/proxy/guardrails/quick_start#default-on-guardrails

[btiernay]

I'm configuring this in my config file:

guardrails:
- guardrail_name: "presidio-pre-guard"
  litellm_params:
    guardrail: presidio
    mode: "pre_call"
    output_parse_pii: True

Is this not sufficient?

[btiernay]

Oh, default_on: true is the missing config?

[btiernay]

🎉 It works!

Thanks for your feedback here, and apologies for causing alarm 🙇

odd - why is it not replacing <PERSON> with bobby

[VonNao]

odd - why is it not replacing <PERSON> with bobby

This seems the case on end aswell. It Only returns person and not the name

Hey @ishaan-jaff are you looking into this?

[crp4222]

The de-anonymization issue @VonNao is describing (responses coming back with <PERSON> instead of the original names) is a hard problem when done as a guardrail callback inside the proxy — the mapping between original and redacted values needs to survive the full request/response cycle, including streaming.

I took a different approach: a standalone proxy that sits upstream of litellm (or replaces it for the routing part). The PII mapping is maintained per-request, and the response de-anonymization happens at the SSE token level with a trie-based buffer, so it works with streaming too.

For the default_on confusion — having PII masking as an opt-in per-request header is risky for exactly the reason @btiernay described. A proxy-level approach makes it the default for all traffic, no config to miss.

Built it as an open-source project if anyone wants to look at the implementation: https://github.com/crp4222/PrivAiTe