[Detail Bug] get_response_text crashes when last list item is not a dict

[detail-app]

Summary

• Context: get_response_text() is a utility function in sv_shared/utils.py that normalizes completion outputs from the Verifiers library into plain strings for reward functions and parsers across all security verification environments.
• Bug: The function crashes with AttributeError when passed a list whose last element is not a dictionary.
• Actual vs. expected: The function calls .get("content", "") on the last list item without checking if it's a dict, but the function signature accepts Any type and should handle edge cases gracefully.
• Impact: Any environment or custom code passing a malformed completion list with non-dict items will crash with an AttributeError, causing reward calculation and parsing to fail entirely.

Code with bug

def get_response_text(completion: Any) -> str:
    """Extract text content from a completion structure.

    The Verifiers library may return either a raw string or a list of
    message dictionaries. This helper normalizes those inputs to a plain
    string for reward functions and parsers.
    """

    if isinstance(completion, list):
        return completion[-1].get("content", "") if completion else ""  # <-- BUG 🔴 Assumes last item has .get() method
    return str(completion)

Evidence

Example

Consider a completion list where the last item is a string instead of a dict:

messages = [
    {"role": "assistant", "content": "First message"},
    "just a string",  # Last item is a string, not a dict
]

Step-by-step execution:

1. isinstance(completion, list) → True
2. completion is non-empty, so evaluate completion[-1].get("content", "")
3. completion[-1] → "just a string" (a string, not a dict)
4. Call .get("content", "") on the string
5. Crash: AttributeError: 'str' object has no attribute 'get'

The same crash occurs for any non-dict type as the last list item: None, int, float, bool, etc.

Failing test

Test script

#!/usr/bin/env python3
"""Failing test demonstrating the bug in get_response_text."""

import pytest
from sv_shared.utils import get_response_text


def test_list_with_non_dict_as_last_item_should_not_crash():
    """
    Test that get_response_text handles lists where the last item is not a dict.

    This is an edge case that could occur if:
    1. The completion list is malformed or corrupted
    2. Future changes to the Verifiers library introduce new message types
    3. Custom environments pass non-standard completion structures

    The function should gracefully handle this case rather than crashing with AttributeError.
    """
    # Case 1: Last item is a string
    messages = [
        {"role": "assistant", "content": "First message"},
        "just a string",
    ]
    # Should not crash with AttributeError: 'str' object has no attribute 'get'
    result = get_response_text(messages)
    assert isinstance(result, str)  # Should return some string, not crash

    # Case 2: Last item is None
    messages = [
        {"role": "assistant", "content": "First message"},
        None,
    ]
    # Should not crash with AttributeError: 'NoneType' object has no attribute 'get'
    result = get_response_text(messages)
    assert isinstance(result, str)  # Should return some string, not crash

    # Case 3: Last item is an integer
    messages = [
        {"role": "assistant", "content": "First message"},
        42,
    ]
    # Should not crash with AttributeError: 'int' object has no attribute 'get'
    result = get_response_text(messages)
    assert isinstance(result, str)  # Should return some string, not crash


if __name__ == "__main__":
    pytest.main([__file__, "-v", "--tb=short"])

Test output

============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.0.2, pluggy-1.6.0
rootdir: /home/user/security-verifiers
configfile: pyproject.toml
plugins: anyio-4.12.1, cov-7.0.0
collected 1 item

test_bug_failing.py F                                                    [100%]

=================================== FAILURES ===================================
____________ test_list_with_non_dict_as_last_item_should_not_crash _____________
test_bug_failing.py:25: in test_list_with_non_dict_as_last_item_should_not_crash
    result = get_response_text(messages)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
sv_shared/utils.py:17: in get_response_text
    return completion[-1].get("content", "") if completion else ""
           ^^^^^^^^^^^^^^^^^^
E   AttributeError: 'str' object has no attribute 'get'
=========================== short test summary info ============================
FAILED test_bug_failing.py::test_list_with_non_dict_as_last_item_should_not_crash
============================== 1 failed in 3.42s ===============================

Inconsistency within the codebase

Reference code

sv_shared/utils_test.py (lines 86-93)

def test_list_with_mixed_types(self) -> None:
    """Test list with non-dict items (edge case)."""
    # Should still try to get last item's content
    messages = [
        "string item",
        {"role": "assistant", "content": "Dict content"},
    ]
    assert get_response_text(messages) == "Dict content"

Current code

sv_shared/utils.py (lines 16-17)

if isinstance(completion, list):
    return completion[-1].get("content", "") if completion else ""

Contradiction

The test comment says "Should still try to get last item's content" when there are "non-dict items", implying the function should handle mixed-type lists gracefully. However, the test only passes because the last item is a dict. The implementation will crash if the last item is not a dict, which contradicts the intended behavior suggested by the test comment. The test's specific setup (non-dict first, dict last) masks the bug that occurs when the last item is non-dict.

Full context

get_response_text() is a core utility function used throughout the Security Verifiers codebase to normalize completion outputs from the Verifiers library. The function is called by:

1. Parsers (sv_shared/parsers.py):

   • JsonClassificationParser._parse_json() uses it to extract text before JSON parsing
   • This parser is used by the NetworkLogParser in E1 (network-logs environment)

2. Environment-specific parsers:

   • ConfigVerificationParser in sv-env-config-verification (E2)
   • CodeVulnerabilityParser in sv-env-code-vulnerability (E3)

3. Reward functions:

   • Called by parsers before extracting structured data (labels, confidence scores)
   • Used in format validation reward functions
   • Affects accuracy, calibration, and cost-sensitive reward calculations

The Verifiers library typically returns either:

• A raw string for simple completions
• A list of ChatMessage dicts (OpenAI message format) for multi-turn interactions

In normal operation with the Verifiers library, completion lists always contain dict-like message objects, so the bug doesn't manifest. However, the function signature accepts Any type, making it part of the public API contract that could receive malformed inputs from:

• Custom environments
• Future Verifiers library changes
• Corrupted completion data
• Test code or mock objects

When the crash occurs, it prevents:

• Reward calculation from completing
• Parser output from being generated
• Environment evaluation from proceeding
• Any meaningful error message about the actual problem with the completion

The function is critical infrastructure shared across all six security verification environments (E1-E6).

External documentation

• Verifiers Parser source code

The Verifiers library's Parser.parse_answer() method shows the correct pattern:

def parse_answer(self, completion: Messages) -> str | None:
    if isinstance(completion, str):
        return self.parse(completion)
    else:
        assistant_messages = self.get_assistant_messages(completion)
        if not assistant_messages:
            return None
        ans = str(assistant_messages[-1].get("content", ""))
        return self.parse(ans)

Key differences:

1. Filters for assistant messages first (line 45)
2. Checks if the filtered list is empty (line 46-47)
3. Only then accesses the last message's content (line 48)

Why has this bug gone undetected?

This bug has gone undetected for several reasons:

1. Verifiers library contract: In normal operation, the Verifiers library always returns either a string or a list of properly-formatted ChatMessage dicts (OpenAI format). The library maintains this invariant, so production code paths never trigger the bug.

2. Test coverage gap: The existing test test_list_with_mixed_types appears to test mixed-type lists, but actually has the dict as the LAST item, so it passes. The comment "Should still try to get last item's content" suggests the test author intended to test more edge cases, but the specific test case doesn't expose the bug.

3. Type system limitations: The function signature uses Any, which disables static type checking. A more precise type hint like Union[str, List[dict]] or Messages would have made the assumption more explicit.

4. Graceful degradation elsewhere: The calling code (parsers and reward functions) already has error handling for malformed JSON and missing fields, so developers may not have anticipated needing additional validation at this utility level.

5. Production usage patterns: All current environments (E1-E6) use either:

   • String completions for single-turn responses
   • Well-formed message lists from the Verifiers library for multi-turn interactions

   None of the production code paths pass malformed lists with non-dict last items.

6. Recent addition: The function was added in commit 295547c (September 2025) to handle None confidence values, and may not have undergone extensive edge case testing yet.

Recommended fix

Add a type check before calling .get() on the last list item:

def get_response_text(completion: Any) -> str:
    """Extract text content from a completion structure.

    The Verifiers library may return either a raw string or a list of
    message dictionaries. This helper normalizes those inputs to a plain
    string for reward functions and parsers.
    """

    if isinstance(completion, list):
        if not completion:
            return ""
        last_item = completion[-1]
        if isinstance(last_item, dict):  # <-- FIX 🟢 Check if dict before calling .get()
            return last_item.get("content", "")
        return str(last_item)  # <-- FIX 🟢 Fallback to str() for non-dict items
    return str(completion)

This fix:

1. Preserves the existing behavior for valid inputs (strings and dict lists)
2. Gracefully handles edge cases by converting non-dict items to strings
3. Matches the pattern used by the Verifiers library's Parser.parse_answer() method
4. Makes the function truly defensive as suggested by its Any type signature

[intertwine]

Fixed in PR #52. get_response_text now handles non-dict items at the end of a list by returning an empty string or converting to string.