Skip to content

Flaky grounding test #54

@abrookins

Description

@abrookins

This test is flaking (TestThreadAwareContextualGrounding.test_multi_entity_conversation):

=================================== FAILURES ===================================
______ TestThreadAwareContextualGrounding.test_multi_entity_conversation _______

self = <tests.test_thread_aware_grounding.TestThreadAwareContextualGrounding object at 0x7f806c145970>

    @pytest.mark.requires_api_keys
    async def test_multi_entity_conversation(self):
        """Test contextual grounding with multiple entities in conversation."""
    
        session_id = f"test-multi-entity-{ulid.ULID()}"
    
        # Create conversation with multiple people
        messages = [
            MemoryMessage(
                id=str(ulid.ULID()),
                role="user",
                content="John and Sarah are working on the API redesign project.",
                timestamp=datetime.now(UTC).isoformat(),
                discrete_memory_extracted="f",
            ),
            MemoryMessage(
                id=str(ulid.ULID()),
                role="user",
                content="He's handling the backend while she focuses on the frontend integration.",
                timestamp=datetime.now(UTC).isoformat(),
                discrete_memory_extracted="f",
            ),
            MemoryMessage(
                id=str(ulid.ULID()),
                role="user",
                content="Their collaboration has been very effective. His Python skills complement her React expertise.",
                timestamp=datetime.now(UTC).isoformat(),
                discrete_memory_extracted="f",
            ),
        ]
    
        working_memory = WorkingMemory(
            session_id=session_id,
            user_id="test-user",
            namespace="test-namespace",
            messages=messages,
            memories=[],
        )
    
        await set_working_memory(working_memory)
    
        # Extract memories
        extracted_memories = await extract_memories_from_session_thread(
            session_id=session_id,
            namespace="test-namespace",
            user_id="test-user",
        )
    
        assert len(extracted_memories) > 0
    
        all_memory_text = " ".join([mem.text for mem in extracted_memories])
    
        print(f"\nMulti-entity extracted memories: {len(extracted_memories)}")
        for i, mem in enumerate(extracted_memories):
            print(f"{i + 1}. [{mem.memory_type}] {mem.text}")
    
        # Should mention both John and Sarah by name
        assert "john" in all_memory_text.lower(), "Should mention John by name"
>       assert "sarah" in all_memory_text.lower(), "Should mention Sarah by name"
E       AssertionError: Should mention Sarah by name
E       assert 'sarah' in 'john is handling the backend of the api redesign project.'
E        +  where 'john is handling the backend of the api redesign project.' = <built-in method lower of str object at 0x7f806114c5e0>()
E        +    where <built-in method lower of str object at 0x7f806114c5e0> = 'John is handling the backend of the API redesign project.'.lower

tests/test_thread_aware_grounding.py:207: AssertionError
----------------------------- Captured stdout call -----------------------------

Multi-entity extracted memories: 1
1. [MemoryTypeEnum.EPISODIC] John is handling the backend of the API redesign project.
------------------------------ Captured log call -------------------------------
INFO     agent_memory_server.working_memory:working_memory.py:206 Set working memory for session test-multi-entity-01K3P5M1H0YCCGZ31FW5FNR5NK with no TTL
INFO     agent_memory_server.long_term_memory:long_term_memory.py:192 Extracting memories from 3 messages in session test-multi-entity-01K3P5M1H0YCCGZ31FW5FNR5NK
INFO     openai._base_client:_base_client.py:1608 Retrying request to /chat/completions in 0.495191 seconds
INFO     agent_memory_server.long_term_memory:long_term_memory.py:247 Extracted 1 memories from session thread test-multi-entity-01K3P5M1H0YCCGZ31FW5FNR5NK
=============================== warnings summary ===============================
tests/test_extraction.py::TestTopicExtractionIntegration::test_bertopic_integration
  /home/runner/work/agent-memory-server/agent-memory-server/.venv/lib/python3.12/site-packages/hdbscan/plots.py:448: SyntaxWarning: invalid escape sequence '\l'
    axis.set_ylabel('$\lambda$ value')

tests/test_extraction.py::TestTopicExtractionIntegration::test_bertopic_integration
  /home/runner/work/agent-memory-server/agent-memory-server/.venv/lib/python3.12/site-packages/hdbscan/robust_single_linkage_.py:175: SyntaxWarning: invalid escape sequence '\{'
    $max \{ core_k(a), core_k(b), 1/\alpha d(a,b) \}$.

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/test_thread_aware_grounding.py::TestThreadAwareContextualGrounding::test_multi_entity_conversation - AssertionError: Should mention Sarah by name
assert 'sarah' in 'john is handling the backend of the api redesign project.'
 +  where 'john is handling the backend of the api redesign project.' = <built-in method lower of str object at 0x7f806114c5e0>()
 +    where <built-in method lower of str object at 0x7f806114c5e0> = 'John is handling the backend of the API redesign project.'.lower
====== 1 failed, 375 passed, 26 skipped, 2 warnings in 151.50s (0:02:31) =======
Error: Process completed with exit code 1.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions