some tests fail on cmd line, but it works in VS Code Test Explorer

[bmerkle]

running tests in windows powershell fails 6 tests unfortunately (example) below.
running tests with VS Code Test Explorer runs all tests successfully.

seems to be a environment problem, (googling or asking chatGPT leads to same direction) but i did not find a real solution or trace.
python interpreter and env variables are the same

not a big issue but i usually prefer cmd line over clicking ;) and github ci obviously runs fine on linux ubuntu

(.venv) PS C:\work\microsoft\typeagent-py\test> .\make.bat test
=================================================== short test summary info ==================================================== 
FAILED test_factory.py::test_create_conversation_and_add_messages - RuntimeError: Knowledge extraction failed: Expecting value: line 1 column 1 (char 0) -- MESSAGE='Hello, how are you?'
FAILED test_incremental_index.py::test_incremental_index_with_vtt_files - RuntimeError: Failed to parse VTT file testdata/Confuse-A-Cat.vtt: [Errno 2] No such file or directory: 'testdata/Confuse-A-...
FAILED test_mcp_server.py::test_mcp_server_query_conversation_slow - JSONDecodeError('Expecting value: line 1 column 1 (char 0)') [single exception in ExceptionGroup]
FAILED test_podcasts.py::test_ingest_podcast - FileNotFoundError: [Errno 2] No such file or directory: 'testdata/FakePodcast.txt'
FAILED test_query_method.py::test_query_method_basic - RuntimeError: Knowledge extraction failed: Expecting value: line 1 column 1 (char 0) -- MESSAGE='Welcome to the Python progr...
FAILED test_query_method.py::test_query_method_empty_conversation - AssertionError: assert ('no answer' in 'search failed: expecting value: line 1 column 1 (char 0)' or 'unable to find' in 'se...
========================================== 6 failed, 345 passed, 4 skipped in 48.54s =========================================== 

[gvanrossum]

Hm, all of these pass for me with the latest main. Maybe try make clean and make venv? (where "make" is .\make.bat)

[gvanrossum]

@bmerkle - if that doesn't fix it, can you post the full output from running at least one of the failing tests? Also, what Windows version?

[bmerkle]

below is

• short output of envrionment
• longer output of test trace
• screenshot of Test Exporer (only 1 test fails)
• git is on latest main.

I also see some permission error messages, so maybe the company virusscanner or some other system scanning is involved.
I will spin up a wsl ubuntu and give it a try on linux :-)

C:\work\microsoft\typeagent-py>ver

Microsoft Windows [Version 10.0.26100.6899]

C:\work\microsoft\typeagent-py>make clean

C:\work\microsoft\typeagent-py>make venv
Creating virtual environment...
Python 3.13.5
black, 25.9.0 (compiled: yes)
Python (CPython) 3.13.5
WARNING: there is a new pyright version available (v1.1.406 -> v1.1.407).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`

pyright 1.1.406
pytest 8.4.2

C:\work\microsoft\typeagent-py>.\.venv\Scripts\activate

(typeagent) C:\work\microsoft\typeagent-py>where python
C:\work\microsoft\typeagent-py\.venv\Scripts\python.exe
C:\Users\merklbe1\AppData\Local\Microsoft\WindowsApps\python.exe

(typeagent) C:\work\microsoft\typeagent-py>python --version
Python 3.13.5

(typeagent) C:\work\microsoft\typeagent-py>make test
Running tests...
========================================================== test session starts ===========================================================
platform win32 -- Python 3.13.5, pytest-8.4.2, pluggy-1.6.0
rootdir: C:\work\microsoft\typeagent-py
configfile: pyproject.toml
testpaths: test
plugins: anyio-4.11.0, logfire-4.14.2, asyncio-1.2.0, mock-3.15.1
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function
collected 357 items

test\test_add_messages_with_indexing.py ...                                                                                         [  0%]
test\test_auth.py .......                                                                                                           [  2%]
test\test_collections.py ...........................                                                                                [ 10%]
test\test_conversation_metadata.py ...................E.                                                                            [ 15%]
test\test_demo.py .                                                                                                                 [ 16%]
test\test_embedding_consistency.py ...                                                                                              [ 17%]
test\test_embeddings.py ...........                                                                                                 [ 20%]
test\test_factory.py ..F                                                                                                            [ 21%]
test\test_incremental_index.py ..                                                                                                   [ 21%]
test\test_interfaces.py ......................                                                                                      [ 27%]
test\test_knowledge.py ....                                                                                                         [ 28%]
test\test_kplib.py ......                                                                                                           [ 30%]
test\test_mcp_server.py F.                                                                                                          [ 31%]
test\test_message_text_index_population.py .                                                                                        [ 31%]
test\test_message_text_index_serialization.py ..                                                                                    [ 31%]
test\test_messageindex.py ....s...                                                                                                  [ 34%]
test\test_podcast_incremental.py ..                                                                                                 [ 34%]
test\test_podcasts.py F                                                                                                             [ 35%]
test\test_property_index_population.py .                                                                                            [ 35%]
test\test_propindex.py ............                                                                                                 [ 38%]
test\test_query.py ..................................                                                                               [ 48%]
test\test_query_method.py FF                                                                                                        [ 48%]
test\test_related_terms_fast.py .                                                                                                   [ 49%]
test\test_related_terms_index_population.py .                                                                                       [ 49%]
test\test_reltermsindex.py .............                                                                                            [ 52%]
test\test_searchlib.py .......................................................                                                      [ 68%]
test\test_secindex.py ...                                                                                                           [ 69%]
test\test_secindex_storage_integration.py .                                                                                         [ 69%]
test\test_semrefindex.py ......................                                                                                     [ 75%]
test\test_serialization.py ........                                                                                                 [ 77%]
test\test_sqlite_indexes.py .....................                                                                                   [ 83%]
test\test_sqlitestore.py .....                                                                                                      [ 85%]
test\test_storage_providers_unified.py ..............................                                                               [ 93%]
test\test_timestampindex.py .                                                                                                       [ 93%]
test\test_transcripts.py ......F                                                                                                    [ 95%]
test\test_utils.py ....                                                                                                             [ 96%]
test\test_vectorbase.py ...........                                                                                                 [100%]

================================================================= ERRORS =================================================================

______________________ ERROR at teardown of TestConversationMetadataEdgeCases.test_embedding_model_mismatch_raises _______________________

    @pytest.fixture
    def temp_db_path() -> Iterator[str]:
        """Create a temporary SQLite database file for testing."""
        temp_file = tempfile.NamedTemporaryFile(suffix=".sqlite", delete=False)
        path = temp_file.name
        temp_file.close()
        yield path
        if os.path.exists(path):
>           os.remove(path)
E           PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\merklbe1\\AppData\\Local\\Temp\\tmp_bezht23.sqlite'

test\fixtures.py:80: PermissionError
================================================================ FAILURES ================================================================
_______________________________________________ test_create_conversation_and_add_messages ________________________________________________

really_needs_auth = None

    @pytest.mark.asyncio
    async def test_create_conversation_and_add_messages(really_needs_auth):
        """Test the complete workflow: create conversation and add messages."""
        # 1. Create empty conversation
        test_model = AsyncEmbeddingModel(model_name=TEST_MODEL_NAME)
        settings = ConversationSettings(model=test_model)
        conversation = await create_conversation(
            None,
            TranscriptMessage,
            name="Test Conversation",
            settings=settings,
        )

        # 2. Prepare messages
        messages = [
            TranscriptMessage(
                text_chunks=["Hello, how are you?"],
                metadata=TranscriptMessageMeta(speaker="Alice"),
            ),
            TranscriptMessage(
                text_chunks=["I'm doing great, thanks!"],
                metadata=TranscriptMessageMeta(speaker="Bob"),
            ),
        ]

        # 3. Ingest with automatic indexing
>       result = await conversation.add_messages_with_indexing(messages)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

test\test_factory.py:81:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
typeagent\knowpro\conversation_base.py:152: in add_messages_with_indexing
    await self._add_llm_knowledge_incremental(
typeagent\knowpro\conversation_base.py:210: in _add_llm_knowledge_incremental
    await semrefindex.add_batch_to_semantic_ref_index_from_list(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

conversation = ConversationBase(settings=<typeagent.knowpro.convsettings.ConversationSettings object at 0x0000018B4B73F2B0>, storage_...o.secindex.ConversationSecondaryIndexes object at 0x0000018B204636B0>, _query_translator=None, _answer_translator=None)
messages = [ConversationMessage(text_chunks=['Hello, how are you?'], metadata=ConversationMessageMeta(speaker='Alice', recipients...["I'm doing great, thanks!"], metadata=ConversationMessageMeta(speaker='Bob', recipients=[]), tags=[], timestamp=None)]
batch = [TextLocation(0, 0), TextLocation(1, 0)]
knowledge_extractor = KnowledgeExtractor(model=<typechat._internal.model.HttpxLanguageModel object at 0x0000018B2052B0E0>, max_chars_per_chu...action_knowledge=False, translator=<typechat._internal.translator.TypeChatJsonTranslator object at 0x0000018B4B6F57F0>)
terms_added = None

    async def add_batch_to_semantic_ref_index_from_list[
        TMessage: IMessage, TTermToSemanticRefIndex: ITermToSemanticRefIndex
    ](
        conversation: IConversation[TMessage, TTermToSemanticRefIndex],
        messages: list[TMessage],
        batch: list[TextLocation],
        knowledge_extractor: IKnowledgeExtractor,
        terms_added: set[str] | None = None,
    ) -> None:
        """
        Add a batch of knowledge to semantic ref index, extracting from provided message list.

        Args:
            conversation: The conversation containing semantic refs and index
            messages: List of messages containing the text to extract from
            batch: List of text locations (ordinals) to process
            knowledge_extractor: Extractor for LLM-based knowledge extraction
            terms_added: Optional set to track newly added terms
        """
        # Get the starting ordinal of the message list
        if not batch:
            return
        start_ordinal = batch[0].message_ordinal

        # Extract text from the messages list
        text_batch = []
        for tl in batch:
            # Calculate index in the list from the ordinal
            list_index = tl.message_ordinal - start_ordinal
            if list_index < 0 or list_index >= len(messages):
                raise IndexError(
                    f"Message ordinal {tl.message_ordinal} out of range for list starting at {start_ordinal}"
                )
            message = messages[list_index]
            text = message.text_chunks[tl.chunk_ordinal].strip()
            text_batch.append(text)

        knowledge_results = await extract_knowledge_from_text_batch(
            knowledge_extractor,
            text_batch,
            len(text_batch),
        )
        for i, knowledge_result in enumerate(knowledge_results):
            if isinstance(knowledge_result, Failure):
>               raise RuntimeError(
                    f"Knowledge extraction failed: {knowledge_result.message}"
                )
E               RuntimeError: Knowledge extraction failed: Expecting value: line 1 column 1 (char 0) -- MESSAGE='Hello, how are you?'

typeagent\storage\memory\semrefindex.py:141: RuntimeError
________________________________________________ test_mcp_server_query_conversation_slow _________________________________________________
  + Exception Group Traceback (most recent call last):
  |   File "C:\work\microsoft\typeagent-py\test\test_mcp_server.py", line 95, in test_mcp_server_query_conversation_slow
  |     async with stdio_client(server_params) as (read, write):
  |                ~~~~~~~~~~~~^^^^^^^^^^^^^^^
  |   File "C:\Users\merklbe1\AppData\Roaming\uv\python\cpython-3.13.5-windows-x86_64-none\Lib\contextlib.py", line 235, in __aexit__
  |     await self.gen.athrow(value)
  |   File "C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\mcp\client\stdio\__init__.py", line 183, in stdio_client
  |     anyio.create_task_group() as tg,
  |     ~~~~~~~~~~~~~~~~~~~~~~~^^
  |   File "C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\anyio\_backends\_asyncio.py", line 781, in __aexit__
  |     raise BaseExceptionGroup(
  |         "unhandled errors in a TaskGroup", self._exceptions
  |     ) from None
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Exception Group Traceback (most recent call last):
    |   File "C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\mcp\client\stdio\__init__.py", line 189, in stdio_client
    |     yield read_stream, write_stream
    |   File "C:\work\microsoft\typeagent-py\test\test_mcp_server.py", line 96, in test_mcp_server_query_conversation_slow
    |     async with ClientSession(
    |                ~~~~~~~~~~~~~^
    |         read, write, sampling_callback=sampling_callback
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |     ) as session:
    |     ^
    |   File "C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\mcp\shared\session.py", line 218, in __aexit__
    |     return await self._task_group.__aexit__(exc_type, exc_val, exc_tb)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\anyio\_backends\_asyncio.py", line 781, in __aexit__
    |     raise BaseExceptionGroup(
    |         "unhandled errors in a TaskGroup", self._exceptions
    |     ) from None
    | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
    +-+---------------- 1 ----------------
      | Traceback (most recent call last):
      |   File "C:\work\microsoft\typeagent-py\test\test_mcp_server.py", line 135, in test_mcp_server_query_conversation_slow
      |     assert response_data["success"] is True, response_data
      | AssertionError: {'answer': "MCP sampling failed: McpError('Invalid request parameters')", 'success': False, 'time_used': 443}
      | assert False is True
      +------------------------------------
---------------------------------------------------------- Captured stderr call ----------------------------------------------------------
[11/14/25 07:51:12] INFO     Processing request of type           server.py:674
                             ListToolsRequest
                    INFO     Processing request of type           server.py:674
                             CallToolRequest
----------------------------------------------------------- Captured log call ------------------------------------------------------------
WARNING  root:session.py:363 Failed to validate request: AZURE_OPENAI_ENDPOINT=https://aihubmerklbe1e1830483446.openai.azure.com/ doesn't contain valid api-version field
__________________________________________________________ test_ingest_podcast ___________________________________________________________

really_needs_auth = None, temp_dir = 'C:\\Users\\merklbe1\\AppData\\Local\\Temp\\tmp8kbrph9k'
embedding_model = <typeagent.aitools.embeddings.AsyncEmbeddingModel object at 0x0000018B496EF050>

    @pytest.mark.asyncio
    async def test_ingest_podcast(
        really_needs_auth: None, temp_dir: str, embedding_model: AsyncEmbeddingModel
    ):
        # Import the podcast
        settings = ConversationSettings(embedding_model)
>       pod = await podcast_ingest.ingest_podcast(
            "testdata/FakePodcast.txt",
            settings,
            None,
            Datetime.now(timezone.utc),  # Use timezone-aware datetime
            3.0,
        )

test\test_podcasts.py:24:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
typeagent\podcasts\podcast_ingest.py:133: in ingest_podcast
    await pod.add_messages_with_indexing(batch)
typeagent\knowpro\conversation_base.py:152: in add_messages_with_indexing
    await self._add_llm_knowledge_incremental(
typeagent\knowpro\conversation_base.py:210: in _add_llm_knowledge_incremental
    await semrefindex.add_batch_to_semantic_ref_index_from_list(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

conversation = Podcast(settings=<typeagent.knowpro.convsettings.ConversationSettings object at 0x0000018B55E9A9D0>, storage_provider=...o.secindex.ConversationSecondaryIndexes object at 0x0000018B4BD41B30>, _query_translator=None, _answer_translator=None)
messages = [ConversationMessage(text_chunks=['Welcome to my podcast, Steve and Umesh.'], metadata=ConversationMessageMeta(speaker...speaker='umesh', recipients=['steve', 'guido', 'guido van rossum']), tags=[], timestamp='2025-11-14T06:53:40.119703Z')]
batch = [TextLocation(0, 0), TextLocation(1, 0), TextLocation(2, 0), TextLocation(3, 0), TextLocation(4, 0)]
knowledge_extractor = KnowledgeExtractor(model=<typechat._internal.model.HttpxLanguageModel object at 0x0000018B4BDB2990>, max_chars_per_chu...action_knowledge=False, translator=<typechat._internal.translator.TypeChatJsonTranslator object at 0x0000018B4BDB3110>)
terms_added = None

    async def add_batch_to_semantic_ref_index_from_list[
        TMessage: IMessage, TTermToSemanticRefIndex: ITermToSemanticRefIndex
    ](
        conversation: IConversation[TMessage, TTermToSemanticRefIndex],
        messages: list[TMessage],
        batch: list[TextLocation],
        knowledge_extractor: IKnowledgeExtractor,
        terms_added: set[str] | None = None,
    ) -> None:
        """
        Add a batch of knowledge to semantic ref index, extracting from provided message list.

        Args:
            conversation: The conversation containing semantic refs and index
            messages: List of messages containing the text to extract from
            batch: List of text locations (ordinals) to process
            knowledge_extractor: Extractor for LLM-based knowledge extraction
            terms_added: Optional set to track newly added terms
        """
        # Get the starting ordinal of the message list
        if not batch:
            return
        start_ordinal = batch[0].message_ordinal

        # Extract text from the messages list
        text_batch = []
        for tl in batch:
            # Calculate index in the list from the ordinal
            list_index = tl.message_ordinal - start_ordinal
            if list_index < 0 or list_index >= len(messages):
                raise IndexError(
                    f"Message ordinal {tl.message_ordinal} out of range for list starting at {start_ordinal}"
                )
            message = messages[list_index]
            text = message.text_chunks[tl.chunk_ordinal].strip()
            text_batch.append(text)

        knowledge_results = await extract_knowledge_from_text_batch(
            knowledge_extractor,
            text_batch,
            len(text_batch),
        )
        for i, knowledge_result in enumerate(knowledge_results):
            if isinstance(knowledge_result, Failure):
>               raise RuntimeError(
                    f"Knowledge extraction failed: {knowledge_result.message}"
                )
E               RuntimeError: Knowledge extraction failed: Expecting value: line 1 column 1 (char 0) -- MESSAGE='Welcome to my podcast, Steve and Umesh.'

typeagent\storage\memory\semrefindex.py:141: RuntimeError
________________________________________________________ test_query_method_basic _________________________________________________________

really_needs_auth = None

    @pytest.mark.asyncio
    async def test_query_method_basic(really_needs_auth: None):
        """Test the basic query method workflow."""
        # Create a conversation with some test data
        test_model = AsyncEmbeddingModel(model_name=TEST_MODEL_NAME)
        settings = ConversationSettings(model=test_model)
        conversation = await create_conversation(
            None,
            TranscriptMessage,
            name="Test Conversation",
            settings=settings,
        )

        # Add some test messages
        messages = [
            TranscriptMessage(
                text_chunks=["Welcome to the Python programming tutorial."],
                metadata=TranscriptMessageMeta(speaker="Instructor"),
            ),
            TranscriptMessage(
                text_chunks=["Today we'll learn about async/await in Python."],
                metadata=TranscriptMessageMeta(speaker="Instructor"),
            ),
            TranscriptMessage(
                text_chunks=["Python is a great language for beginners."],
                metadata=TranscriptMessageMeta(speaker="Instructor"),
            ),
        ]

>       await conversation.add_messages_with_indexing(messages)

test\test_query_method.py:46:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
typeagent\knowpro\conversation_base.py:152: in add_messages_with_indexing
    await self._add_llm_knowledge_incremental(
typeagent\knowpro\conversation_base.py:210: in _add_llm_knowledge_incremental
    await semrefindex.add_batch_to_semantic_ref_index_from_list(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

conversation = ConversationBase(settings=<typeagent.knowpro.convsettings.ConversationSettings object at 0x0000018B4F72BB50>, storage_...o.secindex.ConversationSecondaryIndexes object at 0x0000018B4F72BC50>, _query_translator=None, _answer_translator=None)
messages = [ConversationMessage(text_chunks=['Welcome to the Python programming tutorial.'], metadata=ConversationMessageMeta(spe...uage for beginners.'], metadata=ConversationMessageMeta(speaker='Instructor', recipients=[]), tags=[], timestamp=None)]
batch = [TextLocation(0, 0), TextLocation(1, 0), TextLocation(2, 0)]
knowledge_extractor = KnowledgeExtractor(model=<typechat._internal.model.HttpxLanguageModel object at 0x0000018B4F774050>, max_chars_per_chu...action_knowledge=False, translator=<typechat._internal.translator.TypeChatJsonTranslator object at 0x0000018B4F7743E0>)
terms_added = None

    async def add_batch_to_semantic_ref_index_from_list[
        TMessage: IMessage, TTermToSemanticRefIndex: ITermToSemanticRefIndex
    ](
        conversation: IConversation[TMessage, TTermToSemanticRefIndex],
        messages: list[TMessage],
        batch: list[TextLocation],
        knowledge_extractor: IKnowledgeExtractor,
        terms_added: set[str] | None = None,
    ) -> None:
        """
        Add a batch of knowledge to semantic ref index, extracting from provided message list.

        Args:
            conversation: The conversation containing semantic refs and index
            messages: List of messages containing the text to extract from
            batch: List of text locations (ordinals) to process
            knowledge_extractor: Extractor for LLM-based knowledge extraction
            terms_added: Optional set to track newly added terms
        """
        # Get the starting ordinal of the message list
        if not batch:
            return
        start_ordinal = batch[0].message_ordinal

        # Extract text from the messages list
        text_batch = []
        for tl in batch:
            # Calculate index in the list from the ordinal
            list_index = tl.message_ordinal - start_ordinal
            if list_index < 0 or list_index >= len(messages):
                raise IndexError(
                    f"Message ordinal {tl.message_ordinal} out of range for list starting at {start_ordinal}"
                )
            message = messages[list_index]
            text = message.text_chunks[tl.chunk_ordinal].strip()
            text_batch.append(text)

        knowledge_results = await extract_knowledge_from_text_batch(
            knowledge_extractor,
            text_batch,
            len(text_batch),
        )
        for i, knowledge_result in enumerate(knowledge_results):
            if isinstance(knowledge_result, Failure):
>               raise RuntimeError(
                    f"Knowledge extraction failed: {knowledge_result.message}"
                )
E               RuntimeError: Knowledge extraction failed: Expecting value: line 1 column 1 (char 0) -- MESSAGE='Welcome to the Python programming tutorial.'

typeagent\storage\memory\semrefindex.py:141: RuntimeError
----------------------------------------------------------- Captured log call ------------------------------------------------------------
ERROR    asyncio:base_events.py:1871 Task exception was never retrieved
future: <Task finished name='Task-330' coro=<AsyncClient.aclose() done, defined at C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\httpx\_client.py:1978> exception=RuntimeError('Event loop is closed')>
Traceback (most recent call last):
  File "C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\httpx\_client.py", line 1985, in aclose
    await self._transport.aclose()
  File "C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\httpx\_transports\default.py", line 406, in aclose
    await self._pool.aclose()
  File "C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\httpcore\_async\connection_pool.py", line 353, in aclose
    await self._close_connections(closing_connections)
  File "C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\httpcore\_async\connection_pool.py", line 345, in _close_connections
    await connection.aclose()
  File "C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\httpcore\_async\connection.py", line 173, in aclose
    await self._connection.aclose()
  File "C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\httpcore\_async\http11.py", line 258, in aclose
    await self._network_stream.aclose()
  File "C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\httpcore\_backends\anyio.py", line 53, in aclose
    await self._stream.aclose()
  File "C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\anyio\streams\tls.py", line 234, in aclose
    await self.transport_stream.aclose()
  File "C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\anyio\_backends\_asyncio.py", line 1323, in aclose
    self._transport.close()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "C:\Users\merklbe1\AppData\Roaming\uv\python\cpython-3.13.5-windows-x86_64-none\Lib\asyncio\proactor_events.py", line 109, in close
    self._loop.call_soon(self._call_connection_lost, None)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\merklbe1\AppData\Roaming\uv\python\cpython-3.13.5-windows-x86_64-none\Lib\asyncio\base_events.py", line 833, in call_soon
    self._check_closed()
    ~~~~~~~~~~~~~~~~~~^^
  File "C:\Users\merklbe1\AppData\Roaming\uv\python\cpython-3.13.5-windows-x86_64-none\Lib\asyncio\base_events.py", line 556, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
ERROR    asyncio:base_events.py:1871 Task exception was never retrieved
future: <Task finished name='Task-331' coro=<AsyncClient.aclose() done, defined at C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\httpx\_client.py:1978> exception=RuntimeError('Event loop is closed')>
Traceback (most recent call last):
  File "C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\httpx\_client.py", line 1985, in aclose
    await self._transport.aclose()
  File "C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\httpx\_transports\default.py", line 406, in aclose
    await self._pool.aclose()
  File "C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\httpcore\_async\connection_pool.py", line 353, in aclose
    await self._close_connections(closing_connections)
  File "C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\httpcore\_async\connection_pool.py", line 345, in _close_connections
    await connection.aclose()
  File "C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\httpcore\_async\connection.py", line 173, in aclose
    await self._connection.aclose()
  File "C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\httpcore\_async\http11.py", line 258, in aclose
    await self._network_stream.aclose()
  File "C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\httpcore\_backends\anyio.py", line 53, in aclose
    await self._stream.aclose()
  File "C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\anyio\streams\tls.py", line 234, in aclose
    await self.transport_stream.aclose()
  File "C:\work\microsoft\typeagent-py\.venv\Lib\site-packages\anyio\_backends\_asyncio.py", line 1323, in aclose
    self._transport.close()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "C:\Users\merklbe1\AppData\Roaming\uv\python\cpython-3.13.5-windows-x86_64-none\Lib\asyncio\proactor_events.py", line 109, in close
    self._loop.call_soon(self._call_connection_lost, None)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\merklbe1\AppData\Roaming\uv\python\cpython-3.13.5-windows-x86_64-none\Lib\asyncio\base_events.py", line 833, in call_soon
    self._check_closed()
    ~~~~~~~~~~~~~~~~~~^^
  File "C:\Users\merklbe1\AppData\Roaming\uv\python\cpython-3.13.5-windows-x86_64-none\Lib\asyncio\base_events.py", line 556, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
__________________________________________________ test_query_method_empty_conversation __________________________________________________

really_needs_auth = None

    @pytest.mark.asyncio
    async def test_query_method_empty_conversation(really_needs_auth: None):
        """Test query method on an empty conversation."""
        test_model = AsyncEmbeddingModel(model_name=TEST_MODEL_NAME)
        settings = ConversationSettings(model=test_model)
        conversation = await create_conversation(
            None,
            TranscriptMessage,
            name="Empty Conversation",
            settings=settings,
        )

        # Query should handle empty conversation gracefully
        answer = await conversation.query("What was discussed?")

        assert isinstance(answer, str)
        assert len(answer) > 0
        # Should indicate no answer found or no relevant information
>       assert "no answer" in answer.lower() or "unable to find" in answer.lower()
E       AssertionError: assert ('no answer' in 'search failed: expecting value: line 1 column 1 (char 0)' or 'unable to find' in 'search failed: expecting value: line 1 column 1 (char 0)')
E        +  where 'search failed: expecting value: line 1 column 1 (char 0)' = <built-in method lower of str object at 0x0000018B4B72ECD0>()
E        +    where <built-in method lower of str object at 0x0000018B4B72ECD0> = 'Search failed: Expecting value: line 1 column 1 (char 0)'.lower
E        +  and   'search failed: expecting value: line 1 column 1 (char 0)' = <built-in method lower of str object at 0x0000018B4B72ECD0>()
E        +    where <built-in method lower of str object at 0x0000018B4B72ECD0> = 'Search failed: Expecting value: line 1 column 1 (char 0)'.lower

test\test_query_method.py:81: AssertionError
_______________________________________________ test_transcript_knowledge_extraction_slow ________________________________________________

really_needs_auth = None, embedding_model = <typeagent.aitools.embeddings.AsyncEmbeddingModel object at 0x0000018B4B707C50>

    @pytest.mark.asyncio
    async def test_transcript_knowledge_extraction_slow(
        really_needs_auth: None, embedding_model: AsyncEmbeddingModel
    ):
        """
        Test that knowledge extraction works during transcript ingestion.

        This test verifies the complete ingestion pipeline:
        1. Parses first 5 messages from Parrot Sketch VTT file
        2. Creates transcript with in-memory storage (fast)
        3. Runs build_index() with auto_extract_knowledge=True
        4. Verifies both mechanical extraction (entities/actions from metadata)
           and LLM extraction (topics from content) work correctly
        """
        import webvtt
        from typeagent.storage.memory.collections import (
            MemoryMessageCollection,
            MemorySemanticRefCollection,
        )
        from typeagent.storage.memory.semrefindex import TermToSemanticRefIndex
        from typeagent.transcripts.transcript_ingest import extract_speaker_from_text

        # Use in-memory storage for speed
        settings = ConversationSettings(embedding_model)

        # Parse first 5 captions from Parrot Sketch
        vtt_file = "testdata/Parrot_Sketch.vtt"
        if not os.path.exists(vtt_file):
            pytest.skip(f"Test file {vtt_file} not found")

        vtt = webvtt.read(vtt_file)

        # Create messages from first 5 captions
        messages_list = []
        # vtt is indexable but not iterable
        for i in range(min(len(vtt), 5)):
            caption = vtt[i]
            if not caption.text.strip():
                continue

            speaker = getattr(caption, "voice", None)
            text = caption.text.strip()

            # Calculate timestamp from WebVTT start time
            offset_seconds = webvtt_timestamp_to_seconds(caption.start)
            timestamp = format_timestamp_utc(UNIX_EPOCH + timedelta(seconds=offset_seconds))

            metadata = TranscriptMessageMeta(
                speaker=speaker,
                recipients=[],
            )
            message = TranscriptMessage(
                text_chunks=[text], metadata=metadata, timestamp=timestamp
            )
            messages_list.append(message)

        # Create in-memory collections
        msg_coll = MemoryMessageCollection[TranscriptMessage]()
        await msg_coll.extend(messages_list)

        semref_coll = MemorySemanticRefCollection()
        semref_index = TermToSemanticRefIndex()

        # Create transcript with in-memory storage
        transcript = await Transcript.create(
            settings,
            name="Parrot-Test",
            tags=["test", "parrot"],
        )

        # Verify we have messages
        assert len(messages_list) >= 3, "Need at least 3 messages for testing"

        # Enable knowledge extraction
        settings.semantic_ref_index_settings.auto_extract_knowledge = True
        settings.semantic_ref_index_settings.batch_size = 10

        # Add messages with indexing (this should extract knowledge)
>       result = await transcript.add_messages_with_indexing(messages_list)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

test\test_transcripts.py:320:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
typeagent\knowpro\conversation_base.py:152: in add_messages_with_indexing
    await self._add_llm_knowledge_incremental(
typeagent\knowpro\conversation_base.py:210: in _add_llm_knowledge_incremental
    await semrefindex.add_batch_to_semantic_ref_index_from_list(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

conversation = Transcript(settings=<typeagent.knowpro.convsettings.ConversationSettings object at 0x0000018B4BB99DD0>, storage_provid...o.secindex.ConversationSecondaryIndexes object at 0x0000018B4B794AD0>, _query_translator=None, _answer_translator=None)
messages = [ConversationMessage(text_chunks=['[ANIMATION] including dancing Botticelli Venus, which links to pet shop: Mr. Pralin...."], metadata=ConversationMessageMeta(speaker='Shopkeeper', recipients=[]), tags=[], timestamp='1970-01-01T00:00:17Z')]
batch = [TextLocation(0, 0), TextLocation(1, 0), TextLocation(2, 0), TextLocation(3, 0), TextLocation(4, 0)]
knowledge_extractor = KnowledgeExtractor(model=<typechat._internal.model.HttpxLanguageModel object at 0x0000018B4BDF3CE0>, max_chars_per_chu...action_knowledge=False, translator=<typechat._internal.translator.TypeChatJsonTranslator object at 0x0000018B4BDF3AC0>)
terms_added = None

    async def add_batch_to_semantic_ref_index_from_list[
        TMessage: IMessage, TTermToSemanticRefIndex: ITermToSemanticRefIndex
    ](
        conversation: IConversation[TMessage, TTermToSemanticRefIndex],
        messages: list[TMessage],
        batch: list[TextLocation],
        knowledge_extractor: IKnowledgeExtractor,
        terms_added: set[str] | None = None,
    ) -> None:
        """
        Add a batch of knowledge to semantic ref index, extracting from provided message list.

        Args:
            conversation: The conversation containing semantic refs and index
            messages: List of messages containing the text to extract from
            batch: List of text locations (ordinals) to process
            knowledge_extractor: Extractor for LLM-based knowledge extraction
            terms_added: Optional set to track newly added terms
        """
        # Get the starting ordinal of the message list
        if not batch:
            return
        start_ordinal = batch[0].message_ordinal

        # Extract text from the messages list
        text_batch = []
        for tl in batch:
            # Calculate index in the list from the ordinal
            list_index = tl.message_ordinal - start_ordinal
            if list_index < 0 or list_index >= len(messages):
                raise IndexError(
                    f"Message ordinal {tl.message_ordinal} out of range for list starting at {start_ordinal}"
                )
            message = messages[list_index]
            text = message.text_chunks[tl.chunk_ordinal].strip()
            text_batch.append(text)

        knowledge_results = await extract_knowledge_from_text_batch(
            knowledge_extractor,
            text_batch,
            len(text_batch),
        )
        for i, knowledge_result in enumerate(knowledge_results):
            if isinstance(knowledge_result, Failure):
>               raise RuntimeError(
                    f"Knowledge extraction failed: {knowledge_result.message}"
                )
E               RuntimeError: Knowledge extraction failed: Expecting value: line 1 column 1 (char 0) -- MESSAGE='[ANIMATION] including dancing Botticelli Venus, which links to pet shop: Mr. Praline walks into the shop carrying a dead parrot in a cage. He walks to counter where shopkeeper tries to hide below cash register.'

typeagent\storage\memory\semrefindex.py:141: RuntimeError
======================================================== short test summary info =========================================================

[gvanrossum]

Ah, that's a different failure than your first message, and this one I see as well.

I will get to the bottom of this next week; I think it may be because a test opens the same db twice. We may just have to skip that test on Windows (it's not a very realistic scenario).

[gvanrossum]

Oh wait. You still have all the Failures you reported, and you have the cleanup Error in the temp_db_path fixture. I only see the latter.

I have a fix for the latter (just ignore the PermissionError). For the rest I will have to study your tracebacks carefully.

[bmerkle]

I can confirm that it is a windows only problem.
Test ran fine on same machine in my wsl/linux envrionment (Ubuntu 24.04.2 LTS)

merklbe1@H69LST3:~/typeagent-py$ make test
.venv/bin/pytest
================================================= test session starts ==================================================
platform linux -- Python 3.12.3, pytest-8.4.2, pluggy-1.6.0
rootdir: /home/merklbe1/typeagent-py
configfile: pyproject.toml
testpaths: test
plugins: logfire-4.14.2, anyio-4.11.0, asyncio-1.2.0, mock-3.15.1
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function
collected 357 items

test/test_add_messages_with_indexing.py ...                                                                      [  0%]
test/test_auth.py .......                                                                                        [  2%]
test/test_collections.py ...........................                                                             [ 10%]
test/test_conversation_metadata.py ....................                                                          [ 15%]
test/test_demo.py s                                                                                              [ 16%]
test/test_embedding_consistency.py ...                                                                           [ 17%]
test/test_embeddings.py ...........                                                                              [ 20%]
test/test_factory.py ..s                                                                                         [ 21%]
test/test_incremental_index.py ..                                                                                [ 21%]
test/test_interfaces.py ......................                                                                   [ 27%]
test/test_knowledge.py s...                                                                                      [ 28%]
test/test_kplib.py ......                                                                                        [ 30%]
test/test_mcp_server.py s.                                                                                       [ 31%]
test/test_message_text_index_population.py .                                                                     [ 31%]
test/test_message_text_index_serialization.py ..                                                                 [ 31%]
test/test_messageindex.py ....s...                                                                               [ 34%]
test/test_podcast_incremental.py ..                                                                              [ 34%]
test/test_podcasts.py s                                                                                          [ 35%]
test/test_property_index_population.py s                                                                         [ 35%]
test/test_propindex.py ............                                                                              [ 38%]
test/test_query.py ..................................                                                            [ 48%]
test/test_query_method.py ss                                                                                     [ 48%]
test/test_related_terms_fast.py .                                                                                [ 49%]
test/test_related_terms_index_population.py s                                                                    [ 49%]
test/test_reltermsindex.py ........s....                                                                         [ 52%]
test/test_searchlib.py .......................................................                                   [ 68%]
test/test_secindex.py ...                                                                                        [ 69%]
test/test_secindex_storage_integration.py .                                                                      [ 69%]
test/test_semrefindex.py ......................                                                                  [ 75%]
test/test_serialization.py ........                                                                              [ 77%]
test/test_sqlite_indexes.py .....................                                                                [ 83%]
test/test_sqlitestore.py .....                                                                                   [ 85%]
test/test_storage_providers_unified.py ..............................                                            [ 93%]
test/test_timestampindex.py .                                                                                    [ 93%]
test/test_transcripts.py ......s                                                                                 [ 95%]
test/test_utils.py ..s.                                                                                          [ 96%]
test/test_vectorbase.py ...........                                                                              [100%]

=========================================== 344 passed, 13 skipped in 24.79s ===========================================

[bmerkle]

Hi @gvanrossum

i set up a second windows test on a private machine, just to be sure that it is not a issue with my company environment
(firewall, content scanning etc etc.) and we do not search into the wrong direction. ;-)

the news is:

• on my private machine the same 6 tests fail in windows
• however this time they fail with http 429 which is API rate limit too many request

I am using plain OPENAI access (so OPENAI_API_KEY and OPENAI_MODEL=gpt-4o, so i am a bit puzzeled why the test fail with 429.
In my openAI i do not see that I hit any token/rate/account limit

So my question to you is: (to have the same setup)

• are you using OPENAI for testing ?
• if yes, do you have a pro account ? (i have free)
• or are you using AZURE_OPEN for testing ?

[gvanrossum]

Belatedly answering your questions:

• I am using an internal Azure/OpenAI instance that has tons of capacity.
• I pay for my OpenAI requests (I guess that's a Pro account).
• Let me see if I can repro this with OpenAI or if I can create a second, free OpenAI account to try to repro.

[gvanrossum-ms]

Another observation on your report. In one of your latest messages you show output of a successful test run on Linux. However I see that several tests are skipped there that shouldn't be skipped. They are skipped when no API key is found (neither OPENAI nor AZURE_OPENAI). Possibly these tests would start to fail if you set then to the same OPENAI API key that failed on Windows? Because it's definitely failing due to rate limiting by the openai server. EDIT See fixtures.really_needs_auth.