test pollution case: module-level sys.modules pollution.

[jiaminc-cmu]

When I ran pdd sync on a module, it created a test that passed. But it failed test collection when running with other modules. This is because it created test pollutions module-level sys.modules pollution

[jiaminc-cmu]

Step 1: Duplicate Check

Status: No duplicates found

Search Performed

• Searched for: "test pollution sys.modules", "test collection failed", "pdd sync test", "module pollution", "test isolation", "collection error", "sys.modules", "import"
• Issues reviewed: 15+ issues examined in detail

Findings

This appears to be a new, distinct issue. While several related issues were found, none are true duplicates:

Issue	Relationship	Why Not Duplicate
#342	Related	Focuses on sys.path ordering causing imports from site-packages vs local code. Different root cause (path precedence) vs this issue (sys.modules pollution between test runs).
#361	Related	Deals with incorrect import statements in generated test files (wrong module name). This is a test generation bug, not runtime test pollution.
#88	Similar symptoms	Closed. Addressed wrong function names and import mismatches in generated tests. Different from module-level pollution during test collection.

Key distinction: Issue #349 specifically describes:

1. Tests passing when run in isolation
2. Tests failing during collection when run with other modules
3. Root cause: module-level sys.modules pollution (likely tests modifying sys.modules without cleanup)

This is a test isolation/cleanup problem at runtime, distinct from the test generation issues in #342, #361, and #88.

---

Proceeding to Step 2: Documentation Check

[jiaminc-cmu]

Step 2: Documentation Check

Status: Confirmed Bug

Documentation Reviewed

• README.md - PDD overview and workflow
• CONTRIBUTING.md - Development guidelines and test requirements
• docs/faq.md - FAQ on determinism and test accumulation
• context/pytest_isolation_example.py - Comprehensive test isolation patterns (430 lines)
• pdd/prompts/generate_test_LLM.prompt - Test generation prompt with isolation requirements
• CHANGELOG.md - Recent fixes for test isolation issues (v0.0.122-v0.0.124)

Findings

The reported behavior contradicts documented best practices and is a confirmed bug:

1. Documentation explicitly prohibits this pattern:

context/pytest_isolation_example.py (lines 329-369) explicitly documents the anti-pattern:

# PATTERN 9: ANTI-PATTERN - patcher.start() Without stop()
# This is the #1 cause of test pollution! NEVER do this:
# patcher = patch.dict(sys.modules, module_mocks)
# patcher.start()  # <-- DANGER: Never stopped!

2. Test generation prompt requires isolation:

pdd/prompts/generate_test_LLM.prompt (lines 36-41) explicitly requires:

% CRITICAL: Each test MUST be isolated and not pollute state for other tests.
- Tests must clean up any state they modify (environment variables, global state, files, mocks)

3. CHANGELOG shows active bug fixes for same issue:

Recent versions (v0.0.122-v0.0.124) have been addressing this exact class of bugs:

• "Refactored tests/commands/test_fix.py to restore sys.modules immediately after import"
• "Mock _get_cached_jwt in device flow tests to prevent test pollution"
• "Add test isolation patterns to context/test.prompt: module-level patcher anti-pattern"

Root Cause Analysis

The test generation LLM is producing code that uses module-level sys.modules patching without proper cleanup (PATTERN 9 anti-pattern), despite the prompt explicitly requiring test isolation. This causes:

1. Tests to pass in isolation
2. Tests to fail during pytest collection when run with other modules (sys.modules pollution persists)

The correct pattern (PATTERN 7 in pytest_isolation_example.py) requires saving originals BEFORE patching and restoring IMMEDIATELY after import.

---

Proceeding to Step 3: Triage

[jiaminc-cmu]

Step 3: Triage

Status: Proceed

Summary

Based on the information from the issue and previous investigation steps, we have sufficient context to attempt reproduction:

What we understand:

• Expected behavior: Tests generated by pdd sync should pass in isolation AND when run with other tests/modules
• Actual behavior: Generated tests pass in isolation but fail during test collection when run alongside other modules
• Root cause: Module-level sys.modules pollution in generated test code

Why we can proceed:

1. The bug pattern is well-documented and understood (sys.modules pollution is a known test isolation anti-pattern)
2. Step 2 confirmed this contradicts documented test isolation requirements
3. The CHANGELOG shows similar issues were fixed in v0.0.122-v0.0.124, providing reference points
4. We can examine the test generation code to identify where sys.modules manipulation occurs without proper cleanup

Investigation approach:

1. Search the test generation code for sys.modules manipulation patterns
2. Compare against the documented isolation patterns in context/pytest_isolation_example.py
3. Attempt to reproduce by running pdd sync on a sample module and examining the generated test

---

Proceeding to Step 4: Reproduction

[jiaminc-cmu]

Step 4: Reproduction

Status: Reproduced

Environment

• macOS 23.5.0 (Darwin)
• Python 3.12.7
• pytest 8.3.5 / 9.0.0
• pdd codebase at commit d6ddb97

Steps Taken

1. Identified the anti-pattern in codebase: Module-level sys.modules[...] = MagicMock() without restoration
2. Created minimal reproduction test case mimicking the pattern
3. Ran pytest with multiple test files to confirm collection-time pollution

Observed Behavior

============================= test session starts ==============================
collected 2 items

test_polluting.py::test_polluting_test PASSED                            [ 50%]
test_needs_real_pdd.py::test_check_pdd_not_mock FAILED                   [100%]

=================================== FAILURES ===================================
___________________________ test_check_pdd_not_mock ____________________________

E               AssertionError: POLLUTION DETECTED: sys.modules['pdd'] is a MagicMock!
E               This was set by another test file's module-level code.
E               pdd module type: <class 'unittest.mock.MagicMock'>

Key finding: The pollution occurs during pytest collection, not test execution. Even running pytest test_needs_real_pdd.py test_polluting.py -v (victim first) still fails because pytest imports all test modules during collection before running any tests.

Affected Files

4 files in context/ with module-level sys.modules pollution:

• context/commands/sessions_example.py:17-26 - Sets sys.modules["pdd"], ["pdd.core"], ["pdd.core.cloud"], etc.
• context/server/routes/auth_example.py:17-36 - Sets sys.modules["pdd"], ["pdd.get_jwt_token"]
• context/server/routes/commands_example.py:115-116 - Sets sys.modules["pdd.server.models"], ["pdd.server.jobs"]
• context/server/routes/prompts_example.py:15-50 - Sets sys.modules["pdd.security"], ["pdd.token_counter"], ["pdd.preprocess"]

Root Cause

These context/ example files use module-level sys.modules assignments without the save/restore pattern documented in context/pytest_isolation_example.py (Pattern 7 and Anti-Pattern 9).

The correct pattern per the documentation:

# Save BEFORE patching
_original_modules = {key: sys.modules.get(key) for key in _module_mocks}
sys.modules.update(_module_mocks)

# Import code under test here

# RESTORE IMMEDIATELY after import
for key, original in _original_modules.items():
    if original is None:
        sys.modules.pop(key, None)
    else:
        sys.modules[key] = original

Impact

When pdd sync generates tests based on these example files, the generated tests may include the same anti-pattern, causing:

1. Tests pass when run in isolation (pytest tests/test_foo.py)
2. Tests fail during collection when run with other modules (pytest tests/)

Notes

The issue is particularly insidious because:

• Pollution happens at import time (during collection), not test execution
• Test order doesn't matter - collection imports all modules first
• Running the "victim" test alone always passes

---

Proceeding to Step 5: Root Cause Analysis

[jiaminc-cmu]

Step 5: Root Cause Analysis

Root Cause Identified: Yes

Summary

The bug is caused by module-level sys.modules pollution in 4 context example files. These files unconditionally assign MagicMock() objects to sys.modules entries at the top level of the module (outside any function or fixture), with no cleanup or restoration. When pytest collects tests and imports these files, the mocks persist in sys.modules for the entire pytest session, masking the real pdd package modules and causing subsequent test modules to fail during collection.

Technical Details

Affected Files and Problematic Code:

1. context/commands/sessions_example.py:17-21,26,67,75

# --- MOCKING DEPENDENCIES START ---
sys.modules["pdd"] = MagicMock()
sys.modules["pdd.core"] = MagicMock()
sys.modules["pdd.core.cloud"] = MagicMock()
sys.modules["pdd.remote_session"] = MagicMock()
sys.modules["pdd.utils"] = MagicMock()
# ... more assignments, NEVER cleaned up

2. context/server/routes/auth_example.py:17-18,34-36

sys.modules["pdd"] = MagicMock()
sys.modules["pdd.get_jwt_token"] = MagicMock()
# ... more assignments, NEVER cleaned up

3. context/server/routes/commands_example.py:115-116

sys.modules["pdd.server.models"] = models_mod
sys.modules["pdd.server.jobs"] = jobs_mod
# NEVER cleaned up

4. context/server/routes/prompts_example.py:15,19,43,46,50

sys.modules["pdd.security"] = MagicMock()
sys.modules["pdd.token_counter"] = MagicMock()
sys.modules["pdd.preprocess"] = MagicMock()
# ... NEVER cleaned up

Why it fails:

1. Pollution occurs during pytest collection, not execution. When pytest discovers and imports test modules, it executes all module-level code.
2. These files execute sys.modules["pdd.*"] = MagicMock() at import time with no save/restore pattern.
3. The mocks persist for the entire pytest session, masking the real pdd package.
4. Any subsequent test file that imports from the real pdd package gets MagicMock objects instead, causing AttributeError, TypeError, or other failures during collection.

The Irony: The codebase already has the correct pattern documented in context/pytest_isolation_example.py (Pattern 7, lines 193-241 and Pattern 9, lines 329-368), but these 4 example files don't follow it.

External Research

• Search terms: pytest sys.modules pollution, test collection failure module-level mocking
• Sources checked: pytest documentation, Medium articles on mocking imports, pytest GitHub issues
• Relevant findings:
  • pytest's monkeypatch fixture automatically restores changes after each test, but it only works inside test functions/fixtures, not at module level
  • The patch.dict(sys.modules, {...}) context manager should be used for safe module patching
  • Module-level code executes during pytest collection, before any fixtures run

Repository History

• Relevant commits:

  • 8de59d57 (2026-01-14): Introduced context/commands/sessions_example.py, context/server/routes/auth_example.py, and context/server/routes/commands_example.py with the polluting pattern
  • b8a4d5d (earlier): Introduced context/server/routes/prompts_example.py with the same pattern

• Related issues:

  • Issue pdd test should generate tests that isolate local code under test #342: Different root cause (sys.path ordering, not sys.modules pollution), but related to test isolation
  • The pytest_isolation_example.py file was added to document correct patterns, but existing example files were not updated

• CHANGELOG entries (v0.0.122-v0.0.124): Document "Test Isolation Framework" additions, including pytest_isolation_example.py, indicating awareness of the problem space

Experiments Performed

1. Simulated module import:

   exec(open('context/commands/sessions_example.py').read())
   print('pdd' in sys.modules)  # True
   print(type(sys.modules['pdd']))  # <class 'unittest.mock.MagicMock'>

   Confirmed that executing the file pollutes sys.modules with MagicMock objects.

2. Attempted real import after pollution:

   import pdd
   print(pdd.__file__)  # AttributeError or "NO FILE" - MagicMock has no __file__

   Confirmed that the real pdd module is masked by the mock.

3. Collection order dependency:
   When these context files are imported before other test files, all subsequent imports of pdd.* return MagicMock objects, explaining the "passes in isolation, fails with other modules" symptom.

Fix Direction

The 4 affected files should be updated to follow the pattern documented in context/pytest_isolation_example.py (Pattern 7):

1. Save original modules before patching:

   _saved_modules = {key: sys.modules.get(key) for key in ["pdd", "pdd.core", ...]}

2. Apply mocks:

   sys.modules["pdd"] = MagicMock()
   # ...

3. Restore immediately after the module under test is loaded:

   for key, original in _saved_modules.items():
       if original is None:
           sys.modules.pop(key, None)
       else:
           sys.modules[key] = original

Alternatively, if these files are not meant to be imported as part of test collection, they should:

• Be moved to a non-discoverable location (e.g., examples/ instead of context/)
• Be excluded from pytest collection via conftest.py or pytest.ini configuration
• Be renamed to not match test file patterns

---

Proceeding to Step 6: Test Plan

[jiaminc-cmu]

Step 6: Test Plan

Existing Test Coverage

• Test file: tests/test_generate_test.py (class TestContextFileExists)
• Current coverage: Verifies that context/pytest_isolation_example.py exists, contains patterns, and covers key pollution sources (env vars, sys.modules, files, fixtures)
• Gap: No tests verify that the example files themselves (e.g., context/commands/sessions_example.py, context/server/routes/*_example.py) follow the documented isolation patterns

Related Existing Patterns

The codebase already has proper isolation infrastructure:

1. tests/conftest.py has autouse fixtures for preserve_pdd_path and restore_standard_streams
2. tests/test_llm_invoke.py demonstrates proper save/restore patterns (lines 629, 698)
3. context/pytest_isolation_example.py documents the correct Pattern 7 (save/mock/import/restore)

Proposed Tests

Test 1: Detect sys.modules pollution in context example files

• Input: Scan all context/**/*_example.py files for module-level sys.modules[...] = assignments
• Expected: Any file that assigns to sys.modules at module level should also have corresponding restoration code (save original → apply mock → restore original pattern)
• Actual (before fix): 4 files have sys.modules[...] = MagicMock() without any restoration:
  • context/commands/sessions_example.py:17-21
  • context/server/routes/auth_example.py:17-18
  • context/server/routes/commands_example.py:115-116
  • context/server/routes/prompts_example.py:15,19,46

Test 2: Verify pytest collection doesn't see polluted sys.modules

• Input: Run pytest --collect-only on the full test suite
• Expected: Collection succeeds without any modules in sys.modules being replaced with MagicMock objects
• Actual (before fix): Importing the context example files during collection pollutes sys.modules, causing subsequent test imports to get MagicMock instead of real modules

Test 3: Verify context example files are excluded from pytest collection

• Input: Check that context/ directory files are not collected as tests
• Expected: Files in context/ should not be collected (they're context/documentation files, not test files)
• Note: This is an alternative fix approach - exclude context files from collection entirely via conftest.py or pytest configuration

Test Location

• File: tests/test_context_example_isolation.py (new)
• Framework: pytest

Test Implementation Approach

"""Tests to prevent sys.modules pollution in context example files."""

import ast
import re
from pathlib import Path
import pytest


def get_project_root() -> Path:
    """Get the project root directory."""
    current = Path(__file__).parent
    while current != current.parent:
        if (current / "context").is_dir():
            return current
        current = current.parent
    raise RuntimeError("Could not find project root")


class TestContextExampleIsolation:
    """Tests ensuring context example files don't pollute sys.modules."""

    def test_sys_modules_assignments_have_restoration(self):
        """Verify all sys.modules assignments have corresponding restoration.
        
        Files that assign to sys.modules at module level MUST also restore
        the original values, otherwise they pollute sys.modules for all
        subsequent test files during pytest collection.
        """
        project_root = get_project_root()
        context_dir = project_root / "context"
        
        violations = []
        
        for example_file in context_dir.rglob("*_example.py"):
            content = example_file.read_text()
            
            # Check for sys.modules assignments
            if re.search(r'sys\.modules\[.+\]\s*=', content):
                # Check for restoration pattern (save original, restore later)
                has_save = re.search(r'_original|_saved|sys\.modules\.get\(', content)
                has_restore = re.search(r'sys\.modules\[.+\]\s*=\s*(_original|_saved|original)', content)
                has_pop = re.search(r'sys\.modules\.pop\(', content)
                
                if not (has_save and (has_restore or has_pop)):
                    rel_path = example_file.relative_to(project_root)
                    violations.append(str(rel_path))
        
        assert not violations, (
            f"The following files assign to sys.modules without restoration:\\n"
            f"{chr(10).join(f'  - {v}' for v in violations)}\\n\\n"
            f"See context/pytest_isolation_example.py Pattern 7 for the correct approach."
        )

Notes

• Alternative fix: Instead of fixing each example file, the context files could be excluded from pytest collection entirely by adding context/ to collect_ignore_glob in tests/conftest.py
• Root cause: Context example files are meant as documentation/examples for the LLM, not as executable tests. They should either follow proper isolation patterns OR be excluded from pytest collection.
• The test should be added regardless of fix approach to prevent regression

---

Proceeding to Step 7: Generate Test

[jiaminc-cmu]

Step 7: Generated Test

Test File

tests/test_context_example_isolation.py

Test Code

"""Tests for verifying context example files follow proper test isolation patterns.

Issue #349: Tests generated by pdd sync may pass in isolation but fail during
collection when run with other modules due to sys.modules pollution at module level.

This test file verifies that context example files do NOT introduce module-level
sys.modules pollution that would affect other test files during pytest collection.
"""

import ast
import os
import re
from pathlib import Path
from typing import List, Tuple

import pytest


def get_project_root() -> Path:
    """Get the project root directory."""
    current = Path(__file__).parent
    while current != current.parent:
        if (current / "context").is_dir():
            return current
        current = current.parent
    raise RuntimeError("Could not find project root with context/ directory")


def get_all_context_example_files() -> List[Path]:
    """Get all context example files that may pollute sys.modules."""
    project_root = get_project_root()
    context_dir = project_root / "context"
    example_files = list(context_dir.glob("**/*_example.py"))
    return example_files


def parse_file_for_sys_modules_assignments(file_path: Path) -> List[Tuple[int, str]]:
    """Parse a Python file and find sys.modules assignments without restoration."""
    try:
        source = file_path.read_text()
    except Exception:
        return []

    problematic_assignments = []
    assignment_pattern = re.compile(
        r'^\s*sys\.modules\s*\[\s*["\'"]([^"\'"]+)["\'"]\s*\]\s*=\s*(.+)$',
        re.MULTILINE
    )
    restore_pattern = re.compile(
        r'(del\s+sys\.modules|sys\.modules\.pop|sys\.modules\s*\[\s*["\'"][^"\'"]+["\'"]\s*\]\s*=\s*(_saved|_original|original|saved))',
        re.IGNORECASE
    )

    try:
        tree = ast.parse(source)
    except SyntaxError:
        return []

    function_ranges = []
    for node in ast.walk(tree):
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
            if hasattr(node, "lineno") and hasattr(node, "end_lineno"):
                function_ranges.append((node.lineno, node.end_lineno or node.lineno))

    lines = source.split("\n")
    for i, line in enumerate(lines, 1):
        match = assignment_pattern.match(line)
        if match:
            module_name = match.group(1)
            is_module_level = True
            for start, end in function_ranges:
                if start <= i <= end:
                    is_module_level = False
                    break

            if is_module_level:
                has_restoration = False
                if restore_pattern.search(source):
                    save_pattern = re.compile(
                        rf'(_saved|_original|original|saved)\s*[=\[].*{re.escape(module_name)}',
                        re.IGNORECASE
                    )
                    if save_pattern.search(source):
                        has_restoration = True

                if not has_restoration:
                    problematic_assignments.append((i, line.strip()))

    return problematic_assignments


class TestContextExampleIsolation:
    """Tests verifying context example files do not pollute sys.modules."""

    def test_no_module_level_sys_modules_pollution(self):
        """Verify no context example files have unrestored sys.modules assignments."""
        example_files = get_all_context_example_files()
        assert len(example_files) > 0, "Should find context example files"

        violations = []
        for file_path in example_files:
            problematic = parse_file_for_sys_modules_assignments(file_path)
            if problematic:
                relative_path = file_path.relative_to(get_project_root())
                for line_num, code in problematic:
                    violations.append(f"  {relative_path}:{line_num}: {code}")

        if violations:
            violation_list = "\n".join(violations)
            pytest.fail(
                f"Found {len(violations)} module-level sys.modules assignment(s) "
                f"without restoration (causes test pollution during collection):\n\n"
                f"{violation_list}\n\n"
                f"Fix: Follow Pattern 7 in context/pytest_isolation_example.py"
            )

    def test_known_problematic_files_are_detected(self):
        """Verify known problematic files from Issue #349 are detected."""
        project_root = get_project_root()
        known_problematic = [
            "context/commands/sessions_example.py",
            "context/server/routes/auth_example.py",
            "context/server/routes/commands_example.py",
            "context/server/routes/prompts_example.py",
        ]
        detected_files = []
        for relative_path in known_problematic:
            file_path = project_root / relative_path
            if file_path.exists():
                if parse_file_for_sys_modules_assignments(file_path):
                    detected_files.append(relative_path)

        if detected_files:
            pytest.fail(
                f"Known problematic files still have unrestored sys.modules assignments:\n"
                f"  {", ".join(detected_files)}"
            )


class TestSysModulesNotPollutedDuringCollection:
    """Tests that verify pytest collection does not see polluted sys.modules."""

    def test_pdd_modules_are_not_mocked_after_collection(self):
        """Verify pdd.* modules in sys.modules are not MagicMock objects."""
        import sys
        from unittest.mock import MagicMock

        polluted_modules = []
        for name, module in sys.modules.items():
            if name.startswith("pdd"):
                if isinstance(module, MagicMock):
                    polluted_modules.append(name)

        if polluted_modules:
            pytest.fail(
                f"sys.modules contains MagicMock objects for pdd modules: "
                f"{", ".join(sorted(polluted_modules))}"
            )

What This Test Verifies

This test uses static analysis to detect sys.modules assignments at module level without corresponding restoration code. It catches the root cause of Issue #349:

1. test_no_module_level_sys_modules_pollution: Scans all context/**/*_example.py files for module-level sys.modules["xxx"] = MagicMock() patterns that lack save/restore logic. Currently detects 12 violations across 4 files.

2. test_known_problematic_files_are_detected: Specifically checks the 4 files identified in root cause analysis:

   • context/commands/sessions_example.py
   • context/server/routes/auth_example.py
   • context/server/routes/commands_example.py
   • context/server/routes/prompts_example.py

3. test_pdd_modules_are_not_mocked_after_collection: Runtime check that verifies pdd.* modules are not MagicMock objects after collection.

Running the Test

pytest tests/test_context_example_isolation.py -v

Current result (2 failed, 2 passed):

• Detects all 12 problematic sys.modules assignments
• All 4 known files are flagged
• Runtime pollution test passes (context files not imported during collection in this isolated run)

---

Proceeding to Step 8: Verification

[jiaminc-cmu]

Step 8: Verification Complete

Test Execution

============================= test session starts ==============================
platform darwin -- Python 3.12.7, pytest-9.0.0, pluggy-1.6.0
asyncio: mode=Mode.STRICT

tests/test_context_example_isolation.py::TestContextExampleIsolation::test_no_module_level_sys_modules_pollution FAILED [ 25%]
tests/test_context_example_isolation.py::TestContextExampleIsolation::test_example_files_found PASSED [ 50%]
tests/test_context_example_isolation.py::TestContextExampleIsolation::test_known_problematic_files_are_detected FAILED [ 75%]
tests/test_context_example_isolation.py::TestSysModulesNotPollutedDuringCollection::test_pdd_modules_are_not_mocked_after_collection PASSED [100%]

=================================== FAILURES ===================================
____ TestContextExampleIsolation.test_no_module_level_sys_modules_pollution ____

E           Failed: Found 12 module-level sys.modules assignment(s) without restoration:
E           
E             context/commands/sessions_example.py:17: sys.modules["pdd"] = MagicMock()
E             context/commands/sessions_example.py:18: sys.modules["pdd.core"] = MagicMock()
E             context/commands/sessions_example.py:19: sys.modules["pdd.core.cloud"] = MagicMock()
E             context/commands/sessions_example.py:20: sys.modules["pdd.remote_session"] = MagicMock()
E             context/commands/sessions_example.py:21: sys.modules["pdd.utils"] = MagicMock()
E             context/server/routes/prompts_example.py:15: sys.modules["pdd.security"] = MagicMock()
E             context/server/routes/prompts_example.py:19: sys.modules["pdd.token_counter"] = MagicMock()
E             context/server/routes/prompts_example.py:46: sys.modules["pdd.preprocess"] = MagicMock()
E             context/server/routes/auth_example.py:17: sys.modules["pdd"] = MagicMock()
E             context/server/routes/auth_example.py:18: sys.modules["pdd.get_jwt_token"] = MagicMock()
E             context/server/routes/commands_example.py:115: sys.modules["pdd.server.models"] = models_mod
E             context/server/routes/commands_example.py:116: sys.modules["pdd.server.jobs"] = jobs_mod

==================== 2 failed, 2 passed, 1 warning in 0.40s ====================

Verification Status

PASS: Test correctly detects the bug

The test successfully identifies all 12 module-level sys.modules pollution instances across the 4 known problematic files. The tests fail for the correct reason (detecting unrestored sys.modules assignments) and provide clear, actionable error messages with exact file paths and line numbers.

Summary

Step	Result
Duplicate Check	✅ No duplicates - new issue
Documentation	✅ Confirmed bug - violates documented patterns
Triage	✅ Proceed - clear expected/actual behavior
Reproduction	✅ Reproduced - pollution during pytest collection
Root Cause	✅ Identified - module-level sys.modules assignments without restoration
Test Plan	✅ Designed - static analysis + runtime verification
Test Generation	✅ Created - tests/test_context_example_isolation.py
Verification	✅ Pass - test correctly detects all 12 violations

Bug Details

• Location: 4 files with 12 total violations:
  • context/commands/sessions_example.py:17-21 (5 violations)
  • context/server/routes/prompts_example.py:15,19,46 (3 violations)
  • context/server/routes/auth_example.py:17-18 (2 violations)
  • context/server/routes/commands_example.py:115-116 (2 violations)
• Root Cause: Module-level sys.modules[...] = MagicMock() assignments without corresponding restoration code cause pollution during pytest collection
• Test File: tests/test_context_example_isolation.py

Next Steps

1. Fix the bug by implementing the save/mock/restore pattern (Pattern 7 from context/pytest_isolation_example.py) in each affected file
2. Run the test to confirm the fix: pytest tests/test_context_example_isolation.py -v
3. Run full test suite to check for regressions: pytest
4. Submit PR with fix and test

---

Proceeding to Step 9: E2E Tests

[jiaminc-cmu]

Step 9: E2E Test

Environment Discovery

• Repository language: Python
• Test framework: pytest 8.3.5
• Existing E2E tests found: Yes - tests/test_e2e_issue_219_duplicate_includes.py, tests/test_pddrc_true_e2e.py, tests/test_operation_logging_e2e.py
• Test configuration: pytest.ini with asyncio_default_fixture_loop_scope = function, testpaths = tests
• Fixtures/helpers used: tmp_path (pytest built-in), subprocess for isolation
• Environment variables required: None for this test
• Auth/credentials: Not required for this test

E2E Test File

tests/test_e2e_issue_349_sys_modules_pollution.py

Test Type

Integration (subprocess-based E2E test)

What This Test Verifies

This E2E test verifies the user-facing behavior of Issue #349: when context example files with module-level sys.modules pollution are imported (as happens during pytest collection), subsequent imports of real pdd modules should not return MagicMock objects instead of real modules.

The test uses subprocess isolation to simulate a fresh Python process (as would happen during pytest tests/) and verifies:

1. Individual context example files don't leave MagicMock in sys.modules after import
2. Pytest collection with these files doesn't pollute pdd module entries
3. Real pdd imports work correctly after context example imports

Test Code

"""
E2E Test for Issue #349: sys.modules pollution during pytest collection.

This test verifies the bug at a system level by simulating what happens when
pytest collects test files that import context example files with module-level
sys.modules assignments.

The Bug:
- Context example files (e.g., context/commands/sessions_example.py) assign
  MagicMock() to sys.modules entries at module level
- These assignments happen during pytest collection when files are imported
- The mocks persist for the entire pytest session
- Subsequent test files importing real pdd modules get MagicMock instead

E2E Test Strategy:
- Use subprocess to run pytest in a fresh process
- This isolates the test from the current pytest session
- Run pytest --collect-only to trigger collection without executing tests
- Check if sys.modules is polluted after collection

The test should:
- FAIL on the current buggy code (pollution detected)
- PASS once the bug is fixed (no pollution)
"""

import json
import os
import subprocess
import sys
from pathlib import Path
import pytest


def get_project_root() -> Path:
    """Get the project root directory."""
    current = Path(__file__).parent
    while current != current.parent:
        if (current / "context").is_dir():
            return current
        current = current.parent
    raise RuntimeError("Could not find project root with context/ directory")


class TestSysModulesPollutionE2E:
    """E2E tests verifying sys.modules pollution during pytest collection."""

    def test_context_example_import_does_not_pollute_sys_modules(self, tmp_path: Path):
        """
        E2E Test: Importing a context example file should not leave MagicMock
        objects in sys.modules after the file is loaded.
        """
        # Uses subprocess to import each problematic file in isolation
        # and checks if sys.modules contains MagicMock objects for pdd.* modules
        ...

    def test_pytest_collection_does_not_pollute_pdd_modules(self, tmp_path: Path):
        """
        E2E Test: Running pytest --collect-only should not leave MagicMock
        objects in pdd.* module entries in sys.modules.
        """
        # Creates a temporary test suite that imports the context files
        # and checks for pollution after collection
        ...

    def test_real_pdd_import_after_context_example_import(self, tmp_path: Path):
        """
        E2E Test: After importing a problematic context example file,
        importing the real pdd module should still work correctly.
        """
        # Tests the user-facing symptom: real pdd imports return MagicMock
        ...

Test Execution Result

============================= test session starts ==============================
platform darwin -- Python 3.12.7, pytest-9.0.0, pluggy-1.6.0
collected 3 items

tests/test_e2e_issue_349_sys_modules_pollution.py::TestSysModulesPollutionE2E::test_context_example_import_does_not_pollute_sys_modules FAILED [ 33%]
tests/test_e2e_issue_349_sys_modules_pollution.py::TestSysModulesPollutionE2E::test_pytest_collection_does_not_pollute_pdd_modules FAILED [ 66%]
tests/test_e2e_issue_349_sys_modules_pollution.py::TestSysModulesPollutionE2E::test_real_pdd_import_after_context_example_import FAILED [100%]

=================================== FAILURES ===================================
_ test_context_example_import_does_not_pollute_sys_modules _
Failed: BUG DETECTED (Issue #349): Context example files pollute sys.modules!

The following files leave MagicMock objects in sys.modules after import:
  context/commands/sessions_example.py: pdd -> MagicMock
  context/commands/sessions_example.py: pdd.core -> MagicMock
  context/commands/sessions_example.py: pdd.core.cloud -> MagicMock
  context/commands/sessions_example.py: pdd.remote_session -> MagicMock
  context/commands/sessions_example.py: pdd.utils -> MagicMock
  context/server/routes/auth_example.py: pdd -> MagicMock
  context/server/routes/auth_example.py: pdd.get_jwt_token -> MagicMock

_ test_pytest_collection_does_not_pollute_pdd_modules _
Failed: BUG DETECTED (Issue #349): sys.modules polluted during pytest collection!

Polluted modules: pdd=MagicMock,pdd.core=MagicMock,pdd.core.cloud=MagicMock,
pdd.remote_session=MagicMock,pdd.utils=MagicMock,pdd.get_jwt_token=MagicMock,
pdd.security=MagicMock,pdd.token_counter=MagicMock,pdd.preprocess=MagicMock

_ test_real_pdd_import_after_context_example_import _
Failed: BUG DETECTED (Issue #349): Real pdd modules are MagicMock after import!

After importing context/commands/sessions_example.py:
  Polluted modules: pdd, pdd.core, pdd.utils

============================== 3 failed in 3.47s ===============================

E2E Test Status

PASS: E2E test correctly detects the bug

All 3 E2E tests fail as expected, detecting the sys.modules pollution at different levels:

1. File-level detection: Individual context example files leave MagicMock in sys.modules
2. Collection-level detection: Pytest collection with these files pollutes 9 pdd modules
3. User-impact detection: Real pdd imports return MagicMock instead of actual modules

The tests will pass once the bug is fixed (either by implementing save/mock/restore pattern in the context files, or by excluding them from pytest collection).

---

Proceeding to Step 10: Create Draft PR

[jiaminc-cmu]

Step 10: Draft PR Created

Pull Request

PR #367: Add failing tests for #349: sys.modules pollution

Branch

fix/issue-349

What's Included

• Failing unit test at tests/test_context_example_isolation.py
• Failing E2E test at tests/test_e2e_issue_349_sys_modules_pollution.py
• Commits: 1

Next Steps for Maintainers

1. Review the failing tests to understand the expected behavior
2. Implement the fix at the identified locations (see root cause below)
3. Verify both unit and E2E tests pass with your fix
4. Run full test suite to check for regressions
5. Mark the PR as ready for review

Root Cause Summary

4 context example files need to be fixed:

• context/commands/sessions_example.py:17-21
• context/server/routes/auth_example.py:17-18
• context/server/routes/commands_example.py:115-116
• context/server/routes/prompts_example.py:15,19,46

Each should use the save/mock/restore pattern documented in context/pytest_isolation_example.py (Pattern 7):

# Save
_saved = {k: sys.modules.get(k) for k in ["pdd", "pdd.core"]}

# Mock
sys.modules["pdd"] = MagicMock()
sys.modules["pdd.core"] = MagicMock()

# Import code under test
from your_module import function_to_test

# Restore immediately
for k, v in _saved.items():
    if v is None:
        sys.modules.pop(k, None)
    else:
        sys.modules[k] = v

PDD Fix Command

To auto-fix this bug using PDD (note: requires prompt files and verification programs for each affected file):

cd /Users/caijiamin/Desktop/pdd/pdd/.pdd/worktrees/fix-issue-349
# For each affected context example file, run:
# pdd --force fix --loop --max-attempts 5 --verification-program context/pytest_isolation_example.py <prompt_file> <code_file> <test_file> fix-issue-349.log

Note: The fix targets are context example files (not code files with .prompt counterparts), so manual fixes following Pattern 7 from context/pytest_isolation_example.py may be more appropriate.

---

Investigation complete. A draft PR with failing tests has been created and linked to this issue.

[jiaminc-cmu]

Step 1: Unit Test Execution (Cycle 1)

Status: All Tests Pass

Unit Tests Identified

• tests/test_context_example_isolation.py::TestContextExampleIsolation::test_no_module_level_sys_modules_pollution
• tests/test_context_example_isolation.py::TestContextExampleIsolation::test_example_files_found
• tests/test_context_example_isolation.py::TestContextExampleIsolation::test_known_problematic_files_are_detected
• tests/test_context_example_isolation.py::TestSysModulesNotPollutedDuringCollection::test_pdd_modules_are_not_mocked_after_collection
• tests/test_e2e_issue_349_sys_modules_pollution.py::TestSysModulesPollutionE2E::test_context_example_import_does_not_pollute_sys_modules
• tests/test_e2e_issue_349_sys_modules_pollution.py::TestSysModulesPollutionE2E::test_pytest_collection_does_not_pollute_pdd_modules
• tests/test_e2e_issue_349_sys_modules_pollution.py::TestSysModulesPollutionE2E::test_real_pdd_import_after_context_example_import

Initial Test Results

• Passed: 7
• Failed: 0

pdd fix Runs

Dev Unit	Result
N/A	Bug already fixed - context example files have save/restore pattern

Fix Details

The 4 affected context example files have already been manually fixed with the save/mock/restore pattern (Pattern 7 from context/pytest_isolation_example.py):

1. context/commands/sessions_example.py: Lines 19-24 (save), 26-30 (mock), 88-92 (restore)
2. context/server/routes/auth_example.py: Lines 19-21 (save), 23-24 (mock), 46-50 (restore)
3. context/server/routes/commands_example.py: Lines 106-108 (save), 121-122 (mock), 126-130 (restore)
4. context/server/routes/prompts_example.py: Lines 16-19 (save), 22/26/53 (mock), 61-65 (restore)

Current State

• Changes are staged but not committed (git status shows 4 modified files)
• All unit and E2E tests pass
• No pdd fix needed as the fix was already applied manually

Final Test Results

• Passed: 7
• Failed: 0

---

Proceeding to Step 2: E2E Test Check

[jiaminc-cmu]

Step 2: E2E Test Check (Cycle 1)

Status: ALL_TESTS_PASS

E2E Tests Run

• tests/test_context_example_isolation.py::TestContextExampleIsolation::test_no_module_level_sys_modules_pollution ✓
• tests/test_context_example_isolation.py::TestContextExampleIsolation::test_example_files_found ✓
• tests/test_context_example_isolation.py::TestContextExampleIsolation::test_known_problematic_files_are_detected ✓
• tests/test_context_example_isolation.py::TestSysModulesNotPollutedDuringCollection::test_pdd_modules_are_not_mocked_after_collection ✓
• tests/test_e2e_issue_349_sys_modules_pollution.py::TestSysModulesPollutionE2E::test_context_example_import_does_not_pollute_sys_modules ✓
• tests/test_e2e_issue_349_sys_modules_pollution.py::TestSysModulesPollutionE2E::test_pytest_collection_does_not_pollute_pdd_modules ✓
• tests/test_e2e_issue_349_sys_modules_pollution.py::TestSysModulesPollutionE2E::test_real_pdd_import_after_context_example_import ✓

Summary

All 7 e2e tests pass. Bug fix complete!

The fix has been successfully applied to the 4 context example files:

• context/commands/sessions_example.py
• context/server/routes/auth_example.py
• context/server/routes/commands_example.py
• context/server/routes/prompts_example.py

Each file now properly saves original sys.modules entries before patching and restores them immediately after, following Pattern 7 from context/pytest_isolation_example.py.

---

Workflow complete - all tests passing