Skip to content

fix: Test suite stability - fix failures, reduce skips, clean warnings #345

Description

@vybe

Summary

The test suite has 3 failures, 97 skipped tests, and 289 deprecation warnings. While the 99.9% pass rate indicates platform health, we need to improve test reliability and coverage by fixing configuration issues.

Test Failures (3)

1. Async Task Execution Flakiness

Files: tests/test_parallel_task.py:595, tests/test_parallel_task.py:731
Tests:

  • TestAsyncModeExecution::test_async_mode_execution_completes
  • TestAsyncModeActivities::test_async_mode_creates_task_activity

Problem: These tests depend on a live Claude Code agent successfully completing a task. When the agent fails (missing API key, out of context, transient Docker issue), both tests fail.

Fix:

  • Add pytest.mark.dependency to cascade skip when execution fails
  • Consider mocking execution results for unit tests
  • Add retry logic or longer polling timeout
  • Verify test agent has valid API key in fixture setup

2. Subscription State Leak

File: tests/test_subscriptions.py:528
Test: TestSubscriptionAutoAssign::test_create_agent_no_subscriptions

Problem: Test expects agent to have no subscription, but subscription from prior test is still assigned.

Fix:

  • Add explicit clear_agent_subscription call in fixture or test setup
  • Ensure module-scoped fixtures don't leak subscription state

Skipped Tests to Enable (97 total)

Priority 1: Missing Test Environment Config

Category Count Fix Required
Agent Server Direct 14 Set TEST_AGENT_NAME env var in test config
Internal API 7 Set INTERNAL_API_SECRET in test environment
Git tests 10 Configure a test agent with git template
SSH access 7 Configure SSH service for test env

Priority 2: Missing Python Packages

Package Tests Enabled Command
docker 6 (test_log_archive.py) pip install docker
markdown-it-py 13 (unit/test_slack_rich_text.py) pip install markdown-it-py

Priority 3: Test Design Improvements

Category Count Action
Public links 4 Require running agent with public URL - consider mocking
Public user memory 9 Same as above
Setup tests 3 Can only run on fresh install - document as expected
Slack integration 3 Mock Slack or configure test workspace
Validation DB 3 Verify VALIDATE-001 tables exist in test DB

Deprecation Warnings (289)

datetime.utcnow() Deprecation (Python 3.12+)

Replace with datetime.now(datetime.UTC) in:

  • src/scheduler/service.py
  • src/scheduler/database.py
  • src/backend/logging_config.py
  • src/backend/services/backlog_service.py
  • src/backend/services/ssh_service.py
  • tests/scheduler_tests/*.py

Pydantic V2 Deprecations

  • Update AgentStatus, Activity, Execution models in src/backend/models.py to use Pydantic V2 config syntax

Acceptance Criteria

  • All 3 test failures resolved
  • Skipped test count reduced from 97 to <30 (only tests requiring external services)
  • Zero deprecation warnings in test output
  • Test suite runs reliably on CI without flaky failures
  • Document any intentionally skipped tests with clear skip reasons

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions