diff --git a/.github/workflows/stage-device-tests.yml b/.github/workflows/stage-device-tests.yml index ae8277a..e918a26 100644 --- a/.github/workflows/stage-device-tests.yml +++ b/.github/workflows/stage-device-tests.yml @@ -36,7 +36,7 @@ jobs: shell: bash run: | set -e - python tests/test_ssh_device_ci.py + python tests/devices/test_ssh_device_ci.py - name: Run benchmark dry-run via SSH shell: bash diff --git a/README.md b/README.md index 52ad1e5..995abbd 100644 --- a/README.md +++ b/README.md @@ -44,6 +44,7 @@ See [Configuration Reference](docs/configuration.md) for details. - **[Android Installer Module](docs/android_installer.md)** - Automated Android SDK/NDK setup - **[Build Guide](docs/build-guide.md)** - Building OpenVINO for mobile - **[Benchmarking Guide](docs/benchmarking.md)** - Running and interpreting benchmarks +- **[Testing Guide](docs/testing.md)** - Running and writing tests - **[CI/CD Integration](docs/ci-cd.md)** - GitHub Actions and automation - **[API Reference](docs/api-reference.md)** - Python API documentation - **[Troubleshooting](docs/troubleshooting.md)** - Common issues and solutions diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 0000000..645da5d --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,546 @@ +# Testing Guide + +## Overview + +OVMobileBench uses pytest for testing with comprehensive coverage across all modules. The test suite is organized to mirror the source code structure for easy navigation and maintenance. + +## Test Organization + +The test suite is structured by functional modules: + +``` +tests/ +├── android/ # Android-specific functionality +│ └── installer/ # Android SDK/NDK installer module +├── builders/ # OpenVINO build system +├── cli/ # Command-line interface +├── config/ # Configuration and validation +├── core/ # Core utilities +├── devices/ # Device abstraction layer +├── packaging/ # Bundle creation and packaging +├── parsers/ # Output parsing +├── pipeline/ # Pipeline orchestration +├── report/ # Report generation +└── runners/ # Benchmark execution +``` + +## Running Tests + +### Basic Commands + +```bash +# Run all tests +pytest tests/ + +# Run with verbose output +pytest tests/ -v + +# Run specific module +pytest tests/config/ +pytest tests/devices/ +pytest tests/pipeline/ + +# Run specific test file +pytest tests/config/test_openvino_config.py + +# Run specific test class +pytest tests/config/test_openvino_config.py::TestOpenVINOConfig + +# Run specific test method +pytest tests/config/test_openvino_config.py::TestOpenVINOConfig::test_build_mode_valid +``` + +### Coverage Analysis + +```bash +# Run with coverage report +pytest tests/ --cov=ovmobilebench --cov-report=term-missing + +# Generate HTML coverage report +pytest tests/ --cov=ovmobilebench --cov-report=html +open htmlcov/index.html + +# Coverage for specific modules +pytest tests/ --cov=ovmobilebench.config --cov=ovmobilebench.pipeline + +# Branch coverage +pytest tests/ --cov=ovmobilebench --cov-branch +``` + +### Test Selection + +```bash +# Run only fast tests (exclude slow) +pytest tests/ -m "not slow" + +# Run only unit tests +pytest tests/ -m unit + +# Run integration tests +pytest tests/ -m integration + +# Skip specific tests +pytest tests/ --ignore=tests/android/installer/ + +# Run failed tests from last run +pytest tests/ --lf + +# Run tests that match a pattern +pytest tests/ -k "openvino" +``` + +### Output Options + +```bash +# Quiet mode (minimal output) +pytest tests/ -q + +# Show print statements +pytest tests/ -s + +# Stop on first failure +pytest tests/ -x + +# Show local variables on failure +pytest tests/ -l + +# Show top 10 slowest tests +pytest tests/ --durations=10 +``` + +## Test Categories + +### Configuration Tests (`tests/config/`) + +Testing configuration loading, validation, and schema enforcement: + +- **test_config.py**: Main experiment configuration +- **test_config_loader.py**: YAML loading and environment variables +- **test_device_config.py**: Device configuration validation +- **test_openvino_config.py**: OpenVINO modes (build/install/link) + +### Device Tests (`tests/devices/`) + +Testing device abstraction and operations: + +- **test_android_device.py**: Android device via ADB +- **test_android_device_complete.py**: Extended Android functionality +- **test_ssh_device.py**: Linux SSH device operations +- **test_devices.py**: Device factory and abstraction + +### Pipeline Tests (`tests/pipeline/`) + +Testing pipeline orchestration and stages: + +- **test_pipeline.py**: Main pipeline flow +- **test_pipeline_openvino_modes.py**: OpenVINO distribution modes + +### Core Tests (`tests/core/`) + +Testing utility functions and helpers: + +- **test_core_artifacts.py**: Artifact management +- **test_core_fs.py**: File system operations +- **test_core_logging.py**: Structured logging +- **test_core_shell.py**: Shell command execution + +### Builder Tests (`tests/builders/`) + +Testing build system integration: + +- **test_builders_openvino.py**: OpenVINO cross-compilation + +### Packaging Tests (`tests/packaging/`) + +Testing bundle creation: + +- **test_packaging_packager.py**: Archive creation and management + +### Runner Tests (`tests/runners/`) + +Testing benchmark execution: + +- **test_runners_benchmark.py**: Matrix expansion and execution + +### Parser Tests (`tests/parsers/`) + +Testing output parsing: + +- **test_parser.py**: Benchmark output parsing + +### Report Tests (`tests/report/`) + +Testing report generation: + +- **test_report_sink.py**: JSON/CSV output generation + +### CLI Tests (`tests/cli/`) + +Testing command-line interface: + +- **test_cli.py**: CLI commands and options +- **test_typer_patch.py**: Typer framework extensions + +## Writing Tests + +### Test Structure + +Follow these conventions: + +1. **File naming**: `test__.py` +2. **Class naming**: `Test` +3. **Method naming**: `test__` +4. **Docstrings**: Brief description of what's being tested + +### Example Test + +```python +"""Tests for OpenVINO configuration.""" + +import pytest +from pydantic import ValidationError + +from ovmobilebench.config.schema import OpenVINOConfig + + +class TestOpenVINOConfig: + """Test OpenVINO configuration validation.""" + + def test_build_mode_valid(self): + """Test valid build mode configuration.""" + config = OpenVINOConfig( + mode="build", + source_dir="/path/to/openvino", + commit="HEAD", + build_type="Release" + ) + assert config.mode == "build" + assert config.source_dir == "/path/to/openvino" + + def test_build_mode_missing_source_dir(self): + """Test build mode without required source_dir.""" + with pytest.raises(ValidationError) as exc_info: + OpenVINOConfig(mode="build") + + assert "source_dir is required" in str(exc_info.value) + + @pytest.mark.parametrize("mode,field", [ + ("build", "source_dir"), + ("install", "install_dir"), + ("link", "archive_url"), + ]) + def test_mode_required_fields(self, mode, field): + """Test that each mode requires its specific field.""" + with pytest.raises(ValidationError): + OpenVINOConfig(mode=mode) +``` + +### Using Fixtures + +```python +import pytest +from unittest.mock import Mock, patch + + +@pytest.fixture +def mock_config(): + """Create mock experiment configuration.""" + config = Mock() + config.project.name = "test" + config.project.run_id = "test-123" + config.openvino.mode = "build" + config.device.kind = "android" + return config + + +@pytest.fixture +def temp_dir(tmp_path): + """Create temporary directory for testing.""" + test_dir = tmp_path / "test_workspace" + test_dir.mkdir() + return test_dir + + +class TestPipeline: + """Test pipeline orchestration.""" + + def test_build_mode(self, mock_config, temp_dir): + """Test pipeline with build mode.""" + with patch("ovmobilebench.pipeline.ensure_dir") as mock_ensure: + mock_ensure.return_value = temp_dir + + pipeline = Pipeline(mock_config) + result = pipeline.build() + + assert result.exists() +``` + +### Mocking Best Practices + +```python +# Mock external dependencies +@patch("ovmobilebench.devices.android.AdbClient") +def test_android_device(mock_adb): + """Test Android device operations.""" + mock_device = Mock() + mock_adb.return_value.device.return_value = mock_device + + device = AndroidDevice("serial123") + device.push("local.txt", "/remote/path") + + mock_device.push.assert_called_once() + +# Mock file operations +@patch("pathlib.Path.exists") +@patch("pathlib.Path.mkdir") +def test_ensure_dir(mock_mkdir, mock_exists): + """Test directory creation.""" + mock_exists.return_value = False + + ensure_dir("/test/path") + + mock_mkdir.assert_called_once() +``` + +## Test Markers + +Use pytest markers to categorize tests: + +```python +@pytest.mark.slow +def test_long_running_operation(): + """Test that takes more than 5 seconds.""" + pass + +@pytest.mark.unit +def test_unit_functionality(): + """Fast unit test.""" + pass + +@pytest.mark.integration +def test_integration_scenario(): + """Test requiring multiple components.""" + pass + +@pytest.mark.skipif(sys.platform == "win32", reason="Linux only") +def test_linux_specific(): + """Test for Linux platform only.""" + pass +``` + +## Coverage Requirements + +### Target Coverage + +- **Overall**: 80%+ coverage +- **Critical modules**: 90%+ coverage + - `ovmobilebench.config` + - `ovmobilebench.pipeline` + - `ovmobilebench.devices` + +### Checking Coverage + +```bash +# Check current coverage +pytest tests/ --cov=ovmobilebench --cov-report=term + +# Find uncovered lines +pytest tests/ --cov=ovmobilebench --cov-report=term-missing + +# Fail if coverage below threshold +pytest tests/ --cov=ovmobilebench --cov-fail-under=80 +``` + +## Continuous Integration + +### GitHub Actions + +Tests run automatically on: + +- Pull requests +- Push to main branch +- Scheduled nightly runs + +Configuration in `.github/workflows/test.yml`: + +```yaml +- name: Run tests + run: | + pytest tests/ --cov=ovmobilebench --cov-report=xml + +- name: Upload coverage + uses: codecov/codecov-action@v3 + with: + file: ./coverage.xml +``` + +### Pre-commit Hooks + +Tests can be run before commit: + +```yaml +# .pre-commit-config.yaml +- repo: local + hooks: + - id: pytest + name: pytest + entry: pytest tests/ -x + language: system + pass_filenames: false + always_run: true +``` + +## Debugging Tests + +### Using pdb + +```python +def test_complex_logic(): + """Test with debugging.""" + import pdb; pdb.set_trace() + # Code to debug +``` + +Run with: + +```bash +pytest tests/ -s # Don't capture output +``` + +### Verbose Failure Output + +```bash +# Show full diff +pytest tests/ -vv + +# Show local variables +pytest tests/ -l + +# Show full traceback +pytest tests/ --tb=long +``` + +### Test Isolation + +```python +# Use tmp_path fixture for file operations +def test_file_operation(tmp_path): + """Test with isolated file system.""" + test_file = tmp_path / "test.txt" + test_file.write_text("content") + + assert test_file.exists() + # tmp_path is automatically cleaned up +``` + +## Performance Testing + +### Benchmarking + +```python +@pytest.mark.benchmark +def test_performance(benchmark): + """Test performance of operation.""" + result = benchmark(expensive_operation, arg1, arg2) + assert result == expected +``` + +Run benchmarks: + +```bash +pytest tests/ --benchmark-only +pytest tests/ --benchmark-compare +``` + +### Profiling + +```bash +# Profile test execution +pytest tests/ --profile + +# Generate call graph +pytest tests/ --profile-svg +``` + +## Test Data + +### Using Fixtures for Test Data + +```python +@pytest.fixture +def sample_yaml(): + """Sample YAML configuration.""" + return """ + project: + name: test + run_id: test-123 + openvino: + mode: build + source_dir: /test/path + """ + +def test_config_loading(sample_yaml, tmp_path): + """Test configuration loading.""" + config_file = tmp_path / "config.yaml" + config_file.write_text(sample_yaml) + + config = load_config(config_file) + assert config.project.name == "test" +``` + +### Parametrized Test Data + +```python +@pytest.mark.parametrize("input,expected", [ + ("1.5K", 1536), + ("2.5M", 2621440), + ("1.2G", 1288490188), +]) +def test_parse_size(input, expected): + """Test size parsing.""" + assert parse_size(input) == expected +``` + +## Troubleshooting + +### Common Issues + +1. **Import errors**: Ensure PYTHONPATH includes project root + + ```bash + export PYTHONPATH="${PYTHONPATH}:$(pwd)" + ``` + +2. **Missing dependencies**: Install test requirements + + ```bash + pip install -e ".[test]" + ``` + +3. **Flaky tests**: Use retry mechanism + + ```python + @pytest.mark.flaky(reruns=3) + def test_network_operation(): + pass + ``` + +4. **Slow tests**: Mark and skip in CI + + ```python + @pytest.mark.slow + @pytest.mark.skipif(os.getenv("CI"), reason="Skip slow tests in CI") + def test_slow_operation(): + pass + ``` + +## Best Practices + +1. **Keep tests fast**: Mock external dependencies +2. **Test one thing**: Each test should verify single behavior +3. **Use descriptive names**: Test names should explain what's tested +4. **Avoid test interdependence**: Tests should run in any order +5. **Use fixtures**: Share setup code via fixtures +6. **Assert specific errors**: Check error messages, not just exceptions +7. **Clean up resources**: Use context managers and fixtures +8. **Document complex tests**: Add comments for non-obvious logic diff --git a/tests/builders/__init__.py b/tests/builders/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/test_builders_openvino.py b/tests/builders/test_builders_openvino.py similarity index 100% rename from tests/test_builders_openvino.py rename to tests/builders/test_builders_openvino.py diff --git a/tests/cli/__init__.py b/tests/cli/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/test_cli.py b/tests/cli/test_cli.py similarity index 100% rename from tests/test_cli.py rename to tests/cli/test_cli.py diff --git a/tests/test_typer_patch.py b/tests/cli/test_typer_patch.py similarity index 100% rename from tests/test_typer_patch.py rename to tests/cli/test_typer_patch.py diff --git a/tests/config/__init__.py b/tests/config/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/test_config.py b/tests/config/test_config.py similarity index 100% rename from tests/test_config.py rename to tests/config/test_config.py diff --git a/tests/test_config_loader.py b/tests/config/test_config_loader.py similarity index 100% rename from tests/test_config_loader.py rename to tests/config/test_config_loader.py diff --git a/tests/test_device_config.py b/tests/config/test_device_config.py similarity index 100% rename from tests/test_device_config.py rename to tests/config/test_device_config.py diff --git a/tests/test_openvino_config.py b/tests/config/test_openvino_config.py similarity index 100% rename from tests/test_openvino_config.py rename to tests/config/test_openvino_config.py diff --git a/tests/conftest.py b/tests/conftest.py index d722fc8..4f5fb98 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -9,7 +9,7 @@ def pytest_collection_modifyitems(config, items): """Automatically skip tests listed in skip_list.txt.""" skip_list_file = Path(__file__).parent / "skip_list.txt" - # Read skip list + # Read the skip list skip_list = set() if skip_list_file.exists(): with open(skip_list_file) as f: diff --git a/tests/core/__init__.py b/tests/core/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/test_core_artifacts.py b/tests/core/test_core_artifacts.py similarity index 100% rename from tests/test_core_artifacts.py rename to tests/core/test_core_artifacts.py diff --git a/tests/test_core_fs.py b/tests/core/test_core_fs.py similarity index 100% rename from tests/test_core_fs.py rename to tests/core/test_core_fs.py diff --git a/tests/test_core_logging.py b/tests/core/test_core_logging.py similarity index 100% rename from tests/test_core_logging.py rename to tests/core/test_core_logging.py diff --git a/tests/test_core_shell.py b/tests/core/test_core_shell.py similarity index 100% rename from tests/test_core_shell.py rename to tests/core/test_core_shell.py diff --git a/tests/devices/__init__.py b/tests/devices/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/test_android_device.py b/tests/devices/test_android_device.py similarity index 100% rename from tests/test_android_device.py rename to tests/devices/test_android_device.py diff --git a/tests/test_android_device_complete.py b/tests/devices/test_android_device_complete.py similarity index 100% rename from tests/test_android_device_complete.py rename to tests/devices/test_android_device_complete.py diff --git a/tests/test_devices.py b/tests/devices/test_devices.py similarity index 100% rename from tests/test_devices.py rename to tests/devices/test_devices.py diff --git a/tests/test_ssh_device.py b/tests/devices/test_ssh_device.py similarity index 100% rename from tests/test_ssh_device.py rename to tests/devices/test_ssh_device.py diff --git a/tests/test_ssh_device_ci.py b/tests/devices/test_ssh_device_ci.py similarity index 96% rename from tests/test_ssh_device_ci.py rename to tests/devices/test_ssh_device_ci.py index e3e7a61..006e6e4 100644 --- a/tests/test_ssh_device_ci.py +++ b/tests/devices/test_ssh_device_ci.py @@ -166,19 +166,19 @@ def test_ssh_in_ci(): if success: print("[OK] SSH connection test successful!") - return True else: print("[FAIL] SSH test failed") - return False + assert False, "SSH test failed" except Exception as e: print(f"[ERROR] SSH test error: {e}") - return False + assert False, f"SSH test error: {e}" if __name__ == "__main__": - success = test_ssh_in_ci() - if not success: + try: + test_ssh_in_ci() + sys.exit(0) + except AssertionError: print("\nSSH test failed") sys.exit(1) - sys.exit(0) diff --git a/tests/packaging/__init__.py b/tests/packaging/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/test_packaging_packager.py b/tests/packaging/test_packaging_packager.py similarity index 100% rename from tests/test_packaging_packager.py rename to tests/packaging/test_packaging_packager.py diff --git a/tests/parsers/__init__.py b/tests/parsers/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/test_parser.py b/tests/parsers/test_parser.py similarity index 100% rename from tests/test_parser.py rename to tests/parsers/test_parser.py diff --git a/tests/pipeline/__init__.py b/tests/pipeline/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/test_pipeline.py b/tests/pipeline/test_pipeline.py similarity index 100% rename from tests/test_pipeline.py rename to tests/pipeline/test_pipeline.py diff --git a/tests/test_pipeline_openvino_modes.py b/tests/pipeline/test_pipeline_openvino_modes.py similarity index 100% rename from tests/test_pipeline_openvino_modes.py rename to tests/pipeline/test_pipeline_openvino_modes.py diff --git a/tests/report/__init__.py b/tests/report/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/test_report_sink.py b/tests/report/test_report_sink.py similarity index 100% rename from tests/test_report_sink.py rename to tests/report/test_report_sink.py diff --git a/tests/runners/__init__.py b/tests/runners/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/test_runners_benchmark.py b/tests/runners/test_runners_benchmark.py similarity index 100% rename from tests/test_runners_benchmark.py rename to tests/runners/test_runners_benchmark.py diff --git a/tests/skip_list.txt b/tests/skip_list.txt index 65e31ad..e6a0181 100644 --- a/tests/skip_list.txt +++ b/tests/skip_list.txt @@ -1,63 +1,78 @@ # List of tests to skip temporarily # Format: test_file.py::TestClass::test_method or test_file.py::test_function # Lines starting with # are comments +# Updated for new directory structure -# Android device tests - complex adbutils mocking required -test_android_device_complete.py::TestAndroidDeviceComplete::test_shell_with_timeout -test_android_device_complete.py::TestAndroidDeviceComplete::test_shell_with_exception -test_android_device_complete.py::TestAndroidDeviceComplete::test_exists_with_exception -test_android_device_complete.py::TestAndroidDeviceComplete::test_get_cpu_info -test_android_device_complete.py::TestAndroidDeviceComplete::test_get_memory_info -test_android_device_complete.py::TestAndroidDeviceComplete::test_get_gpu_info -test_android_device_complete.py::TestAndroidDeviceComplete::test_get_battery_info -test_android_device_complete.py::TestAndroidDeviceComplete::test_set_performance_mode -test_android_device_complete.py::TestAndroidDeviceComplete::test_start_screen_record -test_android_device_complete.py::TestAndroidDeviceComplete::test_stop_screen_record -test_android_device_complete.py::TestAndroidDeviceComplete::test_uninstall_apk -test_android_device_complete.py::TestAndroidDeviceComplete::test_forward_reverse_ports -test_android_device_complete.py::TestAndroidDeviceComplete::test_get_prop -test_android_device_complete.py::TestAndroidDeviceComplete::test_set_prop -test_android_device_complete.py::TestAndroidDeviceComplete::test_clear_logcat -test_android_device_complete.py::TestAndroidDeviceComplete::test_get_logcat +# ============================================ +# Android Device Tests - Complex Mocking Required +# ============================================ +tests/devices/test_android_device_complete.py::TestAndroidDeviceComplete::test_shell_with_timeout +tests/devices/test_android_device_complete.py::TestAndroidDeviceComplete::test_shell_with_exception +tests/devices/test_android_device_complete.py::TestAndroidDeviceComplete::test_exists_with_exception +tests/devices/test_android_device_complete.py::TestAndroidDeviceComplete::test_get_cpu_info +tests/devices/test_android_device_complete.py::TestAndroidDeviceComplete::test_get_memory_info +tests/devices/test_android_device_complete.py::TestAndroidDeviceComplete::test_get_gpu_info +tests/devices/test_android_device_complete.py::TestAndroidDeviceComplete::test_get_battery_info +tests/devices/test_android_device_complete.py::TestAndroidDeviceComplete::test_set_performance_mode +tests/devices/test_android_device_complete.py::TestAndroidDeviceComplete::test_start_screen_record +tests/devices/test_android_device_complete.py::TestAndroidDeviceComplete::test_stop_screen_record +tests/devices/test_android_device_complete.py::TestAndroidDeviceComplete::test_uninstall_apk +tests/devices/test_android_device_complete.py::TestAndroidDeviceComplete::test_forward_reverse_ports +tests/devices/test_android_device_complete.py::TestAndroidDeviceComplete::test_get_prop +tests/devices/test_android_device_complete.py::TestAndroidDeviceComplete::test_set_prop +tests/devices/test_android_device_complete.py::TestAndroidDeviceComplete::test_clear_logcat +tests/devices/test_android_device_complete.py::TestAndroidDeviceComplete::test_get_logcat -# CLI tests - import and mock issues -test_cli.py::TestCLI::test_list_devices_command -test_cli.py::TestCLI::test_list_ssh_devices_command -test_cli.py::TestCLI::test_list_ssh_devices_empty -test_cli.py::TestCLI::test_version_callback +# ============================================ +# CLI Tests - Import and Mock Issues +# ============================================ +tests/cli/test_cli.py::TestCLI::test_list_devices_command +tests/cli/test_cli.py::TestCLI::test_list_ssh_devices_command +tests/cli/test_cli.py::TestCLI::test_list_ssh_devices_empty +tests/cli/test_cli.py::TestCLI::test_version_callback -# Pipeline tests - need fixes -test_pipeline.py::TestPipeline::test_report_dry_run -test_pipeline.py::TestPipeline::test_get_device_android -test_pipeline.py::TestPipeline::test_deploy -test_pipeline.py::TestPipeline::test_deploy_error -test_pipeline.py::TestPipeline::test_run -test_pipeline.py::TestPipeline::test_report +# ============================================ +# Pipeline Tests - Mock Configuration Issues +# ============================================ +tests/pipeline/test_pipeline.py::TestPipeline::test_deploy +tests/pipeline/test_pipeline.py::TestPipeline::test_deploy_error +tests/pipeline/test_pipeline.py::TestPipeline::test_run +tests/pipeline/test_pipeline.py::TestPipeline::test_report +tests/pipeline/test_pipeline.py::TestPipeline::test_report_dry_run +tests/pipeline/test_pipeline.py::TestPipeline::test_get_device_android -# Core artifacts tests - mock issues -test_core_artifacts.py::TestArtifactManager::test_register_artifact_file -test_core_artifacts.py::TestArtifactManager::test_register_artifact_directory -test_core_artifacts.py::TestArtifactManager::test_cleanup_old_artifacts +# ============================================ +# Core Artifacts Tests - Path Mock Issues +# ============================================ +tests/core/test_core_artifacts.py::TestArtifactManager::test_register_artifact_file +tests/core/test_core_artifacts.py::TestArtifactManager::test_register_artifact_directory +tests/core/test_core_artifacts.py::TestArtifactManager::test_cleanup_old_artifacts -# Core fs tests - mock issues -test_core_fs.py::TestCopyTree::test_copy_tree_file_permission_error -test_core_fs.py::TestFormatSize::test_format_size_kilobytes -test_core_fs.py::TestFormatSize::test_format_size_megabytes -test_core_fs.py::TestFormatSize::test_format_size_gigabytes +# ============================================ +# Core FS Tests - Function Import Issues +# ============================================ +tests/core/test_core_fs.py::TestCopyTree::test_copy_tree_file_permission_error +tests/core/test_core_fs.py::TestFormatSize::test_format_size_kilobytes +tests/core/test_core_fs.py::TestFormatSize::test_format_size_megabytes +tests/core/test_core_fs.py::TestFormatSize::test_format_size_gigabytes -# Packaging tests - mock issues -test_packaging_packager.py::TestPackager::test_create_bundle_custom_name -test_packaging_packager.py::TestPackager::test_create_bundle_missing_libs -test_packaging_packager.py::TestPackager::test_create_bundle_with_extra_files -test_packaging_packager.py::TestPackager::test_copy_libs -test_packaging_packager.py::TestPackager::test_copy_libs_no_files -test_packaging_packager.py::TestPackager::test_copy_libs_directories_ignored -test_packaging_packager.py::TestPackager::test_copy_models_success -test_packaging_packager.py::TestPackager::test_copy_models_missing_xml -test_packaging_packager.py::TestPackager::test_copy_models_missing_bin -test_packaging_packager.py::TestPackager::test_create_bundle_logs_completion +# ============================================ +# Packaging Tests - Bundle Creation Mock Issues +# ============================================ +tests/packaging/test_packaging_packager.py::TestPackager::test_create_bundle_custom_name +tests/packaging/test_packaging_packager.py::TestPackager::test_create_bundle_missing_libs +tests/packaging/test_packaging_packager.py::TestPackager::test_create_bundle_with_extra_files +tests/packaging/test_packaging_packager.py::TestPackager::test_copy_libs +tests/packaging/test_packaging_packager.py::TestPackager::test_copy_libs_no_files +tests/packaging/test_packaging_packager.py::TestPackager::test_copy_libs_directories_ignored +tests/packaging/test_packaging_packager.py::TestPackager::test_copy_models_success +tests/packaging/test_packaging_packager.py::TestPackager::test_copy_models_missing_xml +tests/packaging/test_packaging_packager.py::TestPackager::test_copy_models_missing_bin +tests/packaging/test_packaging_packager.py::TestPackager::test_create_bundle_logs_completion -# Android installer CLI tests - mock setup issues +# ============================================ +# Android Installer CLI Tests - Environment Setup +# ============================================ tests/android/installer/test_cli.py::TestAndroidInstallerCLI::test_setup_command_basic tests/android/installer/test_cli.py::TestAndroidInstallerCLI::test_setup_command_with_ndk tests/android/installer/test_cli.py::TestAndroidInstallerCLI::test_setup_command_dry_run @@ -69,63 +84,59 @@ tests/android/installer/test_cli.py::TestAndroidInstallerCLI::test_setup_verbose tests/android/installer/test_cli.py::TestAndroidInstallerCLI::test_setup_with_jsonl tests/android/installer/test_cli.py::TestAndroidInstallerCLI::test_setup_with_force -# Android installer NDK coverage tests - resolver path issues +# ============================================ +# Android Installer Core Tests - Filesystem Access +# ============================================ +tests/android/installer/test_core.py::TestAndroidInstaller::test_ensure_low_disk_space_warning +tests/android/installer/test_core.py::TestAndroidInstaller::test_ensure_logs_host_info + +# ============================================ +# Android Installer Integration Tests - External Dependencies +# ============================================ +tests/android/installer/test_integration.py::TestAndroidInstallerIntegration::test_full_installation_flow +tests/android/installer/test_integration.py::TestAndroidInstallerIntegration::test_ndk_only_installation +tests/android/installer/test_integration.py::TestAndroidInstallerIntegration::test_environment_export +tests/android/installer/test_integration.py::TestAndroidInstallerIntegration::test_api_function_export +tests/android/installer/test_integration.py::TestAndroidInstallerIntegration::test_concurrent_component_installation +tests/android/installer/test_integration.py::TestEndToEndScenarios::test_ci_environment_setup +tests/android/installer/test_integration.py::TestEndToEndScenarios::test_development_environment_setup +tests/android/installer/test_integration.py::TestEndToEndScenarios::test_windows_environment_setup + +# ============================================ +# Android Installer NDK Coverage Tests - Download Required +# ============================================ +tests/android/installer/test_ndk_coverage.py::TestNdkResolverCoverage::test_install_via_download_zip_success +tests/android/installer/test_ndk_coverage.py::TestNdkResolverCoverage::test_install_via_download_network_error +tests/android/installer/test_ndk_coverage.py::TestNdkResolverCoverage::test_install_via_download_http_error +tests/android/installer/test_ndk_coverage.py::TestNdkResolverCoverage::test_install_via_download_tar_success +tests/android/installer/test_ndk_coverage.py::TestNdkResolverCoverage::test_install_via_download_dmg_success +tests/android/installer/test_ndk_coverage.py::TestNdkResolverCoverage::test_install_via_download_unpack_error +tests/android/installer/test_ndk_coverage.py::TestNdkResolverCoverage::test_install_via_download_no_valid_ndk +tests/android/installer/test_ndk_coverage.py::TestNdkResolverCoverage::test_get_download_url +tests/android/installer/test_ndk_coverage.py::TestNdkResolverCoverage::test_install_ndk_with_sdkmanager_success tests/android/installer/test_ndk_coverage.py::TestNdkResolverCoverage::test_resolve_path_with_ndk_home_env tests/android/installer/test_ndk_coverage.py::TestNdkResolverCoverage::test_resolve_path_with_android_ndk_env tests/android/installer/test_ndk_coverage.py::TestNdkResolverCoverage::test_resolve_path_env_invalid tests/android/installer/test_ndk_coverage.py::TestNdkResolverCoverage::test_list_installed_with_multiple_ndks -# Additional Android device complete tests - mock issues -tests/test_android_device_complete.py::TestAndroidDeviceComplete::test_shell_with_timeout -tests/test_android_device_complete.py::TestAndroidDeviceComplete::test_shell_with_exception -tests/test_android_device_complete.py::TestAndroidDeviceComplete::test_exists_with_exception -tests/test_android_device_complete.py::TestAndroidDeviceComplete::test_get_cpu_info -tests/test_android_device_complete.py::TestAndroidDeviceComplete::test_get_memory_info -tests/test_android_device_complete.py::TestAndroidDeviceComplete::test_get_gpu_info -tests/test_android_device_complete.py::TestAndroidDeviceComplete::test_get_battery_info -tests/test_android_device_complete.py::TestAndroidDeviceComplete::test_set_performance_mode -tests/test_android_device_complete.py::TestAndroidDeviceComplete::test_start_screen_record -tests/test_android_device_complete.py::TestAndroidDeviceComplete::test_stop_screen_record -tests/test_android_device_complete.py::TestAndroidDeviceComplete::test_uninstall_apk -tests/test_android_device_complete.py::TestAndroidDeviceComplete::test_forward_reverse_ports -tests/test_android_device_complete.py::TestAndroidDeviceComplete::test_get_prop -tests/test_android_device_complete.py::TestAndroidDeviceComplete::test_set_prop -tests/test_android_device_complete.py::TestAndroidDeviceComplete::test_clear_logcat -tests/test_android_device_complete.py::TestAndroidDeviceComplete::test_get_logcat - -# Additional CLI tests - import and mock issues -tests/test_cli.py::TestCLI::test_list_devices_command -tests/test_cli.py::TestCLI::test_list_ssh_devices_command -tests/test_cli.py::TestCLI::test_list_ssh_devices_empty -tests/test_cli.py::TestCLI::test_version_callback - -# Additional pipeline tests - mock issues -tests/test_pipeline.py::TestPipeline::test_deploy -tests/test_pipeline.py::TestPipeline::test_deploy_error -tests/test_pipeline.py::TestPipeline::test_run -tests/test_pipeline.py::TestPipeline::test_report -tests/test_pipeline.py::TestPipeline::test_report_dry_run -tests/test_pipeline.py::TestPipeline::test_get_device_android +# ============================================ +# Android Installer NDK Tests - External Tool Required +# ============================================ +tests/android/installer/test_ndk.py::TestNdkResolver::test_install_via_download_zip -# Additional core artifacts tests - mock issues -tests/test_core_artifacts.py::TestArtifactManager::test_register_artifact_file -tests/test_core_artifacts.py::TestArtifactManager::test_register_artifact_directory -tests/test_core_artifacts.py::TestArtifactManager::test_cleanup_old_artifacts +# ============================================ +# Android Installer SDK Manager Tests - External Tool Required +# ============================================ +tests/android/installer/test_sdkmanager.py::TestSdkManager::test_get_sdkmanager_path_windows +tests/android/installer/test_sdkmanager.py::TestSdkManager::test_ensure_cmdline_tools_install +tests/android/installer/test_sdkmanager.py::TestSdkManager::test_ensure_cmdline_tools_download_failure -# Additional core fs tests - mock issues -tests/test_core_fs.py::TestCopyTree::test_copy_tree_file_permission_error -tests/test_core_fs.py::TestFormatSize::test_format_size_kilobytes -tests/test_core_fs.py::TestFormatSize::test_format_size_megabytes -tests/test_core_fs.py::TestFormatSize::test_format_size_gigabytes +# ============================================ +# Android Installer AVD Tests - Windows Only +# ============================================ +tests/android/installer/test_avd.py::TestAvdManager::test_get_avdmanager_path_windows -# Additional packaging tests - mock issues -tests/test_packaging_packager.py::TestPackager::test_create_bundle_custom_name -tests/test_packaging_packager.py::TestPackager::test_create_bundle_missing_libs -tests/test_packaging_packager.py::TestPackager::test_create_bundle_with_extra_files -tests/test_packaging_packager.py::TestPackager::test_copy_libs -tests/test_packaging_packager.py::TestPackager::test_copy_libs_no_files -tests/test_packaging_packager.py::TestPackager::test_copy_libs_directories_ignored -tests/test_packaging_packager.py::TestPackager::test_copy_models_success -tests/test_packaging_packager.py::TestPackager::test_copy_models_missing_xml -tests/test_packaging_packager.py::TestPackager::test_copy_models_missing_bin -tests/test_packaging_packager.py::TestPackager::test_create_bundle_logs_completion +# ============================================ +# Android Installer Environment Tests - GitHub Actions Only +# ============================================ +tests/android/installer/test_env.py::TestEnvExporter::test_export_to_github_env