Testing and Quality Assurance

EveryO — Engineering Quality, Numerical Verification & Continuous Integration

Test · Validate · Secure · Review · Release

A comprehensive engineering reference for verifying EveryO's functionality, mathematical correctness, reliability, compatibility, security, and release readiness.

Home · Automatic Differentiation · Performance Optimization · Contributing

Maintained By & Owner: EveryO Contributors & @krishanth7

Important

Documentation Scope

This guide reflects the test configuration and CI workflows inspected in the EveryO repository. It distinguishes existing checks from recommended future quality gates.

The presence of a test or workflow does not establish that the latest run passed. Consult GitHub Actions for actual execution results.

Overview
Quality Engineering Architecture
Development Environment
Testing Framework
Test Suite Organization
Unit Testing
Automatic Differentiation Testing
Numerical Gradient Verification
Neural Network Testing
Optimizer and Loss Testing
Data Pipeline Testing
Integration Testing
Backend and Device Testing
ONNX and Serialization Testing
Performance Regression Testing
Code Coverage
Static Analysis and Formatting
Continuous Integration
Security and Dependency Quality
Documentation Testing
Cross-Platform Compatibility
Test Reproducibility
Quality Gates and Release Criteria
Troubleshooting
Contributor Testing Checklist
Developer References
Next Steps

1. Overview

Testing and quality assurance ensure that EveryO behaves correctly across its supported functionality.

For a machine learning framework, correctness extends beyond successful execution. Mathematical gradients, numerical stability, model behavior, and serialization compatibility must also be verified.

1.1 Quality Objectives

Functional correctness.
Mathematical correctness.
Numerical stability.
API reliability.
Cross-platform compatibility.
Reproducible behavior.
Regression prevention.
Secure development.
Release confidence.

1.2 Quality Dimensions

Dimension	Objective
Functionality	APIs behave as documented
Mathematics	Operations and gradients are correct
Reliability	Edge cases are handled
Compatibility	Supported environments work
Performance	Regressions are identified
Security	Risks are detected and managed
Maintainability	Changes remain testable
Documentation	Examples match implementation

2. Quality Engineering Architecture

2.1 Testing Pipeline

                    SOURCE CODE
                         |
                         v
                  Static Analysis
                         |
                         v
                    Unit Tests
                         |
                         v
                Numerical Validation
                         |
                         v
                 Integration Tests
                         |
                         v
                  Backend Tests
                         |
                         v
                Documentation Tests
                         |
                         v
                 Package Validation
                         |
                         v
                  Release Review

2.2 Test Categories

Category	Primary Purpose
Unit	Individual components
Integration	Interacting components
Numerical	Mathematical correctness
Regression	Prevent previously fixed defects
Compatibility	Environment differences
Performance	Runtime efficiency
Security	Identify vulnerabilities
Acceptance	Release requirements

2.3 Testing Philosophy

Every meaningful code change should be accompanied by tests appropriate to its risk.

For example, changing tensor broadcasting may require unit tests, gradient tests, shape edge cases, and integration tests.

3. Development Environment

3.1 Clone Repository

git clone https://github.com/krishanth7/EveryO.git

cd EveryO

3.2 Create Virtual Environment

python -m venv .venv

Activate on Linux or macOS:

source .venv/bin/activate

Activate on Windows PowerShell:

.venv\Scripts\Activate.ps1

3.3 Install Development Dependencies

python -m pip install --upgrade pip

python -m pip install -e ".[dev]"

The inspected development extra includes:

pytest
pytest-cov
Ruff
PyYAML

3.4 Verify Installation

python -m everyo info

python -m everyo doctor

4. Testing Framework

EveryO uses pytest.

4.1 Run Tests

python -m pytest

4.2 Quiet Output

python -m pytest -q

4.3 Run Specific Test File

python -m pytest tests/test_autograd.py

4.4 Run a Specific Test

python -m pytest \
  tests/test_autograd.py::TestBasicGradients::test_square_gradient

4.5 Stop on First Failure

python -m pytest -x

4.6 Detailed Output

python -m pytest -v

4.7 Pytest Configuration

The repository declares:

[tool.pytest.ini_options]
minversion = "7.0"
testpaths = ["tests"]
addopts = "-q --strict-markers"

The configuration also treats RuntimeWarning as an error.

5. Test Suite Organization

The inspected repository contains dedicated test files for multiple framework subsystems.

5.1 Examples

tests/
│
├── conftest.py
├── test_autograd.py
├── test_operations.py
├── test_layers.py
├── test_losses.py
├── test_optimizers.py
├── test_dataloader.py
├── test_integration.py
├── test_attention.py
├── test_conv.py
├── test_normalization.py
├── test_devices.py
├── test_distributed.py
├── test_precision.py
├── test_profiler.py
├── test_onnx.py
└── test_onnx_trace.py

This is a representative selection, not an exhaustive directory listing.

5.2 Test Ownership

Subsystem	Test File
Autograd	`test_autograd.py`
Tensor operations	`test_operations.py`
Neural layers	`test_layers.py`
Loss functions	`test_losses.py`
Optimizers	`test_optimizers.py`
DataLoader	`test_dataloader.py`
Integration	`test_integration.py`
Attention	`test_attention.py`
Convolution	`test_conv.py`
Devices	`test_devices.py`
Profiling	`test_profiler.py`
ONNX	`test_onnx.py`

6. Unit Testing

Unit tests verify individual operations in isolation.

6.1 Example: Tensor Addition

import numpy as np
import everyo as eo

def test_tensor_addition():
    a = eo.tensor([1.0, 2.0])
    b = eo.tensor([3.0, 4.0])

    result = a + b

    np.testing.assert_allclose(
        result.numpy(),
        [4.0, 6.0]
    )

6.2 Example: Shape Validation

def test_matrix_multiplication_shape():
    a = eo.ones(3, 4)
    b = eo.ones(4, 5)

    result = eo.matmul(a, b)

    assert result.shape == (3, 5)

6.3 Unit Test Design

A good unit test should:

Have a clear purpose.
Use small deterministic inputs.
Check meaningful outputs.
Cover relevant edge cases.
Produce actionable failure messages.

7. Automatic Differentiation Testing

Automatic differentiation is a critical component of EveryO.

7.1 Gradient Example

For:

[ f(x)=x^2 ]

the derivative is:

[ f'(x)=2x ]

At (x=2):

[ f'(2)=4 ]

7.2 Test

import numpy as np
import everyo as eo

def test_square_gradient():
    x = eo.tensor(
        [2.0],
        requires_grad=True
    )

    y = x * x
    y.backward()

    np.testing.assert_allclose(
        x.grad,
        [4.0]
    )

This test corresponds to an existing gradient test pattern in EveryO.

7.3 Required Gradient Cases

Addition.
Subtraction.
Multiplication.
Division.
Chain rule.
Broadcasting.
Gradient accumulation.
Reused tensors.
Gradient reset.
Matrix multiplication.

8. Numerical Gradient Verification

Analytical gradients should be compared against numerical approximations.

8.1 Central Finite Difference

For scalar-valued (f):

[ \frac{\partial f}{\partial x_i} \approx \frac{f(x_i+\epsilon)-f(x_i-\epsilon)} {2\epsilon} ]

where (\epsilon) is a small perturbation.

8.2 Numerical Gradient Helper

import numpy as np

def numerical_gradient(fn, values, epsilon=1e-5):
    values = np.asarray(
        values,
        dtype=np.float64
    ).copy()

    gradient = np.zeros_like(values)

    for index in np.ndindex(values.shape):
        original = values[index]

        values[index] = original + epsilon
        positive = fn(values)

        values[index] = original - epsilon
        negative = fn(values)

        values[index] = original

        gradient[index] = (
            positive - negative
        ) / (2.0 * epsilon)

    return gradient

8.3 Comparison

np.testing.assert_allclose(
    analytical_gradient,
    numerical_gradient_result,
    rtol=1e-4,
    atol=1e-6
)

These tolerances are consistent with the inspected autograd test configuration, but should be adjusted when numerical conditions require it.

8.4 Limitations

Finite differences may be unreliable near:

Discontinuities.
Non-differentiable points.
Numerical overflow.
Severe cancellation.
Poorly scaled inputs.

9. Neural Network Testing

Neural network layers should be tested independently and in combinations.

9.1 Testing Requirements

Correct output shape.
Parameter initialization.
Forward computation.
Backward gradients.
Batch handling.
Evaluation behavior.
Numerical stability.

9.2 Linear Layer Example

import everyo as eo

def test_linear_output_shape():
    layer = eo.Linear(
        4,
        8,
        seed=0
    )

    x = eo.zeros(16, 4)

    output = layer(x)

    assert output.shape == (16, 8)

9.3 Model-Level Test

def test_sequential_model_shape():
    model = eo.Sequential(
        eo.Linear(4, 8, seed=0),
        eo.ReLU(),
        eo.Linear(8, 2, seed=1)
    )

    output = model(eo.zeros(16, 4))

    assert output.shape == (16, 2)

10. Optimizer and Loss Testing

Optimizers and losses must produce mathematically meaningful results.

10.1 Optimizer Checks

Parameters update after a step.
Zero gradients are handled.
Optimizer state behaves correctly.
Invalid hyperparameters are rejected.
Updates remain numerically stable.

10.2 Loss Checks

Expected output shape.
Known reference values.
Gradient correctness.
Target compatibility.
Numerical stability.

10.3 Testing Recommendations

Use simple functions with known solutions.

For example:

[ L(w)=(w-3)^2 ]

The optimum is:

[ w^*=3 ]

An optimizer test can verify that repeated updates reduce this objective under a suitable learning rate.

11. Data Pipeline Testing

The data pipeline should preserve sample correctness and batch integrity.

11.1 Example

import numpy as np
import everyo as eo

def test_dataloader_batch_count():
    features = np.zeros(
        (10, 2),
        dtype=np.float32
    )

    targets = np.zeros(
        10,
        dtype=np.int64
    )

    dataset = eo.ArrayDataset(
        features,
        targets
    )

    loader = eo.DataLoader(
        dataset,
        batch_size=4
    )

    assert len(loader) == 3

11.2 Important Cases

Empty datasets.
Invalid batch size.
Incomplete final batch.
Shuffling.
Feature-target alignment.
Reproducibility.
Custom dataset collation.

12. Integration Testing

Integration tests verify complete workflows.

12.1 Example Pipeline

Dataset
   |
   v
DataLoader
   |
   v
Neural Network
   |
   v
Loss Function
   |
   v
Autograd
   |
   v
Optimizer
   |
   v
Updated Model

12.2 Recommended Tests

Complete training loop.
Validation workflow.
Callback execution.
Checkpoint restoration.
Prediction after training.
Training history recording.

12.3 Integration Test Requirements

Integration tests should verify not only that execution completes, but that meaningful invariants hold.

For example, an appropriately configured model should demonstrate learning on a simple deterministic problem.

13. Backend and Device Testing

EveryO includes CPU and optional accelerator functionality.

13.1 CPU Baseline

The CPU path should remain testable without CUDA hardware.

13.2 CUDA Tests

CUDA-specific tests should:

Check availability.
Skip gracefully when unavailable.
Verify supported operations.
Compare numerical results with CPU.
Test error handling.

13.3 Device Resolution

from everyo.core.device import device

def test_cpu_device():
    resolved = device("cpu")

    assert resolved.type == "cpu"

13.4 Optional Backend Markers

The repository defines pytest markers:

cuda
tensorflow
slow

Example:

import pytest

@pytest.mark.cuda
def test_cuda_specific_operation():
    ...

Markers classify tests; they do not automatically implement environment checks or skipping.

14. ONNX and Serialization Testing

Model serialization must preserve architecture and numerical behavior.

14.1 Serialization Tests

Save a model.
Load the archive.
Compare parameters.
Compare predictions.
Verify metadata.
Reject malformed files.

14.2 ONNX Testing

The repository includes:

tests/test_onnx.py
tests/test_onnx_trace.py

14.3 ONNX Round-Trip Validation

EveryO Model
      |
      v
   ONNX Export
      |
      v
  ONNX Runtime
      |
      v
Compare Predictions
      |
      v
Numerical Verification

14.4 Recommended Comparison

np.testing.assert_allclose(
    everyo_output,
    onnx_output,
    rtol=1e-5,
    atol=1e-6
)

Choose tolerances appropriate to the operation and backend.

15. Performance Regression Testing

Performance tests detect unintended slowdowns.

15.1 Key Metrics

Forward-pass latency.
Backward-pass latency.
Training throughput.
Memory consumption.
DataLoader throughput.
Serialization time.

15.2 Regression Formula

[ R= \frac{T_{new}-T_{baseline}} {T_{baseline}}\times100 ]

where (T) is elapsed time.

15.3 Recommended Procedure

Record a stable baseline.
Use equivalent inputs.
Warm up the workload.
Run repeated measurements.
Compare distributions.
Investigate meaningful regressions.

15.4 Important Limitation

Performance results from shared CI runners can be noisy.

A dedicated benchmark environment is preferable for enforcing strict performance thresholds.

16. Code Coverage

Code coverage indicates which code was exercised during testing.

16.1 Install Coverage Tool

python -m pip install pytest-cov

16.2 Generate Coverage Report

python -m pytest \
  --cov=everyo \
  --cov-report=term-missing

16.3 Generate HTML Report

python -m pytest \
  --cov=everyo \
  --cov-report=html

Open:

htmlcov/index.html

16.4 Recommended Targets

Area	Proposed Target
Core tensor operations	95%
Autograd	95%
Loss functions	95%
Optimizers	90%
Data utilities	90%
Serialization	90%
Overall project	90%

Note

These are proposed engineering targets, not measured or currently enforced EveryO coverage results.

16.5 Coverage Limitations

High line coverage does not guarantee mathematical correctness.

Combine coverage with meaningful assertions, edge cases, numerical gradient checks, and integration tests.

17. Static Analysis and Formatting

EveryO uses Ruff for linting and formatting checks.

17.1 Lint

ruff check .

17.2 Format Check

ruff format --check .

17.3 Apply Formatting

ruff format .

17.4 Configuration

The repository declares:

[tool.ruff]
line-length = 100
target-version = "py39"

The development dependency pins Ruff to a specific version to reduce formatter drift between local development and CI.

17.5 Recommended Practices

Resolve lint violations before submitting a PR.
Avoid unrelated formatting changes.
Keep imports organized.
Prefer readable and testable code.

18. Continuous Integration

EveryO uses GitHub Actions for automated validation.

18.1 Workflow Files

.github/workflows/
├── tests.yml
└── cuda-build.yml

18.2 Main CI Workflow

The inspected tests.yml runs on:

Pushes.
Pull requests.
Manual workflow dispatch.

18.3 Existing Jobs

Job	Purpose
`test`	Python tests and doctests
`lint`	Ruff checks
`tensorflow`	TensorFlow integration
`examples`	Example scripts
`onnx`	ONNX round-trip testing
`build`	Package build and metadata validation

18.4 Python Compatibility Matrix

The inspected workflow tests:

Platform	Python Versions
Ubuntu	3.9, 3.10, 3.11, 3.12
macOS	3.12
Windows	3.12

18.5 CI Quality Pipeline

Push / Pull Request
        |
        v
   GitHub Actions
        |
        +-- Unit Tests
        |
        +-- Integration Tests
        |
        +-- Doctests
        |
        +-- Ruff Lint
        |
        +-- Formatting
        |
        +-- TensorFlow Tests
        |
        +-- ONNX Tests
        |
        +-- Examples
        |
        +-- Package Build

18.6 CI Status

View the current execution results:

EveryO GitHub Actions

19. Security and Dependency Quality

Security checks should complement functional tests.

19.1 Recommended Security Controls

Dependency vulnerability scanning.
Secret detection.
Static security analysis.
Safe deserialization testing.
Input validation.
Least-privilege CI permissions.

19.2 Dependency Audit

A proposed local audit command is:

python -m pip install pip-audit

python -m pip_audit

This requires the external pip-audit package.

19.3 Recommended Additional Checks

Check	Proposed Tool
Dependency vulnerabilities	pip-audit
Secret scanning	Gitleaks
Python security patterns	Bandit
Dependency updates	Dependabot
Supply-chain review	GitHub dependency review

These are recommendations; they are not claims that every check is already configured in EveryO.

20. Documentation Testing

Documentation examples must remain consistent with the source code.

20.1 Doctests

The existing CI workflow runs:

python -m pytest \
  --doctest-modules everyo -q \
  --ignore=everyo/serialization/onnx_export.py \
  --ignore=everyo/serialization/onnx_trace.py

The ONNX exporters have dedicated doctest execution in the ONNX job.

20.2 Documentation Requirements

Use valid public APIs.
Verify imports.
Include correct expected outputs.
Explain optional dependencies.
Avoid fabricated benchmark values.
Distinguish conceptual examples from executable code.

20.3 Wiki Review Checklist

Page title is accurate.
Code examples are valid.
API signatures match the source.
Internal links are correct.
Version requirements are documented.
Mathematical formulas are verified.
Limitations are explained.

21. Cross-Platform Compatibility

Cross-platform testing helps identify operating-system and runtime differences.

21.1 Important Platforms

Linux.
Windows.
macOS.

21.2 Potential Compatibility Issues

Filesystem path differences.
Environment activation.
Dependency installation.
Numerical library behavior.
Native extension compilation.
Multiprocessing behavior.

21.3 Recommended Practice

Use platform-independent paths:

from pathlib import Path

output = Path("artifacts") / "model.evo"

Avoid hard-coded platform-specific separators when unnecessary.

22. Test Reproducibility

Reproducible tests produce consistent outcomes under controlled conditions.

22.1 Seeded Fixture

EveryO's tests/conftest.py includes a seeded NumPy random generator fixture.

Example:

import numpy as np
import pytest

@pytest.fixture
def rng():
    return np.random.default_rng(1234)

22.2 Reproducibility Requirements

Explicit random seeds.
Deterministic reference inputs.
Stable test tolerances.
Documented dependency versions.
Controlled environment assumptions.

22.3 Avoid Fragile Tests

Do not depend unnecessarily on:

Exact execution timing.
Unstable sample ordering.
External network services.
Machine-specific paths.
Uncontrolled random values.

23. Quality Gates and Release Criteria

The following gates are recommended for a formal EveryO release process.

23.1 Proposed Release Gates

Gate	Recommended Requirement
Unit tests	All required tests pass
Integration tests	All required tests pass
Gradient checks	Within defined tolerances
Lint	No violations
Formatting	No violations
Documentation	Examples validated
Packaging	Build and metadata checks pass
Security	No unresolved critical findings
Compatibility	Supported matrix passes
Release review	Maintainer approval

23.2 Release Decision Model

Candidate Release
       |
       v
Functional Tests
       |
       v
Numerical Validation
       |
       v
Compatibility Tests
       |
       v
Security Review
       |
       v
Package Verification
       |
       v
Maintainer Approval
       |
       v
Release Publication

23.3 Important Distinction

These are recommended release criteria. They become enforceable only when adopted through repository governance, CI configuration, and branch protection or equivalent controls.

24. Troubleshooting

pytest cannot import EveryO

Install the package in editable mode:

python -m pip install -e ".[dev]"

Confirm the intended Python environment is active.

Numerical gradient tests fail

Check analytical gradient formulas, input dtype, finite-difference epsilon, broadcasting behavior, and numerical tolerance.

CUDA tests fail without a GPU

Check that tests requiring CUDA skip appropriately when the required extension is unavailable.

Ruff fails in CI but passes locally

Confirm that the local Ruff version matches the version specified in the development dependencies.

ONNX tests fail

Install the ONNX optional dependencies and verify compatibility between the exporter and ONNX Runtime.

Coverage is lower than expected

Identify untested branches and edge cases. Prioritize meaningful tests rather than writing assertions solely to increase coverage percentages.

25. Contributor Testing Checklist

Before submitting a pull request:

Recommended Local Commands

python -m pytest -q

ruff check .

ruff format --check .

python -m compileall -q everyo

python -m build

The final command requires the build package, which can be installed with:

python -m pip install build

26. Developer References

Resource	Link
EveryO Repository	GitHub
Test Suite	tests/
Test Fixtures	conftest.py
Autograd Tests	test_autograd.py
Integration Tests	test_integration.py
DataLoader Tests	test_dataloader.py
ONNX Tests	test_onnx.py
CI Workflow	tests.yml
CUDA Workflow	cuda-build.yml
Project Configuration	pyproject.toml
GitHub Actions	Workflow Runs
Issues	GitHub Issues

27. Next Steps

Continue with EveryO's development and contributor documentation.

Page	Focus
Development Setup	Contributor environment and tooling
Contributing	Contribution workflow
Coding Standards	Style and maintainability
Pull Request Guidelines	Review requirements
Release Process	Versioning and release management

Wiki links become active after their corresponding pages are created.