Repository navigation
Experiment Tracking and Reproducibility
Record · Reproduce · Compare · Validate · Improve
A systematic methodology for designing, tracking, comparing, validating, and reproducing machine learning experiments using EveryO.
Home · Model Training · Benchmarking Guide · Numerical Accuracy and Validation
Repository: krishanth7/EveryO
Document Classification: Research and engineering methodology
Framework Status: Experimental / Alpha
Wiki Page: 39 of 40
Important
Reproducibility Notice
EveryO provides neural computing, training, model serialization, and diagnostic capabilities. A complete experiment-tracking platform with a hosted dashboard, centralized experiment database, or automatic artifact registry is not established as a built-in EveryO feature.
The file structures, metadata formats, and tracking utilities in this guide are recommended application-level patterns. They can be implemented alongside EveryO without implying that the framework provides them automatically.
Reproducibility is a measurable engineering objective, not a guarantee that all hardware and software environments will produce bitwise-identical results.
- Overview
- Objectives
- Reproducibility Principles
- Experiment Lifecycle
- Experiment Architecture
- Environment Preparation
- Experiment Identification
- Configuration Management
- Random Seed Management
- Dataset Versioning
- Data Integrity
- Model Configuration
- Training Configuration
- Metric Tracking
- Experiment Logging
- Artifact Management
- Checkpoint Management
- Model Serialization
- Experiment Comparison
- Hyperparameter Experiments
- Benchmark Reproducibility
- Numerical Reproducibility
- CPU and CUDA Differences
- Distributed Experimentation
- Experiment Metadata Schema
- Practical Tracking Implementation
- Experiment Reporting
- Research Integrity
- Security and Privacy
- Automation and CI
- Troubleshooting
- Reproducibility Checklist
- Official Resources
Experiment tracking is the systematic recording of machine learning development activities.
It enables developers to answer:
- Which model configuration produced a result?
- Which dataset was used?
- Which code revision was executed?
- Which optimizer and learning rate were selected?
- Which metrics were observed?
- Which checkpoint corresponds to the experiment?
- Can another developer reproduce the result?
A well-documented experiment is easier to validate, compare, and maintain.
EveryO experiment management should prioritize five objectives.
| Objective | Description |
|---|---|
| Traceability | Connect results to code, data, and configuration |
| Reproducibility | Enable equivalent experiments to be repeated |
| Comparability | Compare runs under controlled conditions |
| Integrity | Preserve original measurements and limitations |
| Auditability | Document decisions and experimental changes |
A complete experiment record should identify:
- Experiment ID.
- Model architecture.
- Dataset version.
- Training configuration.
- Random seeds.
- Software environment.
- Hardware environment.
- Evaluation results.
- Model artifacts.
- Known limitations.
An experiment can be repeated using equivalent code, configuration, and computational resources.
Independent runs produce results consistent with an expected statistical distribution.
Computations produce outputs within defined numerical tolerances.
Outputs are identical at the binary level.
Bitwise reproducibility is a stronger requirement and may be difficult across hardware, compiler, library, or execution-order changes.
Report differences rather than concealing them.
DEFINE HYPOTHESIS
|
v
DESIGN EXPERIMENT
|
v
FREEZE CONFIGURATION
|
v
RECORD ENVIRONMENT
|
v
EXECUTE RUN
|
v
CAPTURE METRICS
|
v
PRESERVE ARTIFACTS
|
v
VALIDATE RESULTS
|
v
COMPARE RUNS
|
v
DOCUMENT FINDINGS
Recommended application-level states:
| State | Meaning |
|---|---|
| Planned | Configuration prepared |
| Running | Execution underway |
| Completed | Execution finished |
| Failed | Execution ended unexpectedly |
| Interrupted | Execution stopped before completion |
| Validated | Results passed required checks |
| Archived | Results preserved for reference |
These states are a suggested tracking convention, not built-in EveryO status values.
A simple file-based tracking system can be sufficient for individual researchers and small teams.
experiments/
|
+-- EXP-001/
| |
| +-- config.json
| +-- environment.json
| +-- dataset.json
| +-- metrics.jsonl
| +-- results.json
| +-- model.evo
| +-- report.md
|
+-- EXP-002/
| |
| +-- config.json
| +-- environment.json
| +-- dataset.json
| +-- metrics.jsonl
| +-- results.json
| +-- model.evo
| +-- report.md
|
+-- README.md
- Human-readable metadata.
- Simple backup.
- Easy comparison.
- No mandatory tracking server.
- Straightforward integration with existing development tools.
A file-based system does not automatically provide concurrent database transactions, access controls, or a searchable hosted experiment dashboard.
git clone https://github.com/krishanth7/EveryO.git
cd EveryO
python -m venv .venvActivate the environment and install:
python -m pip install -e ".[dev]"everyo info
everyo doctorpython --versionpython -m pip freeze > requirements-lock.txtgit rev-parse HEADRecord whether the working tree contains uncommitted modifications.
A commit hash alone does not capture uncommitted changes.
Each experiment should have a unique identifier.
EXP-0001
EXP-0002
EXP-0003
{
"experiment_id": "EXP-0001",
"name": "baseline-mlp",
"status": "planned",
"framework": "EveryO",
"description": "Baseline multilayer perceptron experiment"
}Use descriptive names that communicate the experiment's objective.
Avoid relying on generic labels such as:
test1
final2
latest-final
Experiment configurations should be recorded before execution.
{
"model": {
"type": "mlp",
"input_features": 8,
"hidden_features": 32,
"output_features": 3
},
"training": {
"epochs": 20,
"batch_size": 32,
"optimizer": "adam",
"learning_rate": 0.001
},
"seed": 42
}This is an example application configuration, not a built-in EveryO schema.
- Record all material parameters.
- Avoid undocumented defaults.
- Preserve the original configuration.
- Record modifications as separate runs.
- Validate configuration values before execution.
Randomness can affect:
- Parameter initialization.
- Data shuffling.
- Dataset splitting.
- Dropout.
- Randomized preprocessing.
import numpy as np
seed = 42
rng = np.random.default_rng(seed)
samples = rng.normal(
size=(100, 8)
)import random
random.seed(42)Seeding one random number generator does not necessarily control all randomness in a complete application.
Record every relevant seed and random generator used.
Dataset changes can materially affect experimental results.
- Dataset name.
- Dataset version.
- Source.
- Sample count.
- Feature count.
- Label definition.
- Preprocessing method.
- Train/validation/test split.
{
"dataset": {
"name": "synthetic-classification",
"version": "1.0",
"samples": 10000,
"features": 8,
"classes": 3,
"split_seed": 42
}
}Training and evaluation sets must not accidentally overlap when the evaluation design requires separation.
Prevent preprocessing or feature engineering from using information that would not be available at the intended prediction time.
Dataset integrity can be supported through cryptographic hashes.
import hashlib
def sha256_file(path):
digest = hashlib.sha256()
with open(path, "rb") as file:
for chunk in iter(
lambda: file.read(1024 * 1024),
b"",
):
digest.update(chunk)
return digest.hexdigest()checksum = sha256_file(
"dataset.csv"
)
print(checksum)A matching hash provides evidence that file bytes have not changed.
A file hash does not establish dataset quality, legality, correctness, or freedom from bias.
Record the model's architecture.
import everyo as eo
model = eo.Sequential(
eo.Linear(8, 32),
eo.ReLU(),
eo.Linear(32, 16),
eo.ReLU(),
eo.Linear(16, 3),
)- Layer types.
- Input dimensions.
- Hidden dimensions.
- Output dimensions.
- Activation functions.
- Initialization configuration.
- Relevant regularization.
For a linear layer with bias:
[ P=I\times O+O ]
where:
- (I): Input features.
- (O): Output features.
For an 8-to-32 linear layer:
[ P=8(32)+32=288 ]
A training record should capture the conditions under which the model was optimized.
| Field | Example |
|---|---|
| Optimizer | Adam |
| Learning rate | 0.001 |
| Epochs | 20 |
| Batch size | 32 |
| Loss | Cross-entropy |
| Device | CPU |
| Seed | 42 |
| Gradient clipping | Record actual setting |
| Scheduler | Record actual setting |
If the learning rate or architecture changes, preserve the original configuration and create a distinct run record.
Metrics help evaluate experiment outcomes.
- Training loss.
- Validation loss.
- Accuracy.
- Learning rate.
- Epoch duration.
Depending on the task:
- Precision.
- Recall.
- F1 score.
- Mean absolute error.
- Mean squared error.
{
"epoch": 1,
"train_loss": 1.24,
"validation_loss": 1.31,
"validation_accuracy": 0.52
}Values are illustrative, not measured EveryO results.
Document how each metric is calculated.
JSON Lines is a practical format for recording metrics over time.
{"epoch": 1, "train_loss": 1.24, "val_loss": 1.31}
{"epoch": 2, "train_loss": 1.08, "val_loss": 1.16}
{"epoch": 3, "train_loss": 0.94, "val_loss": 1.02}import json
from pathlib import Path
def append_metric(path, record):
path = Path(path)
path.parent.mkdir(
parents=True,
exist_ok=True,
)
with path.open(
"a",
encoding="utf-8",
) as file:
file.write(
json.dumps(record) + "\n"
)append_metric(
"experiments/EXP-0001/metrics.jsonl",
{
"epoch": 1,
"train_loss": 1.24,
"val_loss": 1.31,
},
)Avoid silently rewriting historical results.
For concurrent writers, use a suitable coordinated storage mechanism rather than assuming unrestricted append operations are safe.
Artifacts are outputs generated during an experiment.
- Model checkpoints.
- Configuration files.
- Metrics.
- Plots.
- Evaluation reports.
- Benchmark results.
artifacts/
|
+-- checkpoints/
|
+-- metrics/
|
+-- plots/
|
+-- reports/
Record:
- Artifact filename.
- Experiment ID.
- Creation timestamp.
- Content hash.
- Relevant software version.
Checkpoints preserve model state at selected stages.
- Save after every epoch.
- Save best validation result.
- Save periodic recovery checkpoints.
- Save final model.
Select the best checkpoint using a predefined metric.
For example:
[ BestEpoch= \arg\min_t ValidationLoss_t ]
A serialized model artifact is not necessarily a complete training-resumption checkpoint.
Exact resumption may also require:
- Optimizer state.
- Scheduler state.
- Random generator state.
- Current epoch.
- Data iteration state.
Verify which state the selected serialization workflow actually preserves.
EveryO documents the .evo model format.
import everyo as eo
eo.save(
model,
"experiments/EXP-0001/model.evo",
)restored = eo.load(
"experiments/EXP-0001/model.evo"
)model.evo
|
+-- manifest.json
|
+-- parameters.npz
Compare original and restored model outputs on fixed inputs.
Custom modules may require registration and supported reconstruction metadata.
Experiments should be compared under controlled conditions.
| Experiment | Learning Rate | Batch Size | Validation Accuracy |
|---|---|---|---|
| EXP-0001 | 0.001 | 32 | 82.4% |
| EXP-0002 | 0.0005 | 32 | 84.1% |
| EXP-0003 | 0.001 | 64 | 81.7% |
All values are illustrative.
- Use the same evaluation dataset.
- Use the same metric definition.
- Record different random seeds.
- Document architectural differences.
- Consider variability across repeated runs.
Report unsuccessful experiments when they materially inform the conclusion.
Hyperparameters influence training behavior.
- Learning rate.
- Batch size.
- Hidden layer width.
- Network depth.
- Weight decay.
- Dropout rate.
Evaluate combinations from a predefined parameter grid.
Sample configurations from defined distributions.
When testing the effect of one variable, hold other variables constant where practical.
Do not repeatedly tune on the final test dataset.
Use validation data for model selection and reserve the test dataset for appropriate final evaluation.
Performance results depend on execution conditions.
- CPU model.
- GPU model, if applicable.
- RAM.
- Operating system.
- Python version.
- NumPy version.
- EveryO revision.
- Input shape.
- Dtype.
- Batch size.
Use a monotonic high-resolution timer such as:
import time
start = time.perf_counter()
# Execute the workload here.
elapsed = time.perf_counter() - startPerform warm-up iterations where appropriate.
Report distributions, not only a single best result.
Synchronize asynchronous device execution when required.
Floating-point computations may vary due to:
- Dtype.
- Operation order.
- Reduction order.
- Hardware.
- Numerical libraries.
A common acceptance rule is:
[ |a-r|\leq atol+rtol|r| ]
import numpy as np
np.testing.assert_allclose(
actual,
reference,
rtol=1e-5,
atol=1e-7,
)Here, actual and reference must be prepared by the experiment.
Use operation- and dtype-appropriate tolerances.
Do not treat the example thresholds as universal requirements.
EveryO provides CPU execution and optional selected CUDA functionality.
CPU experiments may depend on:
- BLAS implementation.
- Thread count.
- CPU architecture.
- Memory bandwidth.
CUDA experiments additionally depend on:
- GPU model.
- Driver.
- CUDA Toolkit.
- Native extension build.
- Supported kernels.
Do not assume CPU and GPU results must be bitwise identical.
Actual CUDA correctness requires testing on compatible hardware.
EveryO documents experimental process-based distributed functionality.
Record:
- Worker count.
- Process rank configuration.
- Dataset partitioning.
- Synchronization strategy.
- Communication environment.
[ E(p)= \frac{T(1)}{pT(p)}\times100% ]
where:
- (T(1)): Single-worker time.
- (T(p)): Time with (p) workers.
Same-host process testing does not establish cross-machine reliability.
The documented TCP implementation assumes a trusted network.
The following JSON illustrates a consolidated experiment record.
{
"experiment_id": "EXP-0001",
"name": "baseline-mlp",
"status": "completed",
"framework": {
"name": "EveryO",
"version": "record actual version"
},
"source": {
"git_commit": "record actual commit"
},
"dataset": {
"name": "synthetic-classification",
"version": "1.0",
"split_seed": 42
},
"model": {
"architecture": "mlp",
"input_features": 8,
"hidden_features": 32,
"output_features": 3
},
"training": {
"optimizer": "adam",
"learning_rate": 0.001,
"batch_size": 32,
"epochs": 20,
"seed": 42
},
"results": {
"validation_accuracy": null,
"validation_loss": null
},
"artifacts": {
"model": "model.evo",
"metrics": "metrics.jsonl"
}
}The fields above form a suggested project-level schema.
They are not an official EveryO experiment-tracking API.
This example demonstrates a minimal local experiment record writer.
It uses only Python's standard library.
import json
import platform
import sys
from datetime import datetime, timezone
from pathlib import Path
def create_experiment(
experiment_id,
configuration,
root="experiments",
):
directory = Path(root) / experiment_id
directory.mkdir(
parents=True,
exist_ok=False,
)
metadata = {
"experiment_id": experiment_id,
"created_at": datetime.now(
timezone.utc
).isoformat(),
"python_version": sys.version,
"platform": platform.platform(),
"configuration": configuration,
}
metadata_path = directory / "config.json"
metadata_path.write_text(
json.dumps(
metadata,
indent=2,
),
encoding="utf-8",
)
return directoryconfiguration = {
"model": "mlp",
"learning_rate": 0.001,
"epochs": 20,
"seed": 42,
}
experiment_directory = create_experiment(
"EXP-0001",
configuration,
)
print(experiment_directory)The helper:
- Creates a unique experiment directory.
- Records a UTC timestamp.
- Captures Python and platform information.
- Writes the supplied configuration.
- Prevents accidental reuse of an existing experiment directory.
This utility does not automatically capture:
- EveryO framework version.
- Git commit.
- Dataset hashes.
- Training metrics.
- Model checkpoints.
- Hardware details.
These should be added explicitly when required.
Each completed experiment should have a human-readable report.
# Experiment Report
## Experiment Information
- Experiment ID:
- Name:
- Objective:
- Status:
## Environment
- EveryO version:
- Python version:
- Operating system:
- Hardware:
- Git commit:
## Dataset
- Dataset name:
- Version:
- Split strategy:
- Preprocessing:
## Model
- Architecture:
- Parameter count:
## Training
- Optimizer:
- Learning rate:
- Batch size:
- Epochs:
- Random seeds:
## Results
- Training metrics:
- Validation metrics:
- Test metrics:
## Reproducibility
- Configuration:
- Dependency record:
- Dataset hash:
- Model artifact:
## Limitations
Describe known limitations.
## Conclusion
Summarize supported findings.Include tables, plots, or numerical comparisons where they materially support conclusions.
Separate:
- Observed results.
- Interpretations.
- Hypotheses.
- Future work.
Reliable experimentation requires transparent reporting.
- Preserve original results.
- Disclose failed runs.
- Avoid selective reporting.
- Do not fabricate metrics.
- Clearly label synthetic examples.
- Disclose limitations.
Where results depend on random initialization or sampling, use repeated experiments when practical.
Avoid claims such as:
"EveryO is always faster."
Instead, report a specific result with the exact benchmark conditions.
Experiment tracking can contain sensitive information.
- Dataset access credentials.
- API keys.
- Private training data.
- Personally identifiable information.
- Proprietary model artifacts.
- Exclude secrets from logs.
- Restrict artifact access.
- Review files before publication.
- Use trusted dependencies.
- Validate external model archives.
Define how long experiment records and datasets should be retained.
Report suspected EveryO vulnerabilities privately through:
Continuous integration can support reproducible testing.
- Install dependencies.
- Run unit tests.
- Validate formatting.
- Run numerical checks.
- Generate test reports.
- Archive selected artifacts.
python -m pytest -q
ruff check .
ruff format --check .Tests requiring CUDA or other optional infrastructure should report when they are skipped.
Performance regression checks should account for normal timing variability and CI hardware differences.
| Problem | Possible Cause | Recommended Action |
|---|---|---|
| Different results between runs | Randomness | Record all relevant seeds |
| Missing experiment files | Incomplete tracking | Verify artifact writing |
| Changed dataset | Dataset version mismatch | Compare hashes |
| Missing checkpoint | Save failure | Validate artifact paths |
| Different validation metrics | Split or preprocessing mismatch | Compare data pipelines |
| Different CPU performance | BLAS or thread differences | Record environment |
| Different CUDA results | Hardware or precision differences | Compare with tolerances |
| Serialization failure | Unsupported model structure | Verify compatibility |
| Missing dependencies | Environment drift | Recreate the recorded environment |
| Unreproducible benchmark | Timing or workload differences | Standardize methodology |
everyo info
everyo doctor
python -m pytest -q- Hypothesis is documented.
- Objective is measurable.
- Evaluation criteria are defined.
- Experiment ID is unique.
- Python version recorded.
- EveryO version recorded.
- NumPy version recorded.
- Git revision recorded.
- Hardware recorded.
- Dataset version recorded.
- Dataset hash recorded where appropriate.
- Splits are reproducible.
- Preprocessing is documented.
- Data leakage checks are performed.
- Architecture documented.
- Hyperparameters recorded.
- Random seeds recorded.
- Metrics preserved.
- Checkpoint behavior documented.
- Results validated.
- Numerical tolerances justified.
- Failed runs disclosed.
- Limitations documented.
- Reproduction instructions provided.
- Model artifact preserved.
- Configuration preserved.
- Dependency record preserved.
- Metrics preserved.
- Report completed.
| Resource | Link |
|---|---|
| EveryO Repository | GitHub |
| README | README.md |
| Model Training | Model Training |
| Model Serialization | Model Serialization |
| Data Pipeline | Data Pipeline |
| Benchmarking Guide | Benchmarking Guide |
| Numerical Validation | Numerical Accuracy and Validation |
| Testing and QA | Testing and Quality Assurance |
| Model Deployment | Model Deployment and Inference |
| Contributing | CONTRIBUTING.md |
| GitHub Issues | Issues |
| GitHub Discussions | Discussions |
| NumPy | numpy.org |
| Python | python.org |
Documentation Page 39 of 40
Next and Final Page: Documentation Index and Project Handbook →
Home · Benchmarking Guide · Model Training
EveryO — Open Machine Learning. Built for Developers.
Maintainer: @krishanth7
Open Source · MIT Licensed · Community Driven
© 2026 EveryO Contributors