Data Pipeline

Data Pipeline

EveryO — Dataset Management, Preprocessing & Data Loading

Collect · Validate · Transform · Batch · Train

A technical guide to EveryO's dataset abstractions, preprocessing utilities, data splitting, batching, CSV integration, and reproducible machine learning data workflows.

Home · Getting Started · Model Training · API Reference

---

Maintained By & Owner: EveryO Contributors & @krishanth7

Important

API Compatibility

This guide is based on the EveryO data package source from the repository's default branch.

The examples use public dataset and preprocessing APIs. Runtime results should be verified with the installed release.

EveryO's CSV reader is designed for numeric tables; complex CSV parsing and categorical encoding may require preprocessing outside the built-in loader.

---

Table of Contents

1. Overview
2. Data Pipeline Architecture
3. Data Module API
4. Dataset Interface
5. ArrayDataset
6. Custom Datasets
7. TransformDataset
8. Subset
9. DataLoader
10. Batching and Shuffling
11. CSV Dataset Loading
12. Data Splitting
13. StandardScaler
14. MinMaxScaler
15. Normalization and Encoding
16. Data Quality and Validation
17. Preventing Data Leakage
18. Complete Data Pipeline Example
19. Training Integration
20. Performance and Memory
21. Testing and Reproducibility
22. Troubleshooting
23. Developer References
24. Next Steps

---

1. Overview

A data pipeline transforms raw information into structured batches suitable for machine learning.

EveryO provides dataset abstractions, preprocessing functions, data splitting, and mini-batch iteration.

1.1 Core Responsibilities

• Organize features and targets.
• Validate dataset shapes.
• Apply transformations.
• Split datasets.
• Standardize or normalize features.
• Encode categorical targets.
• Construct mini-batches.
• Shuffle training samples.
• Supply data to the training engine.

1.2 Data Pipeline Components

Component	Responsibility
Dataset	Base dataset interface
ArrayDataset	NumPy-backed dataset
TransformDataset	Apply sample transformations
Subset	Select dataset samples
DataLoader	Mini-batch iteration
StandardScaler	Feature standardization
MinMaxScaler	Feature scaling
load_csv_dataset	Numeric CSV loading

---

2. Data Pipeline Architecture

2.1 Conceptual Architecture

                  RAW DATA
                     |
                     v
              Data Validation
                     |
                     v
              Data Splitting
                     |
                     v
             Feature Processing
                     |
                     v
              Dataset Objects
                     |
                     v
                 DataLoader
                     |
                     v
               Mini-Batches
                     |
                     v
              EveryO Trainer
                     |
                     v
                Model Output

2.2 Data Processing Principles

A reliable pipeline should be:

• Deterministic where required.
• Easy to test.
• Consistent across training and inference.
• Resistant to data leakage.
• Clear about feature shapes.
• Efficient for the intended dataset size.

---

3. Data Module API

EveryO's everyo.data package exports:

API	Purpose
Dataset	Abstract dataset
ArrayDataset	Array-backed dataset
TransformDataset	Transformation wrapper
Subset	Indexed subset
DataLoader	Mini-batch loader
default_collate	Combine samples into batches
load_csv_dataset	Numeric CSV reader
train_test_split	Training/test splitting
random_split	Random dataset splitting
stratified_split	Class-aware splitting
StandardScaler	Standardization
MinMaxScaler	Range scaling
standardize	Standardization function
min_max_scale	Min-max scaling function
normalize	Unit-norm normalization
one_hot_encode	One-hot label encoding
shuffle_arrays	Shared array shuffling

3.1 Import

import everyo as eo

Or import directly from the data package:

from everyo.data import (
    ArrayDataset,
    DataLoader,
    StandardScaler,
    MinMaxScaler,
)

---

4. Dataset Interface

The Dataset class defines the core dataset contract.

4.1 Required Methods

__len__()
__getitem__(index)

4.2 Responsibilities

__len__() returns the number of samples.

__getitem__() returns a sample or a feature-target pair.

4.3 Conceptual Structure

Dataset
  |
  +-- __len__()
  |
  +-- __getitem__(index)
  |
  +-- __iter__()

4.4 Expected Sample Formats

For supervised learning:

(features, target)

For feature-only datasets:

features

---

5. ArrayDataset

ArrayDataset stores features and optional targets in NumPy-backed arrays.

5.1 Constructor

ArrayDataset(features, targets=None)

5.2 Example

import numpy as np
import everyo as eo

features = np.array([
    [1.0, 2.0],
    [3.0, 4.0],
    [5.0, 6.0],
], dtype=np.float32)

targets = np.array(
    [0, 1, 0],
    dtype=np.int64
)

dataset = eo.ArrayDataset(
    features,
    targets
)

print("Samples:", len(dataset))
print("Feature shape:", dataset.feature_shape)
print("First sample:", dataset[0])

5.3 Expected Properties

Samples: 3
Feature shape: (2,)

5.4 Validation

The implementation checks:

• Features must have at least one dimension.
• Features and targets must contain equal sample counts.
• Unsupported object-type arrays are rejected.

5.5 Feature-Only Dataset

dataset = eo.ArrayDataset(features)

sample = dataset[0]

This is useful for inference or unsupervised workflows.

---

6. Custom Datasets

Developers can implement custom dataset classes using the base contract.

6.1 Example

import numpy as np
import everyo as eo

class CustomDataset(eo.Dataset):

    def __init__(self):
        self.features = np.array([
            [1.0, 2.0],
            [3.0, 4.0]
        ], dtype=np.float32)

        self.targets = np.array(
            [0, 1],
            dtype=np.int64
        )

    def __len__(self):
        return len(self.features)

    def __getitem__(self, index):
        return (
            self.features[index],
            self.targets[index]
        )

dataset = CustomDataset()

print(len(dataset))

6.2 Design Recommendations

• Return consistent sample structures.
• Validate indices where appropriate.
• Avoid unnecessary data copying.
• Document target encoding.
• Ensure the loader can stack samples.

---

7. TransformDataset

TransformDataset applies transformations to dataset samples.

7.1 Constructor

TransformDataset(
    dataset,
    transform=None,
    target_transform=None
)

7.2 Example

import numpy as np
import everyo as eo

base_dataset = eo.ArrayDataset(
    np.array([
        [1.0, 2.0],
        [3.0, 4.0]
    ], dtype=np.float32),
    np.array([0, 1])
)

transformed = eo.TransformDataset(
    base_dataset,
    transform=lambda x: x * 2.0
)

print(transformed[0])

The first feature sample becomes:

[2. 4.]

7.3 Applications

• Feature transformations.
• Data augmentation.
• Input normalization using fitted statistics.
• Target transformations.

7.4 Important Consideration

Transformations should not unintentionally change sample shapes in ways that prevent batching.

---

8. Subset

Subset creates an indexed view of another dataset.

8.1 Constructor

Subset(dataset, indices)

8.2 Example

import everyo as eo

dataset = eo.ArrayDataset(
    [[1.0], [2.0], [3.0], [4.0]],
    [0, 1, 0, 1]
)

subset = eo.Subset(
    dataset,
    indices=[0, 2]
)

print("Subset size:", len(subset))

Expected output:

Subset size: 2

8.3 Validation

Subset indices must refer to valid samples in the underlying dataset.

8.4 Applications

• Training subsets.
• Validation subsets.
• Controlled experiments.
• Debugging smaller datasets.

---

9. DataLoader

The DataLoader class provides mini-batch iteration.

9.1 Verified Constructor

DataLoader(
    dataset,
    batch_size=32,
    *,
    shuffle=False,
    drop_last=False,
    seed=None
)

9.2 Parameters

Parameter	Description
dataset	Dataset to iterate
batch_size	Samples per batch
shuffle	Shuffle each epoch
drop_last	Drop incomplete final batch
seed	Optional shuffling seed

9.3 Example

import numpy as np
import everyo as eo

features = np.arange(
    40,
    dtype=np.float32
).reshape(20, 2)

targets = np.arange(20)

dataset = eo.ArrayDataset(
    features,
    targets
)

loader = eo.DataLoader(
    dataset,
    batch_size=5,
    shuffle=True,
    seed=42
)

for batch_x, batch_y in loader:
    print(batch_x.shape, batch_y.shape)

Each full batch has shapes:

(5, 2) (5,)

9.4 Loader Properties

print(len(loader))
print(loader.num_samples)

For 20 samples with batch size 5:

4
20

---

10. Batching and Shuffling

10.1 Number of Batches

When incomplete batches are retained:

[ N_{batches}= \left\lceil \frac{N}{B} \right\rceil ]

When incomplete batches are dropped:

[ N_{batches}= \left\lfloor \frac{N}{B} \right\rfloor ]

where:

• (N) = number of samples.
• (B) = batch size.

10.2 Worked Example

For 103 samples and batch size 16:

Without dropping the final batch:

[ \lceil103/16\rceil=7 ]

With drop_last=True:

[ \lfloor103/16\rfloor=6 ]

10.3 Shuffling

loader = eo.DataLoader(
    dataset,
    batch_size=16,
    shuffle=True,
    seed=42
)

The inspected implementation uses a NumPy random generator and reshuffles the order at the start of each iteration when shuffle=True.

10.4 Important Considerations

• Shuffle training data when appropriate.
• Avoid inappropriate shuffling of temporal sequences.
• Preserve feature-target alignment.
• Record seeds for reproducibility.

---

11. CSV Dataset Loading

EveryO includes a numeric CSV reader.

11.1 Verified Signature

load_csv_dataset(
    path,
    *,
    target_column=-1,
    delimiter=",",
    has_header=None,
    skip_columns=()
)

11.2 Parameters

Parameter	Description
path	CSV file location
target_column	Target index, header name, or None
delimiter	Column separator
has_header	Header handling
skip_columns	Columns to exclude by index

11.3 Example CSV

temperature,humidity,pressure,target
28.5,70,1012,1
29.0,68,1010,0
27.8,75,1013,1

11.4 Load CSV

import everyo as eo

dataset = eo.load_csv_dataset(
    "data.csv",
    target_column="target",
    has_header=True
)

print(len(dataset))
print(dataset.feature_shape)

11.5 CSV Limitations

The reader expects numeric data columns.

Non-numeric categories must be encoded before loading.

Complex CSV files containing quoted delimiters or multiline fields may require a more capable CSV preprocessing step.

---

12. Data Splitting

Splitting separates model fitting from evaluation.

12.1 Available Functions

• train_test_split
• random_split
• stratified_split

12.2 Common Split Ratios

Dataset	Example Share
Training	70%
Validation	15%
Testing	15%

These are illustrative proportions, not mandatory defaults.

12.3 Split Selection

Use random splitting when samples are sufficiently independent.

Use stratification when class proportions should be preserved.

Use chronological splitting for time-dependent prediction tasks.

Use group-aware splitting when related samples must not appear in multiple partitions.

12.4 Verify Function Signatures

import inspect
import everyo as eo

for name in [
    "train_test_split",
    "random_split",
    "stratified_split"
]:
    print(
        name,
        inspect.signature(getattr(eo, name))
    )

---

13. StandardScaler

StandardScaler standardizes features using training-data statistics.

13.1 Formula

[ z=\frac{x-\mu}{\sigma} ]

where:

• (x) = input value.
• (\mu) = feature mean.
• (\sigma) = feature standard deviation.

The implementation uses a small positive denominator floor for numerical stability.

13.2 Example

import numpy as np
import everyo as eo

train_x = np.array([
    [10.0, 100.0],
    [20.0, 200.0],
    [30.0, 300.0]
], dtype=np.float32)

scaler = eo.StandardScaler()

scaled_train = scaler.fit_transform(
    train_x
)

print(scaled_train)

13.3 Transform New Data

new_x = np.array([
    [25.0, 250.0]
], dtype=np.float32)

scaled_new = scaler.transform(new_x)

13.4 Available Methods

Method	Purpose
fit()	Learn feature statistics
transform()	Apply scaling
fit_transform()	Fit and transform
inverse_transform()	Reverse scaling

13.5 Important Rule

Fit the scaler only on training data.

Use the fitted scaler to transform validation, test, and inference data.

---

14. MinMaxScaler

MinMaxScaler maps features into a configured numerical range.

14.1 Formula

For the default range [0, 1]:

[ x'=\frac{x-x_{min}}{x_{max}-x_{min}} ]

The implementation protects against zero feature spans.

14.2 Constructor

MinMaxScaler(
    feature_range=(0.0, 1.0)
)

14.3 Example

import numpy as np
import everyo as eo

features = np.array([
    [10.0],
    [20.0],
    [30.0]
], dtype=np.float32)

scaler = eo.MinMaxScaler()

scaled = scaler.fit_transform(features)

print(scaled)

Expected values:

[[0. ]
 [0.5]
 [1. ]]

14.4 Custom Range

scaler = eo.MinMaxScaler(
    feature_range=(-1.0, 1.0)
)

14.5 Important Detail

New values outside the fitted training range can transform to values outside the configured range.

The inspected transform() implementation does not automatically clip such results.

---

15. Normalization and Encoding

EveryO provides additional preprocessing functions.

15.1 Normalization

import numpy as np
import everyo as eo

features = np.array([
    [3.0, 4.0],
    [5.0, 12.0]
])

normalized = eo.normalize(features)

print(normalized)

The default normalization uses the L2 norm along axis 1.

For vector (x):

[ x'=\frac{x}{|x|_2} ]

with a denominator safeguard for zero norms.

15.2 One-Hot Encoding

import everyo as eo

labels = [0, 2, 1]

encoded = eo.one_hot_encode(
    labels,
    num_classes=3
)

print(encoded)

Expected output:

[[1. 0. 0.]
 [0. 0. 1.]
 [0. 1. 0.]]

15.3 Standardization Function

scaled = eo.standardize(features)

15.4 Min-Max Scaling Function

scaled = eo.min_max_scale(features)

15.5 Shuffle Multiple Arrays

shuffled_x, shuffled_y = eo.shuffle_arrays(
    features,
    labels,
    seed=42
)

Shared shuffling preserves sample alignment.

---

16. Data Quality and Validation

Machine learning pipelines should validate data before training.

16.1 Recommended Checks

• Sample count.
• Feature count.
• Target alignment.
• Missing values.
• Infinite values.
• Duplicate records.
• Invalid labels.
• Outliers.
• Data type compatibility.
• Class distribution.

16.2 Example Validation

import numpy as np

def validate_features(features):
    array = np.asarray(features)

    if array.ndim != 2:
        raise ValueError(
            "Expected a 2D feature matrix."
        )

    if not np.isfinite(array).all():
        raise ValueError(
            "Features contain NaN or infinity."
        )

    return array

validated = validate_features(train_x)

16.3 Data Quality Checklist

• Feature dimensions verified.
• Targets aligned.
• Missing values handled.
• Invalid numeric values removed or corrected.
• Class labels validated.
• Data split completed.
• Training-only preprocessing fitted.
• Validation data kept separate.

---

17. Preventing Data Leakage

Data leakage occurs when information unavailable during real prediction influences training.

17.1 Common Leakage Sources

• Fitting scalers on the entire dataset.
• Duplicate samples across partitions.
• Using future information.
• Encoding target-derived information into features.
• Selecting features using test-set results.

17.2 Incorrect Workflow

Entire Dataset
      |
      v
Fit StandardScaler
      |
      v
Split Data

This allows validation/test statistics to influence training preprocessing.

17.3 Correct Workflow

Entire Dataset
      |
      v
Split Data
      |
      +----------------+
      |                |
      v                v
Training Data     Validation Data
      |                |
      v                |
Fit StandardScaler     |
      |                |
      v                v
Transform Train   Transform Validation
      |                |
      +----------------+
              |
              v
          Model Training

17.4 Recommended Practice

Keep all preprocessing statistics derived exclusively from the training partition unless the methodology explicitly requires otherwise.

---

18. Complete Data Pipeline Example

This example combines data generation, splitting, preprocessing, dataset construction, and batching.

18.1 Full Example

import numpy as np
import everyo as eo

# ---------------------------------
# 1. Generate synthetic data
# ---------------------------------

rng = np.random.default_rng(42)

features = rng.normal(
    size=(200, 4)
).astype(np.float32)

targets = (
    features[:, 0] + features[:, 1] > 0
).astype(np.int64)

# ---------------------------------
# 2. Shuffle consistently
# ---------------------------------

order = rng.permutation(len(features))

features = features[order]
targets = targets[order]

# ---------------------------------
# 3. Split data
# ---------------------------------

train_x = features[:160]
train_y = targets[:160]

val_x = features[160:]
val_y = targets[160:]

# ---------------------------------
# 4. Fit preprocessing
# ---------------------------------

scaler = eo.StandardScaler()

train_x = scaler.fit_transform(train_x)
val_x = scaler.transform(val_x)

# ---------------------------------
# 5. Create datasets
# ---------------------------------

train_dataset = eo.ArrayDataset(
    train_x,
    train_y
)

val_dataset = eo.ArrayDataset(
    val_x,
    val_y
)

# ---------------------------------
# 6. Create data loaders
# ---------------------------------

train_loader = eo.DataLoader(
    train_dataset,
    batch_size=16,
    shuffle=True,
    seed=42
)

val_loader = eo.DataLoader(
    val_dataset,
    batch_size=16,
    shuffle=False
)

# ---------------------------------
# 7. Validate output
# ---------------------------------

print("Training samples:", len(train_dataset))
print("Validation samples:", len(val_dataset))
print("Training batches:", len(train_loader))
print("Validation batches:", len(val_loader))

for batch_x, batch_y in train_loader:
    print("Batch X:", batch_x.shape)
    print("Batch Y:", batch_y.shape)
    break

18.2 Expected Shapes

Output	Expected
Training samples	160
Validation samples	40
Training batches	10
Validation batches	3
First training feature batch	(16, 4)
First training target batch	(16,)

18.3 Pipeline Summary

Synthetic Data
     |
     v
Consistent Shuffle
     |
     v
Training / Validation Split
     |
     v
Training-Only StandardScaler Fit
     |
     v
ArrayDataset
     |
     v
DataLoader
     |
     v
Training-Ready Batches

---

19. Training Integration

DataLoaders integrate with EveryO's Trainer.

19.1 Example

import everyo as eo

model = eo.Sequential(
    eo.Linear(4, 8, seed=0),
    eo.ReLU(),
    eo.Linear(8, 2, seed=1)
)

optimizer = eo.Adam(
    model.parameters(),
    lr=0.01
)

trainer = eo.Trainer(
    model,
    optimizer,
    eo.CrossEntropyLoss(),
    metrics=["accuracy"]
)

history = trainer.fit(
    train_loader,
    epochs=20,
    validation_loader=val_loader,
    verbose=False
)

print(history.last())

19.2 Training Data Contract

For supervised training, each batch should provide:

(features, targets)

19.3 Recommended Validation

Confirm that:

• Input shapes match the model.
• Target encoding matches the loss.
• Features are finite.
• Training and validation partitions are separate.
• Batch sizes are appropriate.

Related: Model Training

---

20. Performance and Memory

Data pipelines affect training throughput and memory consumption.

20.1 Performance Factors

• Dataset size.
• Batch size.
• Array layout.
• Preprocessing cost.
• Data copying.
• Transformation complexity.
• Memory capacity.

20.2 NumPy Fast Path

The inspected DataLoader implementation uses a direct array-indexing path for ArrayDataset.

For other dataset types, it uses sample-by-sample collection and default_collate.

20.3 Memory Estimation

For dense features:

[ M=N\times F\times B ]

where:

• (N) = samples.
• (F) = features per sample.
• (B) = bytes per element.

For 100,000 samples with 20 float32 features:

[ M=100000\times20\times4 ]

[ M=8{,}000{,}000\text{ bytes} ]

Approximately 7.63 MiB for the raw feature array, excluding targets and processing overhead.

20.4 Optimization Recommendations

• Avoid unnecessary copies.
• Prefer appropriate numerical dtypes.
• Profile preprocessing.
• Select batch size experimentally.
• Validate memory usage.
• Avoid expensive transformations inside repeated sample access when they can be precomputed safely.

---

21. Testing and Reproducibility

21.1 Run Repository Tests

python -m pytest

21.2 Recommended Pipeline Tests

• Dataset length.
• Feature-target alignment.
• Index validation.
• Batch count.
• Shuffling behavior.
• Scaling correctness.
• Missing-value detection.
• CSV parsing.
• Reproducible random seeds.
• Data leakage prevention.

21.3 Example Assertions

assert len(train_dataset) == 160
assert len(val_dataset) == 40

assert train_dataset.feature_shape == (4,)

assert len(train_loader) == 10
assert len(val_loader) == 3

21.4 Reproducibility

Record:

• Dataset version.
• Split strategy.
• Random seed.
• Preprocessing configuration.
• EveryO version.
• Python version.
• Dependency versions.

---

22. Troubleshooting

Feature and target counts differ

Confirm that both arrays contain the same number of samples in their first dimension.

StandardScaler raises an error

Call fit() or fit_transform() on training data before transform().

Confirm feature dimensions match those used during fitting.

CSV loading fails

Verify the file path, delimiter, header configuration, target column, and numeric values.

Encode categorical columns before loading.

DataLoader returns unexpected batch shapes

Inspect the shape of individual dataset samples and verify that they can be stacked consistently.

Training and validation results differ significantly

Investigate preprocessing consistency, class distributions, leakage, overfitting, and dataset shift.

DataLoader produces fewer batches than expected

Check batch_size and drop_last.

---

23. Developer References

Resource	Link
EveryO Repository	GitHub
Data Package	everyo/data/
Dataset Implementation	dataset.py
DataLoader Implementation	dataloader.py
Preprocessing Implementation	preprocessing.py
Data Splitting	split.py
Training Engine	trainer.py
Tests	tests/
Issues	GitHub Issues

---

24. Next Steps

Continue with the following EveryO Wiki pages.

Page	Focus
Model Serialization	Saving, loading, and checkpoints
Performance Optimization	Profiling and benchmarking
Model Training	Training pipelines
Examples and Tutorials	Practical development
API Reference	Public interfaces

Wiki links become active after their corresponding pages are created.

---

Data Pipeline — Documentation Prepared

Next Recommended Page: Model Serialization →

Home · Model Training · Examples and Tutorials · API Reference

---

EveryO — Open Machine Learning. Built for Developers.

Maintained By & Owner:
EveryO Contributors & @krishanth7

Open Source · MIT Licensed · Community Driven

© 2026 EveryO Contributors