Repository navigation
Custom Modules and Layers
Design · Implement · Differentiate · Validate · Reuse
An engineering guide to building reusable neural network components using EveryO's module system and automatic differentiation engine.
Home · Neural Networks · Automatic Differentiation · API Reference
Repository: krishanth7/EveryO
Maintainer: @krishanth7
Audience: Framework contributors, ML developers, researchers
Document Type: Extensibility and implementation guidance
Important
Compatibility Notice
EveryO is an experimental NumPy-based neural computing framework. Custom modules should use the public APIs supported by the installed release.
The examples below demonstrate conventional module composition using documented EveryO abstractions. More specialized capabilities, including module registration and serialization, must be checked against the current implementation.
A custom module is not automatically serializable, ONNX-exportable, CUDA-compatible, or supported by every optimizer and training workflow.
- Introduction
- Module Architecture
- Prerequisites
- Creating a Custom Module
- Trainable Parameters
- Parameter Registration
- Forward Computation
- Automatic Differentiation
- Custom Linear Layers
- Activation Modules
- Residual Connections
- Sequential Composition
- Normalization Layers
- Dropout and Regularization
- Custom Loss Functions
- Shape Validation
- Gradient Verification
- Training Custom Modules
- Serialization
- ONNX Compatibility
- CUDA Considerations
- Performance Engineering
- Testing Strategy
- Debugging
- Security Considerations
- Documentation Standards
- Contribution Workflow
- Implementation Checklist
- Advanced Exercises
- Official Resources
EveryO provides a module system for constructing neural networks from reusable components.
Custom modules allow developers to implement architectures beyond predefined layers while preserving the framework's tensor and differentiation abstractions.
A well-designed custom module should provide:
- Clear mathematical behavior.
- Predictable input and output shapes.
- Proper parameter management.
- Differentiable forward computation where applicable.
- Reliable tests.
- Understandable documentation.
- Explicit compatibility limitations.
DEFINE MATHEMATICS
|
v
DESIGN MODULE
|
v
REGISTER PARAMETERS
|
v
IMPLEMENT FORWARD
|
v
VALIDATE SHAPES
|
v
CHECK GRADIENTS
|
v
INTEGRATE MODEL
|
v
RUN UNIT TESTS
|
v
DOCUMENT LIMITATIONS
EveryO provides abstractions including:
ModuleParameterSequentialLinearReLU
These form the foundation of reusable neural network components.
A module typically manages:
- Trainable parameters.
- Child modules.
- Forward computation.
- Training and evaluation modes.
- Parameter discovery.
MODULE
|
+--------+--------+
| | |
v v v
PARAMETERS CHILD FORWARD
MODULES LOGIC
| | |
+--------+--------+
|
v
OUTPUT
|
v
AUTOGRAD
git clone https://github.com/krishanth7/EveryO.git
cd EveryO
python -m pip install -e ".[dev]"everyo info
everyo doctorReaders should understand:
- Python classes.
- NumPy.
- Matrix multiplication.
- Gradient descent.
- Tensor shapes.
- Automatic differentiation.
A custom module can combine existing EveryO layers.
import everyo as eo
class TwoLayerNetwork(eo.Module):
def __init__(self, input_size, hidden_size, output_size):
super().__init__()
self.fc1 = eo.Linear(input_size, hidden_size)
self.activation = eo.ReLU()
self.fc2 = eo.Linear(hidden_size, output_size)
def forward(self, x):
x = self.fc1(x)
x = self.activation(x)
return self.fc2(x)
model = TwoLayerNetwork(
input_size=4,
hidden_size=16,
output_size=2,
)
x = eo.tensor([
[1.0, 2.0, 3.0, 4.0],
])
output = model(x)
print(output.shape)Expected shape:
(1, 2)
- Call the parent constructor.
- Store child modules as instance attributes.
- Keep forward logic understandable.
- Document expected shapes.
- Avoid unnecessary state mutation.
A trainable parameter is a tensor whose values are updated during optimization.
For a linear transformation:
[ Y=XW+b ]
where:
- (X): Input tensor.
- (W): Trainable weight matrix.
- (b): Trainable bias.
- (Y): Output tensor.
A trainable parameter should:
- Participate in gradient computation.
- Be discoverable by the optimizer.
- Have a documented shape.
- Be initialized appropriately.
Prefer framework-provided parameter abstractions over manually managing unrelated NumPy arrays.
Parameter registration enables a model to expose trainable values to optimizers.
parameters = list(model.parameters())
print("Parameter tensors:", len(parameters))Unregistered parameters may not be updated during training.
For a custom module:
- Create the module.
- Inspect its parameters.
- Verify expected parameter shapes.
- Run a forward pass.
- Run backward propagation.
- Confirm gradients are available.
When composing modules, verify that child-module parameters are included in the parent module's parameter collection.
The forward method defines how input data is transformed.
def forward(self, x):
x = self.fc1(x)
x = self.activation(x)
return self.fc2(x)- Use documented tensor operations.
- Preserve gradient tracking.
- Avoid unexpected input mutation.
- Maintain consistent shape contracts.
- Keep forward computation deterministic when appropriate.
A layer should not unexpectedly flatten, transpose, or broadcast data without documenting that behavior.
EveryO's autograd engine tracks supported differentiable tensor operations.
import everyo as eo
x = eo.tensor(
[3.0],
requires_grad=True,
)
y = x * x
y.backward()
print(x.grad)Expected gradient:
[6.]
\frac{dy}{du} \frac{du}{dx} ]
A custom module built entirely from supported differentiable EveryO operations can use the existing autograd engine.
A new low-level mathematical operation is not automatically differentiable merely because it is called inside a module.
It may require additional autograd implementation work.
A linear layer computes:
[ Y=XW+b ]
For:
[ X\in\mathbb{R}^{B\times I} ]
and:
[ W\in\mathbb{R}^{I\times O} ]
the output is:
[ Y\in\mathbb{R}^{B\times O} ]
where:
- (B): Batch size.
- (I): Input features.
- (O): Output features.
EveryO already provides Linear.
import everyo as eo
layer = eo.Linear(8, 16)When extending linear behavior:
- Define the mathematical change.
- Decide whether composition is sufficient.
- Validate the output shape.
- Check gradients.
- Compare against a reference.
Use built-in layers when they already provide the required behavior.
Activation functions introduce nonlinear transformations.
[ ReLU(x)=\max(0,x) ]
[ \sigma(x)=\frac{1}{1+e^{-x}} ]
[ \tanh(x) ]
import everyo as eo
model = eo.Sequential(
eo.Linear(8, 16),
eo.ReLU(),
eo.Linear(16, 4),
)For a new activation:
- Define its mathematical formula.
- Define behavior at nondifferentiable points.
- Implement the forward computation.
- Validate gradients.
- Test extreme input magnitudes.
Residual connections allow a module to add its input to a transformed representation.
[ y=x+F(x) ]
where (F(x)) is a learned transformation.
import everyo as eo
class ResidualBlock(eo.Module):
def __init__(self, features):
super().__init__()
self.fc1 = eo.Linear(features, features)
self.activation = eo.ReLU()
self.fc2 = eo.Linear(features, features)
def forward(self, x):
residual = x
y = self.fc1(x)
y = self.activation(y)
y = self.fc2(y)
return residual + y
block = ResidualBlock(features=8)
x = eo.tensor([
[1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0],
])
output = block(x)
print(output.shape)Expected shape:
(1, 8)
The residual and transformed tensors must have compatible shapes.
Verify gradients through both the residual path and the transformation path.
Sequential combines modules into an ordered computation pipeline.
import everyo as eo
model = eo.Sequential(
eo.Linear(16, 32),
eo.ReLU(),
eo.Linear(32, 16),
eo.ReLU(),
eo.Linear(16, 4),
)INPUT (16)
|
v
LINEAR (32)
|
v
RELU
|
v
LINEAR (16)
|
v
RELU
|
v
LINEAR (4)
|
v
OUTPUT
Use it for architectures that naturally form a simple linear chain.
Use a custom module when the architecture requires:
- Multiple branches.
- Residual connections.
- Shared parameters.
- Conditional behavior.
- Multiple intermediate outputs.
EveryO includes normalization components such as:
BatchNorm1DBatchNorm2DLayerNorm
[ \hat{x}= \frac{x-\mu}{\sqrt{\sigma^2+\epsilon}} ]
Normalization behavior may depend on:
- Training mode.
- Evaluation mode.
- Batch statistics.
- Running statistics.
- Normalization dimensions.
Test:
- Forward values.
- Gradient correctness.
- Shape handling.
- Near-zero variance.
- Training/evaluation differences.
Dropout is a regularization technique that randomly disables selected activations during training.
Reduce dependence on individual activations and improve generalization in suitable models.
import everyo as eo
model = eo.Sequential(
eo.Linear(16, 32),
eo.ReLU(),
eo.Dropout(0.2),
eo.Linear(32, 4),
)Verify that dropout behaves appropriately in training and evaluation modes.
Do not assume that adding dropout always improves model quality.
Evaluate its impact using held-out validation data.
Loss functions quantify prediction error.
[ MSE= \frac{1}{N} \sum_{i=1}^{N}(y_i-\hat{y}_i)^2 ]
def custom_mse(prediction, target):
difference = prediction - target
return (difference * difference).mean()This example uses basic differentiable tensor operations.
Verify:
- Output scalar behavior.
- Reduction semantics.
- Gradient correctness.
- Input shape compatibility.
- Numerical stability.
A custom loss should not silently change the expected meaning of targets or predictions.
Shape contracts are essential for reliable custom modules.
def validate_input_shape(x, expected_features):
if x.shape[-1] != expected_features:
raise ValueError(
f"Expected {expected_features} features, "
f"received {x.shape[-1]}"
)- Number of dimensions.
- Feature dimension.
- Batch compatibility.
- Broadcasting assumptions.
- Output dimensions.
Document shapes using consistent notation:
Input: (batch, input_features)
Output: (batch, output_features)
Custom modules should be tested against numerical gradients.
[ g_{\text{numeric}}= \frac{f(x+h)-f(x-h)}{2h} ]
For:
[ f(x)=x^2 ]
[ f'(x)=2x ]
def f(x):
return x * x
x = 3.0
h = 1e-5
numeric = (
f(x + h) - f(x - h)
) / (2 * h)
analytic = 2 * x
print("Numeric:", numeric)
print("Analytic:", analytic)- Compare against an independent reference.
- Test multiple input magnitudes.
- Avoid nondifferentiable points.
- Use justified tolerances.
- Record dtype and test conditions.
Custom modules can be integrated into training workflows when their parameters and operations are compatible.
import everyo as eo
model = TwoLayerNetwork(
input_size=4,
hidden_size=16,
output_size=2,
)
optimizer = eo.Adam(
model.parameters(),
lr=0.001,
)
loss_fn = eo.CrossEntropyLoss()DATA BATCH
|
v
CUSTOM MODULE
|
v
PREDICTION
|
v
LOSS FUNCTION
|
v
BACKWARD PASS
|
v
OPTIMIZER STEP
Before long training runs:
- Confirm the forward pass.
- Confirm loss computation.
- Verify parameter gradients.
- Run a short training test.
- Check whether loss changes as expected.
EveryO supports .evo model archives.
import everyo as eo
eo.save(model, "custom_model.evo")restored = eo.load("custom_model.evo")Custom module types may require registration using EveryO's documented @register_module mechanism.
Do not assume that an arbitrary Python subclass is automatically reconstructible from an .evo archive.
Review:
- Module registration.
- Constructor metadata.
- Supported parameter serialization.
- Restoration behavior.
Compare model outputs before and after serialization.
Treat untrusted archives as untrusted input, even though the documented format avoids pickle execution.
ONNX export is available for supported model structures and operations.
python -m pip install -e ".[onnx]"A custom EveryO module is not automatically ONNX-exportable.
- All operations are supported.
- Input shapes are valid.
- Model is in evaluation mode.
- Export succeeds.
- ONNX Runtime executes the graph.
- Outputs agree within justified tolerances.
Tracing may support additional operation paths, but it captures the executed path and can encode fixed shape or control-flow assumptions.
EveryO includes optional CUDA functionality.
import everyo as eo
print(eo.cuda.is_available())A custom module is not automatically GPU-compatible.
CUDA support depends on the operations and execution path implemented by the framework.
When testing CUDA:
- Verify supported operations.
- Compare against CPU references.
- Record hardware details.
- Measure transfer overhead.
- Test on actual compatible hardware.
Do not assume GPU-resident tensor operations automatically integrate with EveryO's standard autograd graph.
Use profiling to identify actual bottlenecks.
import everyo as eo
with eo.profile() as profiler:
output = model(input_tensor)
print(profiler.summary())The model and input tensor must be initialized beforehand.
Consider:
- Avoiding unnecessary copies.
- Reducing redundant operations.
- Improving memory locality.
- Using suitable tensor shapes.
- Minimizing unnecessary device transfers.
Every optimization should preserve validated mathematical behavior.
Custom modules require multiple levels of testing.
Verify isolated forward computations.
Verify backward computations.
Test valid and invalid dimensions.
Test custom modules inside larger models.
Test save/load behavior when supported.
Test optional backends only where the relevant hardware or dependencies are available.
import everyo as eo
def test_residual_block_shape():
block = ResidualBlock(features=4)
x = eo.tensor([
[1.0, 2.0, 3.0, 4.0],
])
output = block(x)
assert output.shape == (1, 4)python -m pytest -q| Problem | Possible Cause | Recommended Action |
|---|---|---|
| Missing parameters | Registration issue | Inspect parameters()
|
| Shape mismatch | Incorrect dimensions | Validate shape contracts |
| Missing gradients | Unsupported operation | Check autograd path |
| NaN outputs | Numerical instability | Inspect intermediate values |
| Serialization failure | Unregistered module | Review registration |
| ONNX export failure | Unsupported operation | Check exporter support |
| CUDA failure | Unsupported backend path | Verify hardware and operations |
| Poor training | Architecture or optimization issue | Test a small baseline |
REPRODUCE ERROR
|
v
MINIMIZE EXAMPLE
|
v
INSPECT SHAPES
|
v
CHECK PARAMETERS
|
v
VERIFY FORWARD
|
v
VERIFY GRADIENTS
|
v
IMPLEMENT FIX
|
v
ADD REGRESSION TEST
Validate external data and model archives.
Custom C++ or CUDA extensions require additional memory-safety review.
Avoid introducing unnecessary dependencies.
Never commit private credentials or tokens.
Report security vulnerabilities privately:
Each custom module should document:
- Purpose.
- Mathematical definition.
- Constructor parameters.
- Input shapes.
- Output shapes.
- Trainable parameters.
- Example usage.
- Numerical limitations.
- Backend compatibility.
- Serialization compatibility.
# Custom Layer Name
## Purpose
Explain the layer.
## Mathematical Definition
Provide the formula.
## Constructor
Document arguments.
## Input
Shape and dtype.
## Output
Shape and dtype.
## Example
Provide runnable Python.
## Limitations
Document unsupported behavior.
## Tests
Explain validation coverage.Contributors can propose new layers through the repository's normal contribution process.
- Review existing functionality.
- Search for related issues.
- Define the mathematical specification.
- Identify expected API behavior.
- Keep changes focused.
- Add tests.
- Verify gradients.
- Document compatibility.
- Avoid unnecessary dependencies.
Run:
python -m pytest -q
ruff check .
ruff format --check .Use the repository's pull request workflow.
Maintainer approval is required for integration; submitting a PR does not guarantee acceptance.
- Module purpose is clear.
- Mathematical definition is documented.
- Input/output shapes are defined.
- Existing layers are reused where appropriate.
- Parameters are registered.
- Shapes are correct.
- Initialization is appropriate.
- Optimizers can discover trainable parameters.
- Forward operations are differentiable where required.
- Gradients have correct shapes.
- Numerical gradient checks pass.
- Shared computation paths are tested.
- Unit tests pass.
- Invalid inputs are handled.
- Numerical edge cases are tested.
- Training integration is verified.
- Serialization support is evaluated.
- ONNX compatibility is evaluated.
- CUDA limitations are documented.
- Optional dependencies are declared.
- Public API is documented.
- Examples are included.
- Limitations are disclosed.
- Contribution guidance is followed.
Implement and test a residual block.
Deliverable: Forward and gradient tests.
Create a differentiable activation using supported tensor operations.
Deliverable: Mathematical definition and numerical gradient verification.
Implement a regression loss using EveryO tensor operations.
Deliverable: Unit tests and analytical reference comparison.
Create a network with several reusable submodules.
Deliverable: Parameter registration test.
Register a supported custom module and test save/load behavior.
Deliverable: Round-trip verification report.
Evaluate whether a custom architecture can be exported.
Deliverable: Compatibility report with limitations.
Profile a custom module and investigate its most expensive operation.
Deliverable: Reproducible benchmark results.
| Resource | Link |
|---|---|
| EveryO Repository | GitHub |
| README | README.md |
| API Reference | API Reference Wiki |
| Neural Networks | Neural Networks Wiki |
| Automatic Differentiation | Automatic Differentiation Wiki |
| Model Serialization | Model Serialization Wiki |
| Numerical Validation | Numerical Accuracy and Validation |
| Advanced Tutorials | Advanced Tutorials |
| Testing | Testing and Quality Assurance |
| Contributing | CONTRIBUTING.md |
| GitHub Issues | Issues |
| GitHub Discussions | Discussions |
Next Recommended Page: Model Deployment and Inference →
Home · Neural Networks · Advanced Tutorials
EveryO — Open Machine Learning. Built for Developers.
Project Maintainer: @krishanth7
Open Source · MIT Licensed · Community Driven
© 2026 EveryO Contributors