Cpp Implementation

Documentation of the functionality of each section of the native C++ code. Refer to the source code for the exact implementations of the systems described here.

Tensors

The Tensor is the only data structure implemented in the native C++ code. This class serves as the fundamental mathematical unit of data used throughout the framework and implements the complete automatic differentiation functionality. Tensor objects use a linear vector representation of their underlying data with properties defining the abstract shape. Each tensor vaguely resembles a node in a double-linked list, storing references to the tensors directly involved in its creation as well as any tensors which it was used to create. Smart pointers are used throughout the Tensor class whenever possible to improve memory safety. Most mathematical operations relevant to tensors are implemented in the Tensor class and are optimized for performance at large scale. The Tensor class is also the only data structure directly exposed to the managed C# code through the C# / C++ interop layer.

Autodiff Engine

The framework uses an autograd-style engine similar to that used by PyTorch to handle automatic differentiation. All autograd functionality is implemented in the native C++ code via the Tensor class. When tensor operations are performed with the 'Inference' flag set to false, an autograd graph is automatically generated to enable subsequent gradient calculation. Each tensor object stores the tensors involved in the operation which created it, as well as any 'result' tensors it was used to create. Additionally, every tensor instance also stores the gradient calculation function corresponding to the function which created the instance. Automatic differentiation is handled by first generating the topography of the graph and then walking backwards through the topography, using each tensor's gradient calculation function to accumulate gradients into each involved tensor's dedicated gradient vector. To reduce allocations when repeatedly training on batches with identical dimensions, the each operation used during the forward pass will attempt to write its result to an existing tensor allocation, via the 'result' tensors referenced by the involved tensors, whenever possible.

Other Neural Network Functions

Outside of core tensor operations, the native C++ code also implements the mathematical functions behind every cost function and optimizer step, as well as any other miscellaneous functions involved in neural network training. Such functions are implemented via corresponding static classes and exported by the native C++ DLL for use by the managed C# code. Although adding complexity to the framework, this arrangement allows frequently used and high-volume functions, notably optimizer steps which operate on every value of every parameter tensor contained in a neural network model, to utilize the same performance optimizations as the core tensor operation implementations.

SIMD Vectorization

The native C++ code utilizes AVX2 SIMD vectorization for every tensor operation, as well as any other function which operates on a large volume of floating point numbers. Most fundamental mathematical and logical functions have vectorized forms defined within the static MathUtils class found in the native C++ code. All vectorized functions only support processing of vectors of 'float' values and require data to be 32-bit aligned. SIMD operations are implemented via intrinsic functions provided by the immintrin compiler header file.

Parallelization

The native C++ code utilizes CPU parallelization in addition to SIMD vectorization for certain computationally intensive tensor operations. Notably, both the matrix multiplication and convolution functions parallelize their outer-most loops while retaining SIMD vectorization for calculating individual dot products. Parallelization is only enabled when the size of a given operation is larger than the predefined vectorization threshold, with element-wise tensor operations never being parallelized. CPU parallelization is implemented through the Open Multi-Processing (OpenMP) API.

Compiler Settings

The framework is compiled using the Microsoft Visual C++ (MSVC) compiler.
Key compiler settings include the following:

Setting	Selected Option
C++ Language Standard	ISO C++20 Standard
Whole Program Optimization	Use Link Time Code Generation
Optimization	Maximum Optimization (Favor Speed) (/O2)
Enable Intrinsic Functions	Yes (/Oi)
Favor Size Or Speed	Favor fast code (/Ot)
Whole Program Optimization	Yes (/GL)
Runtime Library	Multi-threaded (MT)
Enable Enhanced Instruction Set	Advanced Vector Extensions 2 (X86/X64) (/arch:AVX2)
Floating Point Model	Fast (/fp:fast)
Open MP Support	/openmp:llvm (forced via additional options)
Conformance Mode	Yes (/permissive-)
Precompiled Header	Use (/Yu)
Precompiled Header File	pch.h
Calling Convention	__cdecl (/Gd)