High-efficiency floating-point neural network inference operators for mobile, server, and Web

BladeDISC is an end-to-end DynamIc Shape Compiler project for machine learning workloads.

The Tensor Algebra SuperOptimizer for Deep Learning

Batch normalization fusion for PyTorch

[MLSys 2021] IOS: Inter-Operator Scheduler for CNN Acceleration

Optimize layers structure of Keras model to reduce computation time

A set of tool which would make your life easier with Tensorrt and Onnxruntime. This Repo is designed for YoloV3

cross-platform modular neural network inference library, small and efficient

The blog, read report and code example for AGI/LLM related knowledge.

A constrained expectation-maximization algorithm for feasible graph inference.

Modified inference engine for quantized convolution using product quantization

[WIP] A template for getting started writing code using GGML

Batch Partitioning for Multi-PE Inference with TVM (2020)

🤖️ Optimized CUDA Kernels for Fast MobileNetV2 Inference

MLP-Rank: A graph theoretical approach to structured pruning of deep neural networks based on weighted Page Rank centrality as introduced by the related thesis.

Learn the ins and outs of efficiently serving Large Language Models (LLMs). Dive into optimization techniques, including KV caching and Low Rank Adapters (LoRA), and gain hands-on experience with Predibase’s LoRAX framework inference server.

MIVisionX Python Inference Analyzer uses pre-trained ONNX/NNEF/Caffe models to analyze inference results and summarize individual image results

PyTorch Mobile: iOS examples

Batch estimation on Lie groups

Interface for TensorRT engines inference along with an example of YOLOv4 engine being used.