Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

BitNet: Learning-Based-Bit-Depth-Expansion

Running Microsoft's BitNet inference framework via FastAPI, Uvicorn and Docker.

Distily: Language Model Distillation Toolkit and Library

'The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits' toy example

Long term project about a custom AI architecture. Consist of cutting-edge technique in machine learning such as Flash-Attention, Group-Query-Attention, ZeRO-Infinity, BitNet, etc.

This is the repo for the MixKABRN Neural Network (Mixture of Kolmogorov-Arnold Bit Retentive Networks), and an attempt at first adapting it for training on text, and later adjust it for other modalities.