A compact, AI-optimized data transformation language for LLMs
~33% token reduction vs Python — validated with fine-tuned models on consumer GPUs
git clone https://github.com/pinku/Flow.git && cd Flow
python3 flow.py "1..10 | f:_>5 | m:_*2"
# → [12, 14, 16, 18, 20]No dependencies. No setup. Just Python 3.8+.
LLMs are trained on verbose human-oriented languages, but that verbosity burns tokens without adding value for AI processing:
| Problem | Impact |
|---|---|
Keywords like function, const, return |
Every one costs tokens |
| Verbose variable names for human readability | No benefit for LLMs |
| Syntactic sugar optimized for humans | Wasted context window |
Flow strips away the human-oriented boilerplate, keeping only what the LLM needs to express the transformation. The result: 33% fewer tokens on average, tested and verified with real models.
| Python (24 tokens) | Flow (13 tokens) — 45% fewer |
|---|---|
result = [x * 2 for x in range(1, 11) if x > 5] |
1..10 | f:_>5 | m:_*2 |
More examples in the Operator Reference.
| Feature | Description |
|---|---|
| Minimal Syntax | Single-character operators: f: (filter), m: (map), g: (group), a: (aggregate), etc. |
| Python Compatible | Transpiles to clean, readable Python |
| LLM Optimized | Designed for minimal token usage without sacrificing expressiveness |
| JSON Native | Built-in JSON parsing and serialization |
| Batch Processing | Windowing and batching for large datasets |
| Type Safety | Type casting and validation |
| Zero Dependencies | Pure Python — no pip install required |
git clone https://github.com/pinku/Flow.git
cd Flow
python3 flow.py --help# Filter and map
python3 flow.py "1..10 | f:_>5 | m:_*2"
# Group and aggregate
python3 flow.py "[{name:'Alice',age:30},{name:'Bob',age:25}] | g:_.age>=30 | a:len(v)"
# JSON processing
python3 flow.py "S:data.json" # Save to JSON
python3 flow.py "J" # Parse JSON inputpython3 flow.py
# > 1..10 | f:_>5 | m:_*2
# [12, 14, 16, 18, 20]from flow import flow_run, flow_transpile
# Execute directly
result = flow_run("1..10 | f:_>5 | m:_*2")
print(result) # [12, 14, 16, 18, 20]
# Get transpiled Python
code = flow_transpile("1..10 | f:_>5 | m:_*2")
print(code) # [_*2 for _ in range(1, 11) if _>5]| Operator | Name | Example |
|---|---|---|
f: |
Filter | data | f:_.age > 18 |
m: |
Map | data | m:_.name.upper() |
g: |
Group | data | g:_.category |
a: |
Aggregate | data | g:_.cat | a:sum(v) |
s: |
Sort | data | s:-_.age |
t: |
Take | data | t:10 |
d: |
Drop | data | d:2 |
u |
Unique | data | u |
r |
Reverse | data | r |
b: |
Batch | data | b:32 |
w: |
Window | data | w:3 |
i: |
Split | data | i:'\n' |
o: |
Join | data | o:',' |
T: |
Type cast | data | T:int |
J |
Parse JSON | data | J |
S: |
Save JSON | data | S:output.json |
x |
Flatten | nested | x |
& |
AND filter | data | f:_.a>0 & _.b<10 |
| |
OR filter | data | f:_.a>0 | _.b>10 |
Full interactive report: benchmark_report.html
| Task | Python Tokens | Flow Tokens | Savings |
|---|---|---|---|
| Filter + Map | 22 | 13 | 41% |
| Group + Aggregate | 25 | 14 | 44% |
| Sort + Take | 20 | 14 | 30% |
| JSON Parse | 15 | 4 | 73% |
| Batch Processing | 35 | 8 | 77% |
Average token saving: 33%
Flow has been validated with fine-tuned models on consumer hardware:
| Model | Parameters | VRAM | Training Time | Accuracy |
|---|---|---|---|---|
| phi-2 | 2.7B | ~5.4 GB | ~20 min | 100% |
| Qwen 3B Coder v2 | 3.7B | ~6.0 GB | ~5 min | 100% |
Both models were fine-tuned using QLoRA (4-bit quantization) on an RTX 4060.
Flow includes training scripts for fine-tuning small language models:
# Install dependencies
pip install transformers peft bitsandbytes trl
# Generate training dataset
python3 generate_dataset.py
# Train phi-2 (2.7B)
python3 train_qlora.py
# Train Qwen 3B Coder
python3 train_qwen_flow.pyRequirements:
- NVIDIA GPU with 8GB+ VRAM (RTX 4060 tested)
- CUDA 12.x
- Python 3.8+
Flow includes a Hermes skill for seamless integration:
# Add to your Hermes skills
cp -r .hermes/skills/flow-tool ~/.hermes/skills/
# Use in chat
/hermes run flow-tool "1..10 | f:_>5 | m:_*2"Flow/
├── flow.py # Main implementation (transpiler + runtime)
├── data/
│ └── flow_dataset.json # Training dataset (600 examples)
├── models/ # Fine-tuned LoRA adapters (not in repo)
│ ├── flow-lora/ # phi-2 adapter
│ └── qwen3b-flow-lora-v2/ # Qwen 3B adapter
├── benchmark_report.html # Visual benchmark report
├── train_qlora.py # phi-2 training script
├── train_qwen_flow.py # Qwen 3B training script
└── generate_dataset.py # Dataset generation script
- Hermes Agent — LLM agent platform; Flow is used as a tool skill
- llama.cpp — Local LLM inference; models trained on Flow data run here
MIT License — free for any use.
This project explores the hypothesis that purpose-built languages for LLMs can achieve significant efficiency gains compared to traditional programming languages. The key findings:
- Token efficiency is real: Flow achieves 33% token reduction on average
- Small models can learn: 2.7-3.7B parameter models achieve 100% accuracy after fine-tuning
- Hardware accessible: QLoRA enables training on consumer GPUs (RTX 4060)
For the full analysis and methodology, see benchmark_report.html.