Skip to content

nhtera/graphify-rs

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

50 Commits
docs
docs
 
 
src
src
 
 
tests
tests
 
 
vendor
vendor
 
 
worked
worked
 
 
.gitignore
.gitignore
 
 
ARCHITECTURE.md
ARCHITECTURE.md
 
 
CLAUDE.md
CLAUDE.md
 
 
Cargo.lock
Cargo.lock
 
 
Cargo.toml
Cargo.toml
 
 
README.md
README.md
 
 
SECURITY.md
SECURITY.md
 
 

Repository files navigation

graphify-rs

Blazing-fast knowledge graph builder — Rust port of Graphify, inspired by Andrej Karpathy's LLM Knowledge Base workflow.

Build, query, and visualize knowledge graphs from code, documents, papers, and images. Single static binary, zero Python dependency.

Features

  • Fast & memory-efficient — handles 10,000+ files, targets < 150MB RAM
  • Single binary — zero runtime dependencies
  • 8 CLI commands — build, add, query, report, watch, spec, lint, export
  • Honest AI — every edge tagged EXTRACTED / INFERRED / AMBIGUOUS with confidence score
  • Incremental — only re-processes changed files (mtime + content hash)
  • Parallel — Rayon-powered file processing
  • Multi-format export — JSON, HTML, Obsidian, GraphML, Neo4j, SVG
  • Local-first — privacy-first with local LLM fallback support

How It Works

graphify-rs extracts a knowledge graph in two passes:

  1. AST pass — Tree-sitter parses source code locally (no network). Extracts functions, classes, structs, imports, and // NOTE: rationale comments. Every edge is tagged EXTRACTED with confidence 1.0.

  2. LLM pass — Documents, papers, and images are sent to an LLM API (Anthropic or OpenAI). The LLM returns structured JSON: concept nodes, semantic edges (ConceptuallyRelatedTo, SharesDataWith, RationaleFor), and hyperedges. Every edge is tagged INFERRED (0.4–0.9) or AMBIGUOUS (0.1–0.3).

After extraction, Louvain community detection clusters related nodes, and a report highlights god nodes, surprising connections, and knowledge gaps.

Files → AST parse (local) → LLM extract (optional) → Graph → Cluster → Report

File Types

Category Extensions Extraction Method
Code .rs .py .js .ts .tsx .jsx .go .java .c .cpp .h .rb .cs .kt .scala .php .swift .lua .zig .ex .exs .sh .bash .zsh .ps1 .vue .svelte .m .mm .jl Tree-sitter AST (local)
Documents .md .txt .rst .adoc .org .docx .xlsx LLM semantic extraction
Papers .pdf LLM semantic extraction
Images .png .jpg .jpeg .gif .svg .webp LLM vision extraction
Video/Audio .mp4 .mov .webm .mkv .avi .m4v .mp3 .wav .m4a .ogg .flac Whisper transcription (feature-gated, requires --features video)

19 languages have dedicated tree-sitter parsers. Shell scripts (.sh, .bash, .zsh) and Vue/Svelte (.vue, .svelte) are classified as code but use generic extraction.

Video/audio support is optional and requires the video feature flag: cargo build --features video. Transcriptions are converted to text documents and processed via LLM semantic extraction.

What You Get

After graphify-rs build, the output directory contains:

File Description
graph.json Full knowledge graph — nodes, edges, hyperedges
GRAPH_REPORT.md Human-readable analysis with actionable insights
graph.html Interactive visualization (unless --no-viz)
manifest.json Build metadata — file list, timestamps, hashes

The report includes:

  • God nodes — High-connectivity hubs ranked by edge count × centrality
  • Surprising connections — Cross-community edges the LLM discovered
  • Knowledge gaps — Isolated nodes and thin communities
  • Suggested questions — What to investigate next
  • Hyperedges — Multi-way relationships (e.g., "error isolation is the rationale for validation, enrichment, and pipeline design")

Privacy

  • Code files are parsed locally via tree-sitter — never sent to any API
  • Documents, papers, and images are sent to your configured LLM API for semantic extraction
  • No telemetry — graphify-rs makes zero analytics or tracking calls
  • API keys are read from environment variables, never logged or stored in output
  • To run fully offline, use code-only corpora (no .md, .pdf, or image files) — AST extraction needs no network

Installation

# From source
cargo install --path .

# Or build directly
cargo build --release

Quick Start

# Build a knowledge graph from current directory
graphify-rs build .

# Build with custom output directory
graphify-rs build ./my-project -o ./output

# Show help
graphify-rs --help

Commands

Command Description
build Build knowledge graph from a directory
add Add URL, file, paper, or tweet to the graph
query Ask questions about the knowledge graph
report Generate GRAPH_REPORT.md with god nodes & insights
watch Watch for file changes and rebuild incrementally
spec Ingest graph into a declarative wiki structure
lint Check graph health and find inconsistencies
export Export graph to various formats

Build Command

graphify-rs build [PATH] [OPTIONS]

Options:
  -o, --output <DIR>     Output directory [default: ./graphify-out]
  -j, --jobs <N>         Worker threads (auto-detect if omitted)
      --update           Incremental mode (changed files only)
      --mode <MODE>      Extraction mode: normal | deep [default: normal]
      --no-viz           Skip HTML visualization
  -v, --verbose          Verbose output

Add Command

graphify-rs add <SOURCE> [OPTIONS]

Options:
  -o, --output <DIR>     Output directory [default: ./graphify-out]
      --author <NAME>    Author attribution
      --contributor <NAME>  Who added this

Query Command

graphify-rs query search <QUESTION> [OPTIONS]
graphify-rs query path <FROM> <TO> [OPTIONS]
graphify-rs query explain <LABEL> [OPTIONS]

Subcommands:
  search    Search the graph with a natural language query
  path      Find shortest path between two nodes
  explain   Show full details of a node and its connections

Search Options:
  -g, --graph-dir <DIR>  Graph data directory [default: ./graphify-out]
      --dfs              Use DFS traversal instead of BFS
      --budget <N>       Max token budget [default: 2000]

Path/Explain Options:
  -g, --graph-dir <DIR>  Graph data directory [default: ./graphify-out]

Export Command

graphify-rs export [OPTIONS]

Options:
  -g, --graph-dir <DIR>  Graph data directory [default: ./graphify-out]
  -f, --format <FMT>     json | html | obsidian | graphml | canvas | neo4j | svg [default: json]
  -o, --output <PATH>    Output path
      --neo4j-url <URL>  Neo4j bolt URL for direct push

Worked Examples

Three worked examples demonstrate graphify-rs on different corpus types. Each includes raw input files, generated graph output, and a review evaluating graph quality.

Corpus Files Type Nodes Edges Communities Output
Rust Web API 6 Code-only (Rust) 46 40 12 worked/rust-web-api/output/
Multi-Language 5 Cross-lang (Py+TS+Rs+Go+MD) 64 63 15 worked/multi-lang/output/
Mixed Corpus 4 Code + Docs (Rust+MD) 55 46 21 worked/mixed-corpus/output/ 
# Rebuild any example
graphify-rs build worked/rust-web-api/raw -o worked/rust-web-api/output
graphify-rs report -g worked/rust-web-api/output

See each example's review.md for analysis of god nodes, communities, and surprising connections.

Architecture

src/
├── main.rs              # Entry point, async command dispatch
├── lib.rs               # Public module re-exports
├── cli.rs               # Clap derive CLI definition
├── config.rs            # GlobalConfig (CLI > env > file > defaults)
├── error.rs             # GraphifyError, ParserError (thiserror)
├── commands/            # One module per subcommand
│   ├── build.rs         # File discovery + hashing + graph creation
│   ├── build_discovery.rs  # File traversal, classification, filtering
│   ├── add.rs           # URL/file/paper ingestion
│   ├── query/           # Graph querying (search, path, explain)
│   ├── report.rs        # GRAPH_REPORT.md generation
│   ├── watch.rs         # File watcher (incremental rebuild)
│   ├── spec.rs          # Wiki structure generation
│   ├── lint.rs          # Graph health checks (--fix)
│   └── export.rs        # Multi-format export (7 formats)
├── graph/               # Core graph types + KnowledgeGraph wrapper
│   ├── mod.rs           # KnowledgeGraph (petgraph StableDiGraph)
│   └── types.rs         # Node, Edge, NodeKind, EdgeKind, Confidence
├── parser/              # Tree-sitter AST parsers (19 languages)
├── llm/                 # LLM integration (Anthropic + OpenAI)
├── ingest/              # Ingestion engine (URL, arXiv, tweet, PDF, DOCX)
├── cluster/             # Louvain community detection
├── report/              # Report generation (god nodes, surprising, gaps, questions)
├── export/              # Export formats (JSON, HTML, Obsidian, GraphML, SVG, Neo4j, Canvas)
└── utils/               # Shared utilities (tokenizer, hashing)

Graph Model

  • Directed graph — petgraph::StableGraph<Node, Edge, Directed>
  • 5 node types — Code, Document, Paper, Image, Rationale
  • 18 edge types — structural (Contains, Calls, Imports, Extends, ...) + semantic (Cites, ConceptuallyRelatedTo, ...)
  • 3 confidence levels — Extracted (1.0), Inferred (0.4-0.9), Ambiguous (0.1-0.3)
  • Stable indices survive node removal
  • Use neighbors_undirected() for clustering queries on directed graph

Configuration

Config file: ./graphify.toml or ~/.config/graphify-rs/config.toml

output_dir = "./graphify-out"
jobs = 4
verbose = false
log_level = "info"

Precedence: CLI args > environment variables > config file > defaults.

Environment variables: GRAPHIFY_OUTPUT, GRAPHIFY_JOBS.

LLM Configuration

Set your API key to enable semantic extraction from documents, papers, and images:

# Anthropic (default)
export ANTHROPIC_API_KEY=sk-ant-...

# Or OpenAI
export OPENAI_API_KEY=sk-...

Or in graphify.toml:

[llm]
provider = "anthropic"   # or "openai"
model = "claude-sonnet-4-20250514"

Without an API key, graphify-rs runs in AST-only mode — code files are parsed locally, document/paper/image files are skipped.

.graphifyignore

Create a .graphifyignore file in your project root to exclude files from the graph:

# Skip generated files
*.generated.ts
*.min.js

# Skip vendor directories
vendor/**
third_party/**

# Skip test fixtures
tests/fixtures/**

# Skip large data files
data/**/*.csv

Each line is a glob pattern. Empty lines and # comments are ignored. Patterns are matched against file paths relative to the corpus root.

Additionally, these directories are always skipped: .git, node_modules, target, __pycache__, .venv, venv, .tox, dist, build, graphify-out.

Using graph.json with an LLM

The generated graph.json is designed to be fed directly to an LLM for Q&A:

# 1. Build the graph
graphify-rs build ./my-project

# 2. Generate the report
graphify-rs report -g ./graphify-out

# 3. Ask questions
graphify-rs query search "How does authentication work?" -g ./graphify-out

# 4. Find paths between concepts
graphify-rs query path auth_middleware database -g ./graphify-out

# 5. Or give graph.json directly to your LLM assistant
cat ./graphify-out/graph.json | pbcopy
# Paste into ChatGPT/Claude: "Here's my project's knowledge graph. How is error handling structured?"

The graph preserves provenance — every edge has source_file and confidence so the LLM (or you) can trace claims back to source code.

Development

# Check compilation
cargo check

# Run all tests (unit + E2E)
cargo test

# Run unit tests only
cargo test --lib

# Run E2E tests only
cargo test --test '*'

# Run with tracing
RUST_LOG=debug cargo run -- build .

# Build release
cargo build --release

Testing

The project includes a comprehensive test suite:

Unit Tests (5 tests)

  • File classification — Markdown, Rust, unknown extensions
  • Sensitive file filtering — Env files, credentials detection

E2E Tests (483 active, 15 ignored)

22 test modules covering:

  • CLI — Argument parsing, help output, error handling
  • Build command — File discovery, hashing, graph node creation, incremental mode
  • Add command — URL/file/paper ingestion
  • Query command — Graph traversal, search, path finding, explain
  • Report generation — God nodes, surprising connections, communities, gaps, questions
  • Export formats — JSON, HTML, Obsidian, GraphML, Neo4j, SVG, Canvas
  • Graph operations — Node/edge management, hyperedges, JSON round-trip
  • Confidence — Score propagation, serde naming, export preservation
  • Semantic — Similarity edges, surprise scoring, cross-community detection
  • Multilang — Cross-language parsing, clustering, reports
  • LLM integration — Anthropic/OpenAI extraction, error handling, response parsing
  • Configuration — TOML parsing, CLI override precedence
  • Security — Sensitive file filtering, path traversal prevention, URL validation
  • Watch mode — Incremental file change detection
  • Lint — Graph consistency checks
  • Parser integration — 19-language code parsing, rationale extraction

Test Infrastructure

  • shared helpers — tests/common/mod.rs with setup/teardown utilities
  • fixtures — Sample project + graph JSON in tests/fixtures/
  • Dev dependencies — assert_cmd, predicates for CLI assertions; tempfile for temp directories

Run cargo test -- --nocapture to see detailed test output.

Tech Stack

Crate Purpose
clap CLI argument parsing (derive)
petgraph Core graph data structure
tokio Async runtime
serde + serde_json Serialization
thiserror + anyhow Error handling
walkdir Recursive file traversal
sha2 Content hashing (SHA256)
rayon Parallel processing
tracing Structured logging
indicatif Progress bars
reqwest HTTP client (for URL ingestion)
toml Config file parsing

License

MIT

About

Rust CLI that builds knowledge graphs from code, documents, papers, and images

Resources

Readme

Contributing

Contributing

Security policy

Security policy
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages