better alternative than supabase or neo4j #696
Replies: 9 comments
CodebaseAI: Intelligent Codebase Indexing PlatformProduct Requirements Document (PRD)Executive SummaryCodebaseAI is a next-generation, self-hostable codebase indexing platform that addresses the growing need for intelligent code understanding in AI-powered development workflows. Unlike existing solutions that offer basic text chunking and limited vector search, CodebaseAI provides a comprehensive, multi-dimensional approach to code analysis using SurrealDB's unique combination of vector, graph, and document capabilities. The platform enables developers to maintain complete control over their proprietary code while providing state-of-the-art semantic search, relationship mapping, and contextual understanding through flexible embedding model support and Model Context Protocol (MCP) integration. Problem StatementCurrent Market Pain PointsExisting solutions like claude-context are fundamentally limited:
Target User ChallengesEnterprise Development Teams:
AI/IDE Tool Builders:
Individual Developers:
Product VisionMission: Democratize intelligent codebase understanding through self-hostable, privacy-first technology that provides deeper insights than traditional text-based indexing. Vision: Every developer should have access to enterprise-grade codebase intelligence without sacrificing privacy, flexibility, or control over their data. Product OverviewCore Value Propositions
Key Differentiators
Functional RequirementsCore Features1. Intelligent Code IndexingMulti-Dimensional Embedding Storage:
Smart Chunking Strategies:
Relationship Mapping:
2. Embedding Model ManagementSupported Model Tiers: Tier 1A (Premium Local):
Tier 1B (Cloud Premium):
Tier 2 (Efficient Options):
Model Configuration System: interface EmbeddingMode {
semantic: boolean; // Understand code meaning
structural: boolean; // Code organization awareness
multilingual: boolean; // Multi-language support
realtime: boolean; // Fast enough for live indexing
contextWindow: number; // Maximum token handling
matryoshka: boolean; // Flexible dimension support
}3. SurrealDB IntegrationDatabase Schema Design: -- Code entities with multiple embedding types
DEFINE TABLE codeblock SCHEMAFULL;
DEFINE FIELD file ON TABLE codeblock TYPE string;
DEFINE FIELD content ON TABLE codeblock TYPE string;
DEFINE FIELD embedding_semantic ON TABLE codeblock TYPE array<float>;
DEFINE FIELD embedding_structural ON TABLE codeblock TYPE array<float>;
DEFINE FIELD language ON TABLE codeblock TYPE string;
DEFINE FIELD chunk_type ON TABLE codeblock TYPE string; // function, class, comment, etc.
-- Relationship modeling
DEFINE TABLE RELATION calls;
DEFINE TABLE RELATION imports;
DEFINE TABLE RELATION inherits;
DEFINE TABLE RELATION modifies; // git changesQuery Capabilities:
4. Model Context Protocol (MCP) ServerMCP Endpoints: Integration Targets:
5. Real-Time Indexing SystemFile System Monitoring:
Git Integration:
6. Visual Interface (Surrealist Integration)SurrealDB's Native GUI Support:
Non-Functional RequirementsPerformance RequirementsIndexing Performance:
Resource Requirements:
Recommended System Specifications:
Optimal System Specifications:
Security & Privacy RequirementsData Sovereignty:
Access Control:
Scalability RequirementsCodebase Size Support:
Multi-Project Support:
Technical ArchitectureSystem Componentsgraph TB
A[File System Watcher] --> B[Code Parser & Analyzer]
B --> C[Embedding Generator]
C --> D[SurrealDB]
D --> E[MCP Server]
E --> F[IDE Extensions]
E --> G[Claude Code]
E --> H[Web Interface]
D --> I[Surrealist GUI]
J[Model Manager] --> C
K[Git Integration] --> B
L[Configuration] --> J
Technology StackCore Platform:
Embedding Models:
Integration Layer:
|
|
man codebase indexing would be great this is something ive been planning on implementing in my personal version (ive been playing with memgraph for my kg), even creating a universal way for ai agents to log there additions/subtractions and why noted inside of a graph db, that way if a whole new agent or instance of your current agent connects it will already have everything loaded there to understand the current state of your project. currently ive built a 2d plain looking kg for my copy, but im looking at spicing it up with a custom three js 3d feel graph that fits right in the flow with archon |
yes indexing is a much needed feature and this is the perfect opportunity to do it right. you cant just use vector embeddings or graphs, a truly intelligent index that does what we expect it to do requires a sophisticated multi service hybrid approach. it can be done with seperate services but why thts super overkill and massively difficult, surrealdb provides literaly everything needed to beat every other codebase index out there even augments |
|
I appreciate the idea a lot @RepairYourTech! We are certainly not only considering SQL databases like moving from Supabase to Postgres, so I an open to this. Another option we are considering is ChromaDB (though we would still probably need a SQL db with that). SurrealDB looks awesome. I have never used it before so would have to look into it more. But regardless of the DB choice I agree that code indexing will be a very useful feature for Archon in the future! |
|
Unlike SurrealDB, Chroma only supports vectors. For an effective index you need vectors + graph at the very least. Surrealdb eneters the space as an allnin one solution fir these exact needs because its a teuly multimodal db thwt supports wverything yiu can possibly want exceot sql, but it has its own nosql called SurrealQL so you dont need any additional databases, just SurrealDB. If you use chroma youll still need neo4j + sql and possibky sone time series
|
|
I'm impressed by what I see about the SurrealDB. Looks interesting other than the fact that it uses SurrealQL Not sure how similar this is to industry standards like SQL, GraphQL or Cypher. Some testing will be required for sure but it looks nice and I like seeing the locally hosted in Docker capabilities. @Chillbruhhh, I'm a huge fan of Memgraph. I'd love to know some more about how you're using it? |
currently just as GraphRAG, similar to coles crawl4ai tool, i have ideas but i dont want to build something that wouldn't align with the official architecture, more specifically the db. |
|
moving to discussion |
|
Worth adding ArcadeDB to the comparison here — disclosure: I work on it, so take with the appropriate grain of salt. It covers the same "one engine instead of Supabase + Neo4j + Chroma" goal: vector, graph, document, key-value and time-series in a single ACID engine, embeddable or in Docker. The one thing I'd flag re: @tazmon95's query-language point — instead of a bespoke language, ArcadeDB speaks the existing standards: Cypher and Gremlin for graph, SQL, plus the MongoDB and Postgres wire protocols. So if the goal is combined vector+graph indexing without locking into a new dialect, it's worth a look. Happy to answer questions either way. |
Uh oh!
There was an error while loading. Please reload this page.
@coleam00 Id like to propose an alternative to the current storage and plans to add neo4j for graph.
take a look at SurrealDB
Surreal is amazing. its built in rust so its insanely fast and easy to host in docker or even embed right in any TS or Rust app
it supports multi dimension vector strategies, graph, document storage and more in a FREE solution.
so this solves literaly every need of Archon.
it has SurrealQL, so no need for SQL at all you can drop supabase entirely
you can use advanced embedding strategies with MRL
you can build relationship traversals with graph (codebase index anyone?)
it has native document and knowledge support which is a feature youre already trying to implement according to your ui.
it supports time series so you can track actual task progress and checkpoints etc.
SurrealDB might actually be the absolute best solution for Archon unless you want to shard services to dedicated services which has not only massive rescource needs but development overhead etc.
All reactions