feat: vector index refactor

[jayhack]

Overview

Creates FileIndex and CodeIndex, which support the following API:

# Parse a codebase
codebase = Codebase.from_repo('fastapi/fastapi', language='python')

# Create index
index = FileIndex(codebase)
index.create() # computes per-file embeddings

# Save index to .pkl
index.save('index.pkl')

# Load index into memory
index.load('index.pkl')

# Update index after changes
codebase.files[0].edit('# 🌈 Replacing File Content 🌈')
codebase.commit()
index.update() # re-computes 1 embedding

# Search with natural language
results = index.similarity_search(
    "How does FastAPI handle dependency injection?",
    k=5  # number of results
)

# Print results
for file, score in results:
    print(f"\nScore: {score:.3f} | File: {file.filepath}")
    print(f"Preview: {file.content[:200]}...")

Next

We can extend this to support:

• CloudPickle (e.g. on Modal)
• Other vector supporters (Pinecone etc.)

[codecov]

Codecov Report

Attention: Patch coverage is 28.45188% with 171 lines in your changes missing coverage. Please review.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/codegen/extensions/index/file_index.py	21.64%	105 Missing ⚠️
src/codegen/extensions/index/code_index.py	33.67%	65 Missing ⚠️
src/codegen/extensions/tools/semantic_search.py	50.00%	1 Missing ⚠️

Additional details and impacted files

[github-actions]

🎉 This PR is included in version 0.21.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀