A lightweight, self-contained Python chat system with local AI agents and file operations. This system uses Hyena3-4B-Instruct(Custom Qwen Fork) and llama-cpp-python for inference, with Rich for a beautiful console interface. Features AI agents with workspace-based file operations and dynamic terminal sizing.
- Out-of-the-box experience: Just run
run_app.batto install dependencies and start - Minimal dependencies: Uses UV for fast dependency resolution
- AI Agent System: Multiple specialized agents with different personalities and capabilities
- File Operations: AI can read, write, and list files in a designated workspace
- Dynamic UI: Responsive interface that adapts to terminal size
- Workspace Management: Secure file operations within user-defined directories
- Conversation Management: Save, load, and export chat conversations
- Rich Console UI: Beautiful terminal interface with formatted panels and colors
- PEP 8 Compliant: Clean, documented codebase following Python standards
tiny-local-agent/
├── .venv/ # Python virtual environment (auto-created)
├── app/
│ ├── llm_engine/ # LLM inference engine
│ │ ├── __init__.py
│ │ └── engine.py # Model loading and inference
│ ├── interface/ # User interface and chat system
│ │ ├── chat.py # Main chat system orchestrator
│ │ └── chat_components/ # Modular UI components
│ │ ├── agent_management/ # AI agent configuration
│ │ │ ├── agent_handler.py
│ │ │ └── agents.json # Agent personalities and prompts
│ │ ├── commands/ # Command processing
│ │ │ └── command_processor.py
│ │ ├── conversations/ # Chat history management
│ │ │ └── manager.py
│ │ ├── display/ # UI rendering
│ │ │ └── chat_display.py
│ │ ├── file_operations/ # AI file tools
│ │ │ ├── ai_tools.py # AI-accessible file operations
│ │ │ ├── workspace_manager.py # Workspace management
│ │ │ └── file_manager.py # Secure file operations
│ │ ├── header/ # System header display
│ │ │ └── header.py
│ │ ├── input_management/ # User input handling
│ │ │ └── manager.py
│ │ └── stats/ # Session statistics
│ │ └── manager.py
│ └── app.py # Main entry point
├── models/ # Contains the GGUF model file
│ └── Hyena3-4B-Instruct-2507.Q6_K.gguf
├── pyproject.toml # Project metadata and dependencies
├── README.md # Project documentation
├── run_app.bat # Script to install dependencies and run the app
└── uv.lock # Lockfile for reproducible dependency resolution- Windows OS (for .bat script compatibility)
- Python 3.10+ (recommended: 3.11+)
- Internet connection (only for initial dependency installation)
- ~2GB disk space for the Hyena3-4B GGUF model
Download and extract the project to a folder of your choice.
Double-click run_app.bat or run from command line:
run_app.batThis will:
- Install UV (if not already installed)
- Create a virtual environment (
.venv) - Sync dependencies using
uv sync - Run the application with
uv run -m app.app
- Set workspace:
/workspace test_filesto enable file operations - Chat with AI: Type messages to interact with AI agents
- Use commands: Type
/helpto see all available commands - File operations: Ask AI to read/write files in the workspace
The project uses UV for fast dependency resolution and virtual environment management. Dependencies are listed in pyproject.toml and installed automatically when you run run_app.bat.
engine.py loads the Hyena3-4B model from the models/ directory using llama-cpp-python. The model is configured with:
- Context length: 8192 tokens (can do up to 200k with proper configuration)
- Threads: 8 (adjustable based on your CPU)
- GPU layers: -1 (full GPU acceleration)
When you interact with the system:
- Agent Selection: Choose from multiple specialized AI personalities
- File Operations: AI can read, write, and list files in your workspace
- Dynamic Responses: Each agent has unique capabilities and conversation style
- Tool Integration: AI automatically uses file tools when needed
The system provides secure file operations:
- User-defined workspace: Set any directory as your AI workspace
- AI file tools:
@read_file(),@write_file(),@list_files()functions - Security restrictions: Files limited to workspace directory
- Size limits: 2MB maximum file size for safety
chat.py creates a rich console interface that:
- Accepts user input with auto-completion and command history
- Displays dynamic headers that adapt to terminal size
- Shows conversation history in formatted panels
- Provides real-time feedback for AI operations
Edit app/interface/chat_components/agent_management/agents.json to:
- Add/remove AI agents
- Define agent names, specialties, and system prompts
- Configure file operation capabilities
Modify app/interface/chat_components/file_operations/workspace_manager.py to:
- Change file size limits (default: 2MB)
- Adjust allowed file extensions
- Modify workspace behavior
Edit display components in app/interface/chat_components/ to:
- Change colors and styling
- Modify header layout
- Adjust panel formatting
- Missing dependencies: Ensure
run_app.batcompletes without errors - Model not found: Verify
Hyena3-4B-Instruct-2507.Q6_K.ggufis inmodels/directory - File access errors: Use
/workspace <directory>to set a valid workspace - Display issues: Resize terminal window for better layout
- Slower responses: Reduce context length or use smaller model
- Memory issues: Close other applications when running large models
- File operations: Keep workspace directory small for better performance
- PEP 8 Compliant: All functions and classes have proper docstrings
- Modular Design: Clean separation of concerns
- Type Hints: Consistent type annotations throughout
- Error Handling: Robust error handling and user feedback
The modular architecture makes it easy to add new features:
- Create new manager classes in
chat_components/ - Add commands in
command_processor.py - Update agents in
agents.json - Modify display components as needed
This project is open-source and free to use. The Hyena3 model is licensed under its respective terms.
- llama-cpp-python for LLM inference
- Rich for the beautiful console UI
- Hyena3 for the language model
- UV for fast dependency management