An open-source application for extracting, analyzing, and reconstructing charts from documents using quantized open-source LLMs running on CPU inference.
- RAG system for chart QA in docx/pdf documents.
- Complete solution of a Chart QA system that actually delivers accurate responses
- can be easily extended to cover more statistical calculations or different chart types
- Multi-format Document Support: Extract charts from PDF and DOCX files
- Multi-chart Type Support: Handle Bar, Pie, Scatter, and Line charts, can be extended to handle more
- Semantic Search: Retrieve relevant charts using natural language queries via RAG
- Agentic Q&A: Answer questions about chart data using ReAct reasoning pattern
- Chart Reconstruction: Regenerate charts from extracted data with LLM-powered fallback
- Role-Based Access: Support for user, tester, and admin roles
- Performance Metrics: Track retrieval and inference times for optimization
- Potential: Can be easily extended to handle more chart types or statistical calculations.
- Python 3.9+
- Ollama installed with Mistral model:
ollama pull mistral - ~8GB RAM for smooth operation
python -m venv venv
venv\Scripts\activate
pip install -r requirements.txtstreamlit run test.py- Login/Sign Up - Create an account or log in
- Upload Document - Upload a PDF, DOCX, or PPTX file
- Ask Questions - Type natural language questions about charts
- View Results - Get answers with source chart images
- Redraw Charts - Regenerate charts using the reconstruction engine (Admin/Tester only)
Document Upload ↓ Chart Extraction (chart_extractor2.py) ↓ Vector Database Build (db_build.py) ↓ Semantic Search (test.py → FAISS) ↓ Agentic Reasoning (practice2.py → Ollama) ↓ Answer + Chart Display
chart_extractor2.py- Extract charts and metadata from documents using vision modelsdb_build.py- Build FAISS vector store from extracted chart datachart_reconstructor.py- Reconstruct and visualize charts with LLM fallbackpractice2.py- ReAct agent for chart question answering with statistical toolstest.py- Streamlit web interface and RAG orchestrationauthentication.py- User authentication and role management
- Ollama: Local LLM inference engine
- Mistral-7B: Quantized open-source LLM model
- LangChain: Framework for LLM applications and agents
- FAISS: Semantic similarity search and vector database
- Sentence-Transformers (all-MiniLM-L6-v2): Text embeddings for semantic search
- Pix2Struct: Vision-to-text model for generating structured string from a chart image
- User uploads PDF/DOCX
- System extracts images and metadata
- Charts are identified and segmented
- Chart data is structured and stored
- Chart metadata is embedded using Sentence-Transformers
- FAISS index is built for semantic search
- Vector store is persisted locally
- User asks a question in natural language
- Semantic search finds the most relevant chart
- ReAct agent determines needed calculations
- Statistical tools extract values from chart data
- LLM formulates final answer
- Original chart data is retrieved
- Matplotlib draws the chart
- If drawing fails, LLM generates custom Python code
- Reconstructed chart is displayed to user
| Role | Permissions |
|---|---|
| User | Upload documents, ask questions, view charts |
| Tester | All user permissions + performance metrics + chart redrawing |
| Admin | All permissions + role assignment |
- Modify user roles by editing the users.yml file.
- Document processing: 2-5 minutes (depends on file size)
- Chart question answering: 3-30 seconds
- Chart reconstruction: 2-5 seconds
- Ensure Ollama is running:
ollama serve - Check model is installed:
ollama list
- Ensure document contains readable charts
- Check extracted_images folder for extraction results
- Try rephrasing the query
- Ensure you enter valid Nyckel API credentials in .env file
- Get free credentials by creating an account on https://www.nyckel.com/
- Ollama Documentation
- LangChain Documentation
- FAISS Documentation
- Sentence-Transformers
- Pix2Struct Deplot Model
- Streamlit Documentation