I developed this local LLM from scratch to address my need to have access to an LLM that can be inputed my unpublished academic ideas and works, without worring about it them being stored somewhere in the cloud. While there are great Open-Source options for local LLM, such as "LM Studio", as a hands-on scientist I wanted to have full and granular control over all the facilities the LLM has. Moreover, I saw this as an opportunity to learn and understand more about how to implement a RAG based LLM that truly works, without relying on a fully-fledged counterpart like "LM Studio".
- While the default is using local LLMs through Ollama, you have the option to choose online LLM APIs (e.g., OpenAI).
- For local models, you can choose any model that is available in Ollama.
- You have control over what's included in your database.
- You have control over wether to use RAG or use the entire documents.
- Containerize (with Docker)
- User authentication (OIDC / OAuth)
- Add SSL (HTTPS)
- Role-based access control
- Vector Database
- Add more Database that the user can choose from
- Audit logging and error reporting
- CI/CD pipeline and deployment hardening
docker xxxxcurl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.20.0 shsudo apt-get install tesseract-ocr
If you have another Python version (other than the above version) installed, you can use
pyenv. e.g., while in the project dir, you can usepyenv install 3.11and thenpyenv local 3.11.
python3 -m pip install virtualenv
python3 -m virtualenv venv
source venv/bin/activate
pip install -r requirements.txtOptions:
- --server.port <port_number>: [Optional] give the port number you want streamlit to run on. If not used, the first available port counting from 8501 will be used.
- --server.headless : [Optional] When
true, doesn't allow streamlit to open a browser automatically. It's preferred when running streamlit in a server, especially without a GUI.
streamlit run app.py --server.port 8501 --server.headless true