A modular toolkit to craft AI agents and RAG Pipelines.
## Development log
2024-08-9
----------
Made Changes in the code to log LLM traces to BrainTrust and to retrieve Input and
Completion Tokens from LLMS. Still need to make changes to
save those tokens and use them for later.
2024-08-05
----------
Modified the test "Arxiv Researcher" . The overseer agent decides by viewing the processed
titles if there is need to add more research papers to properly answer the user query ,
after that, the Arxiv Researcher Agent properly answers the user query using the most appropriate RAG chunks
given to it via similarity search. I still need
to update the pipeline to save abstract with the titles as
well , so the overseer agent has a better idea of the research paper.
2024-08-03
----------
Finished the document processing pipeline test and started to implement "Arxiv Researcher" ,a
test pipeline , to utilize the Rag tools incombination with mem0 agent.
2024-08-02
----------
Created functions to save make dense embeddings via hugging face models and sparse vectors using
fastembed library and to store these vectors in qdrant. Also
made functions to implement hybrid search or hybrid search.
2024-08-01
----------
Created functions to help extract information from any document , process them into chunks ,
utilized gpt-4o model to gather information about images present in the documents and insert this information instead of the image url.
2024-07-28
----------
Implement various functions to calculate tokens through models available at hugging face and tiktoken.
2024-08-23
----------
Used "https://github.com/mem0ai/mem0" to create a new agent class "mem0-agent" which utilizes the dynamic memory provided by mem0.
2024-08-20
----------
created memory buffer and memory storage functions for the Agent class.
2024-08-21
----------
integrated instructor into the agent class to get structured outputs.
2024-08-16
----------
Create an agent class and a tool class to create an AI agent which can utilize functions , and call them.
Here are the steps to use this Respository!:
git clone https://github.com/abideenml/LightAgents
- Navigate into project directory
cd LightAgents
- Install poetry via
pip install poetry .
- Run the command
poetry install
to install all the required packages. - Run
poetry shell
to created a new venv with all the required packages.
That's it!
Here are the core features of Light Agents
- Open Source Models via Ollama
- OpenAI Models
- Anthropic Models
- Gemini Models
- Groq Models
- Get Structured Outputs.
- Functions executions through tool abstraction.
- Schema validation for tools
- Executing the appropriate tool directly through the agent class
- Process any document (pdfs,docs) etc
- Make chunks using characters limits or token limits
- Make dense embeddings for the chunks using hugging face models.
- Make sparse embeddings for the chunks by using models available through fastembed library.
- Storing vectors and querying via qdrant.
- Creating examples
- making agents classes more modular.
If you find this code useful, please cite the following:
@misc{Zain2024LightAgents,
author = {Zain, Abideen , Adeel,Qureshi},
title = {LightAgents},
year = {2024},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/abideenml/LightAgents}},
}
If you'd love to have some more AI-related content in your life 🤓, consider:
- Connect and reach me on LinkedIn and Twitter
- Follow me on 📚 Medium
- Check out my 🤗 HuggingFace