Description

Revamps the NIBE prototype to use latest Langchain. Also separates responsibilities:

• asf_hp_installer_chatbot: experimentation of data heavy pipelines, with associated utils
• rag: all things end-to-end RAG Langchain related, this includes document processing pipelines
• api: uses Langserve, a FastAPI wrapper. This enables more ways to interact with the chains throughout dev, test, and prod.

Note: WhatsApp interface hasn't been implemented, and this PR will be updated to reflect these changes. The aim of this PR is to first revamp the prototype to use the latest Langchain LCEL framework.

Addresses #6 , branch is 2_revamp

Instructions for Reviewer

Contact me for Langfuse integration details (env vars needed in your .env file)

1. Checkout branch 2_revamp
2. Run make pip-install if you already have the project conda environment setup
3. Test the integration test script that mimics the prototype test script:

python rag/tests/test_retrieval.py

To test the API:

1. Have poetry installed, feel free to ask me how best to do this
2. conda deactivate, then cd api
3. poetry env use 3.10, then poetry install
4. langchain app serve --app app.server:app
5. Navigate to 0.0.0.0:8000
6. Test out the /invoke endpoint by replacing the string in the JSON request "input": "string", with a query.

Please pay special attention to and other notes

• Setting up, feel free to ask me questions
• I'm still rethinking/ implementing the WhatsApp interface and the messaging system, will be best to not serialise the files but to log conversations to a message queuing system i.e. Redis. Then periodically send incremental batches to an S3 parquet file.

Checklist:

• I have refactored my code out from notebooks/ N/A

• I have checked the code runs

• I have tested the code

• I have run pre-commit and addressed any issues not automatically fixed

• I have merged any new changes from dev

• I have documented the code

  • Major functions have docstrings
  • Appropriate information has been added to READMEs

• I have explained this PR above

• I have requested a code review

Other additional features to review:

• Add CLI for running ingestion and reindexing of docs into vector database
• Add CLI command for saving vector database snapshots to local and desired S3 path
• Add CLI command for recreating an index from snapshot saved locally or in S3 path

Review instructions, after make install or make pip-install, test out CLI commands with chatbot --help, there are descriptions for other commands available.

Hi Solomon!

Thanks for the really comprehensive work, I am loving the configurability of the chatbot and think this enables us to have a lot of flexibility in terms of what we are able to do. The change to FastAPI from Flask makes sense especially when it comes to asynchronous nature/potential of chatbots and also love the FastAPI documentation that comes with it.

I was able to eventually get both the test script up and running and then also run the API with a local Qdrant vector database, so that's good. Ran into a couple of bugs which I've commented on and we've discussed (there have been quite a few changes since this PR was put out so that's unsurprising). It would be great to know what the steps are to deploy the chatbot on the EC2 instance, this is potentially we can discuss at some point in the coming weeks. I think it might be useful @Jack-Vines to review the API part of this PR as myself and @crispy-wonton do not have much experience with these packages or how they set up.

As a more general comment, I think we are missing some documentation at the top of each file, i.e. a clear message which is readable for those who either have no or some experience with either web applications or RAG pipelines and I think potentially some visual representations would really help us understand the functionality of the different files and the architecture overall. I am happy to help out with this!

I think as a whole, we also need documentation on how to set everything up from poetry, to LangFuse and of course with regards to running an EC2 instance! It doesn't seem trivial, at least to a beginner like myself, so I think this would be super helpful.

Thanks again for the stellar work!