[Feature]: Major Refactor. Migrating AI Logic to SDK and Implementing Component-Based Architecture

[Diegi97]

🚀 The feature, motivation and pitch

This issue proposes a significant refactor and enhancement to the architecture of ChatFAQ, aimed at evolving the vision of ChatFAQ to enable the creation of services based on genAI capabilities beyond RAG workflows. This initiative will shift the core AI 'logic' from the backend to our SDK, allowing for more flexible and modular AI component deployment and orchestration.

Overview of Changes:

1. AI Logic Migration: All AI processing logic will be migrated to the SDK. The backend will henceforth focus on deploying AI 'components' such as:

   • LLMs (Language Models)
   • Embedding Models
   • Retrievers

   Future work may include classifiers, NERs (Named Entity Recognition), text-to-speech, text-to-image models, and more.

2. Component Deployment: These AI components will be deployed on a Ray cluster.

3. Endpoint Exposure: The backend will expose specific endpoints to facilitate communication with these deployed components.

4. AI Orchestration Patterns: To streamline usage, common AI orchestration patterns can be predefined as a Finite State Machine (FSM) layer. Users will be able to create and utilize FSMs that uses these patterns directly through the SDK along with the logic that they want to implement. Common patterns include Retrieval Augmented Generation (RAG), agents, information extraction, synthetic data generation, data labeling, etc.

Tasks:

• Expose Endpoints in the Backend: Develop and document endpoints for interacting with the deployed AI components.
• Create RAG Layer in the SDK: Integrate the existing RAG logic within the SDK to handle AI orchestration.
• Remove RAG Logic and Models from the Backend: Refactor the backend to remove redundant AI processing logic, focusing solely on deployment and management.
• Update Admin Interface: Ensure the admin interface is updated to reflect these architectural changes and manage new components effectively.

Additional context