Refactor examples for readability

README.md

@@ -8,8 +8,6 @@
   <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-purple.svg" alt="License: MIT"></a>
 </p>
 
-A semi-opinionated RAG framework.
-
 <img src="./docs/pages/r2r.png" alt="Sciphi Framework">
 R2R was conceived to bridge the gap between experimental RAG models and robust, production-ready systems. Our semi-opinionated framework cuts through the complexity, offering a straightforward path to deploy, adapt, and maintain RAG pipelines in production. We prioritize simplicity and practicality, aiming to set a new industry benchmark for ease of use and effectiveness.
 
@@ -18,7 +16,7 @@ R2R was conceived to bridge the gap between experimental RAG models and robust,
 Using cloud application to deploy the pre-built basic pipeline:
 https://www.loom.com/share/e3b934b554484787b005702ced650ac9
 
-!! Note - The server has been removed from this repo - instead we now recommend using [SciPhi Cloud](https://app.sciphi.ai) to pair with the R2R framework for observability and optimization.
+Note - the example above uses [SciPhi Cloud](https://app.sciphi.ai) to pair with the R2R framework for observability and optimization. We intend on launching a self-hosted version of the cloud framework as our codebase matures.
 
 ### Quick Install:
 
@@ -37,118 +35,84 @@ export LOCAL_DB_PATH=local.sqlite
 # if using cloud providers (e.g. pgvector, supabase, ...)
 ```
 
-### Run the server with Docker:
-
-```bash
-docker pull emrgntcmplxty/r2r:latest
+### Full Install:
 
-# Place your secrets in `.env` before deploying
-docker run -d --name r2r_container -p 8000:8000 --env-file .env r2r
-```
+For a full installation, which is recommended for development, please refer to the [relevant documentation](https://r2r-docs.sciphi.ai/getting-started/full-install).
 
 ## Links
 
 [Join the Discord server](https://discord.gg/p6KqD2kjtB)
 
 [Read the R2R Docs](https://r2r-docs.sciphi.ai/)
 
-## Basic Examples
+## Key Features
+
+- **🚀 Deploy**: production-ready RAG pipelines with streaming in seconds
+- **🧩 Customize**: your pipeline using intuitive configuration files
+- **🔌 Extend**: your pipeline logic with code
+- **⚖️ Autoscale**: your pipeline in the cloud with [SciPhi](https://app.sciphi.ai/) 
+- **🤖 OSS **: framework built for and by the OSS community to make RAG easier.
+
+## Examples
 
-The project includes several basic examples that demonstrate application deployment and interaction:
+The project includes several examples that demonstrate application deployment and interaction:
 
-1. [`basic app`](r2r/examples/basic/app.py): This example runs the backend server, which includes the ingestion, embedding, and RAG pipelines served via FastAPI.
+### Servers
+
+1. [`basic_pipeline.py`](r2r/examples/servers/basic_pipeline.py): This example runs the backend server with the basic RAG pipeline, which includes the ingestion, embedding, and RAG pipelines served via FastAPI.
 
    ```bash
-   # If using a venv, replace `uvicorn` with `venv_path/bin/uvicorn`
-   uvicorn r2r.examples.basic.app:app
+   python -m r2r.examples.servers.basic_pipeline
    ```
 
-2. [`basic client`](r2r/examples/basic/run_client.py): This example should be run after starting the server. It demonstrates uploading text entries as well as a PDF to the local server with the python client. Further, it shows document and user-level vector management with built-in features.
+2. [`synthetic_query_pipeline.py`](r2r/examples/servers/synthetic_query_pipeline.py): This example demonstrates a more advanced pipeline that generates synthetic queries to improve the RAG pipeline's performance.
 
    ```bash
-   python -m r2r.examples.basic.run_client
+   python -m r2r.examples.servers.synthetic_query_pipeline
    ```
 
-3. [`academy`](r2r/examples/academy): A more sophisticated demo demonstrating how to build a more novel pipeline which involves synthetic queries
+3. [`reducto_pipeline.py`](r2r/examples/servers/reducto_pipeline.py): This example showcases the integration of the Reducto adapter for PDF ingestion.
 
    ```bash
-   # Launch the `academy` example application
-   # If using a venv, replace `uvicorn` with `venv_path/bin/uvicorn`
-   uvicorn r2r.examples.academy.app:app
+   python -m r2r.examples.servers.reducto_pipeline
+   ```
 
-   # Ask a question
-   python -m r2r.examples.academy.run_client search "What are the key themes of Meditations?"
+4. [`web_search_pipeline.py`](r2r/examples/servers/web_search_pipeline.py): This example demonstrates the usage of the `WebSearchRAGPipeline` for web search capabilities.
+
+   ```bash
+   python -m r2r.examples.servers.web_search_pipeline
    ```
-4. [`end-to-end`](docs/pages/examples/end-to-end.mdx): An example showing how to combine a complete web application with the basic RAG pipeline above.
 
-5. [`intelligence`](app.sciphi.ai): A cloud platform which can be used to deploy R2R pipelines powered by SciPhi
+### Clients
 
+1. [`run_basic_client.py`](r2r/examples/clients/run_basic_client.py): This example should be run after starting the basic pipeline server. It demonstrates uploading text entries and a PDF to the local server using the Python client. It also showcases document and user-level vector management with built-in features.
 
+   ```bash
+   python -m r2r.examples.clients.run_basic_client
+   ```
 
-### Full Install:
+2. [`run_basic_client_ext.py`](r2r/examples/clients/run_basic_client_ext.py): This example extends the basic client functionality and provides a more interactive experience for uploading and querying documents.
 
-Follow these steps to ensure a smooth setup:
-
-1. **Install Poetry:**
-
-   - Before installing the project, make sure you have Poetry on your system. If not, visit the [official Poetry website](https://python-poetry.org/docs/#installation) for installation instructions.
-
-2. **Clone and Install Dependencies:**
-
-  - Clone the project repository and navigate to the project directory:
-
-     ```bash
-     git clone git@github.com:SciPhi-AI/r2r.git
-     cd r2r
-     ```
-
-  - Copy the `.env.example` file to `.env`. This file is in the main project folder:
-
-     ```bash
-     cp .env.example .env
-
-     # Add secrets, `OPENAI_API_KEY` at a minimum
-     vim .env
-     ```
-
-  - Install the project dependencies with Poetry:
-
-     ```bash
-     # See pyproject.toml for available extras
-     # use "all" to include every optional dependency
-     poetry install -E parsing -E eval
-     ```
-
-  - Execute with poetry run:
-
-     ```bash
-     python -m r2r.examples.pdf_chat.run_client ingest
-     ```
-
-3. **Configure Environment Variables:**
-   - You need to set up cloud provider secrets in your `.env`. At a minimum, you will need an OpenAI key.
-   - The framework currently supports PostgreSQL (locally), pgvector and Qdrant with plans to extend coverage.
+   ```bash
+   python -m r2r.examples.clients.run_basic_client_ext
+   ```
 
-## Key Features
+3. [`run_synthetic_query_client.py`](r2r/examples/clients/run_synthetic_query_client.py): This example is optimized for interaction with the synthetic query pipeline server, showcasing the enhanced RAG pipeline's capabilities.
 
-- **🚀 Rapid Deployment**: Facilitates a smooth setup and development of production-ready RAG systems.
-- **⚖️ Flexible Standardization**: `Ingestion`, `Embedding`, and `RAG` with proper `Observability`.
-- **🧩 Easy to modify**: Provides a structure that can be extended to deploy your own custom pipelines.
-- **📦 Versioning**: Ensures your work remains reproducible and traceable through version control.
-- **🔌 Extensibility**: Enables a quick and robust integration with various VectorDBs, LLMs and Embeddings Models.
-- **🤖 OSS Driven**: Built for and by the OSS community, to help startups and enterprises to quickly build with RAG.
-- **📝 Deployment Support**: Available to help you build and deploy your RAG systems end-to-end.
+   ```bash
+   python -m r2r.examples.clients.run_synthetic_query_client
+   ```
 
 ## Core Abstractions
 
 The framework primarily revolves around three core abstractions:
 
-- The **Ingestion Pipeline**: Facilitates the preparation of embeddable 'Documents' from various data formats (json, txt, pdf, html, etc.). The abstraction can be found in [`ingestion.py`](r2r/core/pipelines/ingestion.py).
+- The **Ingestion Pipeline**: Facilitates the preparation of embeddable 'Documents' from various data formats (json, txt, pdf, html, etc.). The abstraction can be found in [`ingestion.py`](r2r/core/pipelines/ingestion.py) and relevant documentation is available [here](https://r2r-docs.sciphi.ai/core-features/ingestion).
 
-- The **Embedding Pipeline**: Manages the transformation of text into stored vector embeddings, interacting with embedding and vector database providers through a series of steps (e.g., extract_text, transform_text, chunk_text, embed_chunks, etc.). The abstraction can be found in [`embedding.py`](r2r/core/pipelines/embedding.py).
+- The **Embedding Pipeline**: Manages the transformation of text into stored vector embeddings, interacting with embedding and vector database providers through a series of steps (e.g., extract_text, transform_text, chunk_text, embed_chunks, etc.). The abstraction can be found in [`embedding.py`](r2r/core/pipelines/embedding.py) and relevant documentation is available [here](https://r2r-docs.sciphi.ai/core-features/embedding).
 
-- The **RAG Pipeline**: Works similarly to the embedding pipeline but incorporates an LLM provider to produce text completions. The abstraction can be found in [`rag.py`](r2r/core/pipelines/rag.py).
+- The **RAG Pipeline**: Works similarly to the embedding pipeline but incorporates an LLM provider to produce text completions. The abstraction can be found in [`rag.py`](r2r/core/pipelines/rag.py) and relevant documentation is available [here](https://r2r-docs.sciphi.ai/core-features/rag).
 
-- The **Eval Pipeline**: Samples some subset of rag_completion calls for evaluation. Currently [DeepEval](https://github.com/confident-ai/deepeval) is supported. The abstraction can be found in [`eval.py`](r2r/core/pipelines/eval.py).
+- The **Eval Pipeline**: Samples some subset of rag_completion calls for evaluation. Currently [DeepEval](https://github.com/confident-ai/deepeval) is supported. The abstraction can be found in [`eval.py`](r2r/core/pipelines/eval.py) and relevant documentation is available [here](https://r2r-docs.sciphi.ai/core-features/eval).
 
 Each pipeline incorporates a logging database for operation tracking and observability.

docs/pages/core-features/app.mdx

@@ -34,8 +34,7 @@ To customize the application, you can provide your own implementation classes fo
 Once the application is created, it can be launched using a web server such as Uvicorn. Here's an example of how to launch the application:
 
 ```bash
-# If using a venv, replace `uvicorn` with `venv_path/bin/uvicorn`
-uvicorn r2r.examples.basic.app:app
+python -m r2r.examples.basic.app
 ```
 
 In this example, Uvicorn is used to run the FastAPI application. The `r2r.examples.basic.app:app` argument specifies the module path and the variable name of the FastAPI application instance.

docs/pages/examples/basic-example.mdx

@@ -1,58 +1,56 @@
-# R2R Documentation
+# Basic Example
 
-## Launching
+This example demonstrates how to run a basic R2R application that includes ingestion, embedding, and RAG pipelines. It also shows how to interact with the local server using the [`R2RClient`](https://github.com/SciPhi-AI/R2R/blob/main/r2r/client/base.py).
 
-This example runs a basic application, which launches a server to handle ingestion, embedding, and RAG. Further, it demonstrates how to interact with the local server using the [`R2RClient`](https://github.com/SciPhi-AI/R2R/blob/main/r2r/client/base.py)`.
+## Step 1: Launch the Basic Application Server
+
+To launch the basic application server, run the following command:
 
 ```bash
-# Launch the application
-# If using a venv, replace `uvicorn` with `venv_path/bin/uvicorn`
-uvicorn r2r.examples.basic.app:app
+python -m r2r.examples.servers.basic_pipeline
 ```
 
-This launches a server which exposes the default pipeline. The settings of the pipeline are defined in the `config.json`. The default values for these settings are shown in the [config.json section](#configjson) below.
-```python
-# r2r.examples.basic.app
-from r2r.main import E2EPipelineFactory
+This command starts the backend server with the basic RAG pipeline, which includes the ingestion, embedding, and RAG pipelines served via FastAPI.
 
-app = E2EPipelineFactory.create_pipeline()
-```
+The server exposes the default pipeline, and the pipeline settings are defined in the `config.json` file. The default values for these settings are shown in the [config.json section](#configjson) below.
 
+## Step 2: Interact with the R2RClient
 
-## R2RClient Interface
+After starting the basic pipeline server, you can interact with it using the `R2RClient`. The `R2RClient` provides an interface to communicate with the server and perform various operations.
 
-The `R2RClient` provides an interface to interact with the server. Here are the main methods:
+To run the basic client example, use the following command:
 
-- `upload_and_process_file(document_id: str, file_path: str, metadata: Optional[dict] = None, settings: Optional[dict] = None) -> dict`: Uploads a file to the server and processes it. 
+```bash
+python -m r2r.examples.clients.run_basic_client
+```
 
-- `add_entry(document_id: str, blobs: Dict[str, str], metadata: Optional[Dict[str, Any]] = None, do_upsert: Optional[bool] = False, settings: Optional[Dict[str, Any]] = None) -> dict`: Adds an entry to the server. 
+This example demonstrates uploading text entries and a PDF to the local server using the Python client. It also showcases document and user-level vector management with built-in features.
 
-- `add_entries(entries: List[Dict[str, Any]], do_upsert: Optional[bool] = False, settings: Optional[Dict[str, Any]] = None) -> dict`: Adds multiple entries to the server. 
+The `R2RClient` provides the following main methods:
 
-- `search(query: str, limit: Optional[int] = 10, filters: Optional[Dict[str, Any]] = None, settings: Optional[Dict[str, Any]] = None) -> dict`: Searches the server for a query. 
+- `upload_and_process_file(document_id: str, file_path: str, metadata: Optional[dict] = None, settings: Optional[dict] = None) -> dict`: Uploads a file to the server and processes it.
 
-- `rag_completion(query: str, limit: Optional[int] = 10, filters: Optional[Dict[str, Any]] = None, settings: Optional[Dict[str, Any]] = None) -> dict`: Requests a RAG completion from the server. 
+- `add_entry(document_id: str, blobs: Dict[str, str], metadata: Optional[Dict[str, Any]] = None, do_upsert: Optional[bool] = False, settings: Optional[Dict[str, Any]] = None) -> dict`: Adds an entry to the server.
 
-- `filtered_deletion(key: str, value: Union[bool, int, str]) -> dict`: Deletes entries from the server based on a filter. 
+- `add_entries(entries: List[Dict[str, Any]], do_upsert: Optional[bool] = False, settings: Optional[Dict[str, Any]] = None) -> dict`: Adds multiple entries to the server.
 
-- `get_logs() -> dict`: Retrieves logs from the server. 
+- `search(query: str, limit: Optional[int] = 10, filters: Optional[Dict[str, Any]] = None, settings: Optional[Dict[str, Any]] = None) -> dict`: Searches the server for a query.
 
-- `get_logs_summary() -> dict`: Retrieves a summary of logs from the server. 
+- `rag_completion(query: str, limit: Optional[int] = 10, filters: Optional[Dict[str, Any]] = None, settings: Optional[Dict[str, Any]] = None) -> dict`: Requests a RAG completion from the server.
 
+- `filtered_deletion(key: str, value: Union[bool, int, str]) -> dict`: Deletes entries from the server based on a filter.
 
-## config.json
-<a name="configjson"></a>
+- `get_logs() -> dict`: Retrieves logs from the server.
 
+- `get_logs_summary() -> dict`: Retrieves a summary of logs from the server.
 
-During the example pipeline creation a default `config.json` is loaded and passed to the pipeline. It provides settings for the following services:
+## Step 3: Configure the Application
 
-- Database provider
-- LLM settings
-- Embedding settings
-- Parsing logic
-- and more.
+<a name="configjson"></a>
 
-The default values for the config are shown below. 
+During the example pipeline creation, a default `config.json` is loaded and passed to the pipeline. It provides settings for various components, including the database provider, LLM settings, embedding settings, parsing logic, and more.
+
+The default values for the configuration are shown below:
 
 ```json
 {
@@ -94,11 +92,11 @@ The default values for the config are shown below.
 
 _Note: For a full list of options, see [Config Setup](../core_features/config.mdx)._
 
-The pipeline is composed of three main components: `Ingestion`, `Embedding`, and `RAG` together with `Logging`.
+The pipeline consists of three main components: `Ingestion`, `Embedding`, and `RAG`, along with `Logging`.
 
-To launch your own custom application pipeline, you may run with the following
-```python
+To launch your own custom application pipeline, you can use the following code:
 
+```python
 class E2EPipelineFactory:
     ...
     app = E2EPipelineFactory.create_pipeline(
@@ -113,6 +111,12 @@ class E2EPipelineFactory:
     )
 ```
 
-## Demo: Installation and launching Basic App & Client
+This code allows you to customize the pipeline components and provide your own configuration file.
+
+## Demo: Installation and Launching Basic App & Client
 
 [![demo_shot](../getting-started/demo_shot.png)](https://github.com/SciPhi-AI/r2r/assets/68796651/c648ab67-973a-416a-985e-2eafb0a41ef0)
+
+The demo video above showcases the installation process and demonstrates how to launch the basic application server and interact with it using the R2RClient.
+
+By following the steps outlined in this example, you can quickly set up and run a basic R2R application, customize the pipeline components, and interact with the server using the provided client methods.