Split monolithic documentation files into topic-based structure

README.md

@@ -27,11 +27,11 @@ A modern LLM chat interface with MCP (Model Context Protocol) integration.
 
 We have created a set of comprehensive guides to help you get the most out of Atlas UI 3.
 
-*   **[Getting Started](./docs/01_getting_started.md)**: The perfect starting point for all users. This guide covers how to get the application running with Docker or on your local machine.
+*   **[Getting Started](./docs/getting-started/installation.md)**: The perfect starting point for all users. This guide covers how to get the application running with Docker or on your local machine.
 
-*   **[Administrator's Guide](./docs/02_admin_guide.md)**: For those who will deploy and manage the application. This guide details configuration, security settings, access control, and other operational topics.
+*   **[Administrator's Guide](./docs/admin/README.md)**: For those who will deploy and manage the application. This guide details configuration, security settings, access control, and other operational topics.
 
-*   **[Developer's Guide](./docs/03_developer_guide.md)**: For developers who want to contribute to the project. It provides an overview of the architecture and instructions for creating new MCP servers.
+*   **[Developer's Guide](./docs/developer/README.md)**: For developers who want to contribute to the project. It provides an overview of the architecture and instructions for creating new MCP servers.
 
 ## For AI Agent Contributors
 

backend/mcp/progress_updates_demo/QUICKSTART.md

@@ -270,4 +270,4 @@ See `/backend/mcp/progress_updates_demo/main.py` for complete working examples.
 
 ## Documentation
 
-Full documentation: [Developer Guide - Progress Updates](../docs/03_developer_guide.md#progress-updates-and-intermediate-results)
+Full documentation: [Developer Guide - Progress Updates](../../../docs/developer/progress-updates.md)

docs/admin/README.md

@@ -0,0 +1,26 @@
+# Administrator's Guide
+
+This guide is for administrators responsible for deploying, configuring, and managing the Atlas UI 3 application.
+
+## Topics
+
+### Configuration
+- [Configuration Architecture](configuration.md) - Understanding the layered configuration system
+- [MCP Server Configuration](mcp-servers.md) - Setting up and configuring MCP tool servers
+- [LLM Configuration](llm-config.md) - Configuring Large Language Models
+
+### Security & Access Control
+- [Authentication & Authorization](authentication.md) - User authentication and group-based access control
+- [Compliance & Data Security](compliance.md) - Compliance levels and data segregation
+- [Tool Approval System](tool-approval.md) - Managing tool execution permissions
+
+### Storage & Infrastructure
+- [File Storage (S3)](file-storage.md) - Configuring S3-compatible object storage
+
+### Operations
+- [Logging & Monitoring](logging-monitoring.md) - Application logs and health monitoring
+- [Admin Panel](admin-panel.md) - Using the administrative interface
+
+### UI Customization
+- [Help Modal](help-config.md) - Customizing the Help/About modal
+- [Splash Screen](splash-config.md) - Configuring the startup splash screen

docs/admin/admin-panel.md

@@ -0,0 +1,9 @@
+# Admin Panel
+
+The application includes an admin panel that provides access to configuration values and application logs.
+
+*   **Access**: To access the admin panel, a user must be in the `admin` group. This requires a correctly configured `is_user_in_group` function.
+*   **Icon**: Admin users will see a shield icon on the main page, which leads to the admin panel.
+*   **Features**:
+    *   View the current application configuration.
+    *   View the application logs (`app.jsonl`).

docs/admin/authentication.md

@@ -0,0 +1,168 @@
+# Authentication & Authorization
+
+The application is designed with the expectation that it operates behind a reverse proxy in a production environment. It does **not** handle user authentication (i.e., logging users in) by itself. Instead, it trusts a header that is injected by an upstream authentication service.
+
+## Production Authentication Flow
+
+The intended flow for user authentication in a production environment is as follows:
+
+```
+   +-----------+      +-----------------+      +----------------+      +--------------------+
+   |           |      |                 |      |                |      |                    |
+   |   User    |----->|  Reverse Proxy  |----->|  Auth Service  |----->|  Atlas UI Backend  |
+   |           |  1.  |                 |  2.  |                |  3.  |                    |
+   +-----------+      +-----------------+      +----------------+      +--------------------+
+```
+
+1.  The user makes a request to the application's public URL, which is handled by the **Reverse Proxy**.
+2.  The Reverse Proxy communicates with an **Authentication Service** (e.g., an SSO provider, an OAuth server) to validate the user's credentials (like cookies or tokens).
+3.  Once the user is authenticated, the Reverse Proxy **injects the user's identity** (e.g., their email address) into an HTTP header and forwards the request to the **Atlas UI Backend**.
+
+The backend application reads this header to identify the user. The header name is configurable via the `AUTH_USER_HEADER` environment variable (default: `X-User-Email`). This allows flexibility for different reverse proxy setups that may use different header names (e.g., `X-Authenticated-User`, `X-Remote-User`). This model is secure only if the backend is not directly exposed to the internet, ensuring that all requests are processed by the proxy first.
+
+If using AWS Application Load Balancer (ALB) as the Auth Service, the following authentication configuration should be used:
+
+```
+    AUTH_USER_HEADER=x-amzn-oidc-data
+    AUTH_USER_HEADER_TYPE=aws-alb-jwt
+    AUTH_AWS_EXPECTED_ALB_ARN=arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/your-alb-name/...
+    AUTH_AWS_REGION=us-east-1
+```
+
+This configuration will decode the base64-encoded JWT passed in the x-amzn-oidc-data header, validate it, and extract the user's email address from the validated JWT.
+
+## Development Behavior
+
+In a local development environment (when `DEBUG_MODE=true` in the `.env` file), the system falls back to using a default `test@test.com` user if the configured authentication header is not present.
+
+## Configuring the Authentication Header
+
+Different reverse proxy setups use different header names to pass authenticated user information. The application supports configuring the header name via the `AUTH_USER_HEADER` environment variable.
+
+**Default Configuration:**
+```
+AUTH_USER_HEADER=X-User-Email
+```
+
+**Common Alternative Headers:**
+```
+# For Apache mod_auth setups
+AUTH_USER_HEADER=X-Remote-User
+
+# For some SSO providers
+AUTH_USER_HEADER=X-Authenticated-User
+
+# For custom reverse proxy configurations
+AUTH_USER_HEADER=X-Custom-Auth-Header
+```
+
+This setting allows the application to work with various authentication infrastructures without code changes.
+
+## Proxy Secret Authentication (Optional Security Layer)
+
+For additional security, you can configure the application to require a secret value in a specific header to validate that requests are coming from your trusted reverse proxy. This prevents direct access to the backend application, even if it's accidentally exposed.
+
+**When to Use Proxy Secret Authentication:**
+- When you want an additional layer of security beyond network isolation
+- To prevent unauthorized access if the backend accidentally becomes publicly accessible
+- To ensure requests only come from your approved reverse proxy
+
+**Configuration:**
+
+Add the following to your `.env` file:
+
+```bash
+# Enable proxy secret validation
+FEATURE_PROXY_SECRET_ENABLED=true
+
+# Header name for the proxy secret (default: X-Proxy-Secret)
+PROXY_SECRET_HEADER=X-Proxy-Secret
+
+# The actual secret value - use a strong, randomly generated value
+PROXY_SECRET=your-secure-random-secret-here
+
+# Optional: Customize the redirect URL for failed authentication (default: /auth)
+AUTH_REDIRECT_URL=/auth
+```
+
+**Reverse Proxy Configuration:**
+
+Configure your reverse proxy to inject the secret header with every request. Examples:
+
+**NGINX:**
+```nginx
+location / {
+    proxy_pass http://backend:8000;
+    proxy_set_header X-Proxy-Secret "your-secure-random-secret-here";
+    proxy_set_header X-User-Email $remote_user;
+    # ... other headers
+}
+```
+
+**Apache:**
+```apache
+<Location />
+    RequestHeader set X-Proxy-Secret "your-secure-random-secret-here"
+    RequestHeader set X-User-Email %{REMOTE_USER}e
+    ProxyPass http://backend:8000/
+    ProxyPassReverse http://backend:8000/
+</Location>
+```
+
+**Behavior:**
+- When enabled, the middleware validates the proxy secret on every request (except static files and the auth endpoint)
+- If the secret is missing or incorrect:
+  - **API endpoints** (`/api/*`): Return 401 Unauthorized
+  - **Browser endpoints**: Redirect to the configured auth URL
+- **Debug mode** (`DEBUG_MODE=true`): Proxy secret validation is automatically disabled for local development
+
+**Security Best Practices:**
+- Generate a strong, random secret (e.g., 32+ characters)
+- Store the secret securely in environment variables, not in configuration files
+- Use different secrets for different environments (dev, staging, production)
+- Rotate the secret periodically as part of your security policy
+- Never commit the secret to version control
+
+## Customizing Authorization
+
+**IMPORTANT: For production deployments, configuring authorization is essential.** The default implementation is a mock and **must be replaced** with your organization's actual authorization system. You have two primary methods to achieve this:
+
+### Recommended Method: HTTP Endpoint
+
+You can configure the application to call an external HTTP endpoint to check for group membership. This is the most flexible and maintainable solution, requiring no code changes to the application itself.
+
+1.  **Configure the Endpoint in `.env`**:
+    Add the following variables to your `.env` file:
+    ```
+    # The URL of your authorization service
+    AUTH_GROUP_CHECK_URL=https://your-auth-service.example.com/api/check-group
+
+    # The API key for authenticating with your service
+    AUTH_GROUP_CHECK_API_KEY=your-secret-api-key
+    ```
+
+2.  **Endpoint Requirements**:
+    Your authorization endpoint must:
+    *   Accept a `POST` request.
+    *   Expect a JSON body with `user_id` and `group_id`:
+        ```json
+        {
+          "user_id": "user@example.com",
+          "group_id": "admin"
+        }
+        ```
+    *   Authenticate requests using a bearer token in the `Authorization` header.
+    *   Return a JSON response with a boolean `is_member` field:
+        ```json
+        {
+          "is_member": true
+        }
+        ```
+
+If `AUTH_GROUP_CHECK_URL` is not set, the application will fall back to the mock implementation in `backend/core/auth.py`.
+
+When using the mock implementation (no external endpoint configured), **all users are treated as part of the `users` group by default**. This ensures that basic, non-privileged features remain available even without an authorization service. Higher-privilege groups such as `admin` still require explicit membership via the mock group table or your real authorization system.
+
+### Legacy Method: Modifying the Code
+
+For advanced use cases, you can still directly modify the `is_user_in_group` function located in `backend/core/auth.py`. The default implementation is a mock and **must be replaced** if you are not using the HTTP endpoint method.

docs/admin/compliance.md

@@ -0,0 +1,15 @@
+# Compliance and Data Security
+
+The compliance system is designed to prevent the unintentional mixing of data from different security environments. This is essential for organizations that handle sensitive information.
+
+## Compliance Levels
+
+You can assign a `compliance_level` to LLM endpoints, RAG data sources, and MCP servers. These levels are defined in `config/defaults/compliance-levels.json` (which can be overridden).
+
+**Example:** A tool that accesses internal-only data can be marked with `compliance_level: "Internal"`, while a tool that uses a public API can be marked as `compliance_level: "Public"`.
+
+## The Allowlist Model
+
+The compliance system uses an explicit **allowlist**. Each compliance level defines which other levels it is allowed to interact with. This prevents data from a highly secure environment (e.g., "HIPAA") from being accidentally sent to a less secure one (e.g., "Public").
+
+For example, a session running with a "HIPAA" compliance level will not be able to use tools or data sources marked as "Public", preventing sensitive data from being exposed.

docs/admin/configuration.md

@@ -0,0 +1,36 @@
+# Configuration Architecture
+
+The application uses a layered configuration system that loads settings from three primary sources in the following order of precedence:
+
+1.  **Environment Variables (`.env`)**: Highest priority. These override any settings from files.
+2.  **Override Files (`config/overrides/`)**: For custom, instance-specific configurations. These files are not checked into version control.
+3.  **Default Files (`config/defaults/`)**: The base configuration that is part of the repository.
+
+**Note**: The definitive source for all possible configuration options and their default values is the `AppSettings` class within `backend/modules/config/config_manager.py`. This class dictates how the application reads and interprets all its settings.
+
+## Key Override Files
+
+To customize your instance, you will place your own versions of the configuration files in the `config/overrides/` directory. The most common files to override are:
+
+*   **`mcp.json`**: Registers and configures the MCP (tool) servers that provide capabilities to the LLM.
+*   **`llmconfig.yml`**: Defines the list of available Large Language Models and their connection details.
+*   **`compliance-levels.json`**: Defines the security compliance levels (e.g., Public, Internal, HIPAA) and the rules for how they can interact.
+*   **`help-config.json`**: Populates the content of the "Help" modal in the user interface.
+*   **`splash-config.json`**: Configures the startup splash screen for displaying policies and information to users.
+*   **`messages.txt`**: Defines the text for system-wide banner messages that can be displayed to all users.
+
+## The `.env` File
+
+This file is crucial for setting up your instance. Start by copying the example file:
+
+```bash
+cp .env.example .env
+```
+
+Key settings in the `.env` file include:
+
+*   **API Keys**: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, etc.
+*   **Authentication Header**: `AUTH_USER_HEADER` configures the HTTP header name used to extract the authenticated username from your reverse proxy (default: `X-User-Email`).
+*   **Feature Flags**: Enable or disable major features like `FEATURE_AGENT_MODE_AVAILABLE`.
+*   **S3 Connection**: Configure the connection to your S3-compatible storage. For local testing, you can set `USE_MOCK_S3=true` to use an in-memory mock instead of a real S3 bucket. **This mock must never be used in production.**
+*   **Log Directory**: The `APP_LOG_DIR` variable points to the folder where the application log file (`app.jsonl`) will be stored. This path must be updated to a valid directory in your deployment environment.

docs/admin/file-storage.md

@@ -0,0 +1,50 @@
+# File Storage and Tool Integration
+
+The application uses S3-compatible object storage for handling all user-uploaded files. This system is designed to be secure and flexible, allowing tools to access files without ever needing direct S3 credentials.
+
+## Configuration Modes
+
+You can configure the file storage in one of two modes using the `.env` file.
+
+### 1. Development Mode (Mock S3)
+For local development and testing, you can use a built-in mock S3 service.
+
+*   **Setting**: `USE_MOCK_S3=true`
+*   **Behavior**: Files are stored on the local filesystem in the `minio-data/` directory. This mode is convenient as it requires no external services or credentials.
+*   **Use Case**: Ideal for local development. **This must not be used in production.**
+
+### 2. Production Mode (Real S3)
+For production, you must connect to a real S3-compatible object store like AWS S3, MinIO, or another provider.
+
+*   **Setting**: `USE_MOCK_S3=false`
+*   **Configuration**: You must provide the connection details in your `.env` file:
+    ```
+    S3_ENDPOINT_URL=https://your-s3-provider.com
+    S3_BUCKET_NAME=your-bucket-name
+    S3_ACCESS_KEY=your-access-key
+    S3_SECRET_KEY=your-secret-key
+    S3_REGION=us-east-1
+    ```
+
+## How MCP Tools Access Files
+
+The application uses a secure workflow that prevents MCP tools from needing direct access to S3 credentials. Instead, the backend acts as a proxy.
+
+```
+1. User uploads file
+   [User] -> [Atlas UI Backend] -> [S3 Bucket]
+                 |
+                 | 2. LLM calls tool with filename
+                 v
+4. Tool downloads file from Atlas UI API
+   [MCP Tool] <- [Atlas UI Backend] <- [S3 Bucket]
+                  ^
+                  | 3. Backend creates temporary, secure URL
+```
+
+1.  **File Upload**: A user uploads a file, which is stored in the configured S3 bucket.
+2.  **Tool Call**: The LLM decides to use a tool that needs the file and passes the `filename` as an argument.
+3.  **Secure URL Generation**: The Atlas UI backend intercepts the tool call. It generates a temporary, secure URL that points back to its own API (e.g., `/api/files/download/...`). This URL contains a short-lived capability token that grants access only to that specific file.
+4.  **Tool Execution**: The backend replaces the original `filename` argument with this new secure URL and sends it to the MCP tool. The tool can then make a simple `GET` request to the URL to download the file content.
+
+This process ensures that MCP tools can access the files they need without ever handling sensitive S3 credentials, enhancing the overall security of the system.

docs/admin/help-config.md

@@ -0,0 +1,29 @@
+# Customizing the Help Modal
+
+You can customize the content that appears in the "Help" or "About" modal in the UI by creating a `help-config.json` file.
+
+*   **Location**: Place your custom file at `config/overrides/help-config.json`.
+
+The file consists of a title and a list of sections, each with a title and content that can include markdown for formatting.
+
+## Example `help-config.json`
+
+```json
+{
+  "title": "About Our Chat Application",
+  "sections": [
+    {
+      "title": "Welcome",
+      "content": "This is a custom chat application for our organization. It provides access to internal tools and data sources."
+    },
+    {
+      "title": "Available Tools",
+      "content": "You can use tools for:\n\n*   Querying databases\n*   Analyzing documents\n*   Searching our internal knowledge base"
+    },
+    {
+      "title": "Support",
+      "content": "For questions or issues, please contact the support team at [support@example.com](mailto:support@example.com)."
+    }
+  ]
+}
+```