CareCompanion

AI-Agentic Medical & Bureaucracy Assistant for the Elderly

CareCompanion is an AI-powered inclusive automation platform designed to bridge the digital gap for the elderly and digitally illiterate. By combining Multimodal Generative AI (Gemini 2.5 Flash) with RPA (UiPath), CareCompanion enables users to schedule medical appointments, order prescriptions, and navigate complex healthcare bureaucracies using simple voice notes or text via Telegram.

---

🚀 System Architecture & Flow

The system operates in a closed-loop agentic workflow:

sequenceDiagram
    actor Elderly as Elderly User
    participant Bot as Telegram Bot
    participant BE as Cloud Run Backend
    participant Gemini as Gemini 2.5 Flash
    participant DB as Cloud SQL Postgres
    actor Caregiver as Caregiver (Dashboard)
    participant UiPath as UiPath Robot
    participant Portal as Demo Booking Portal

    Elderly->>Bot: Sends voice note or text message
    Bot->>BE: Triggers webhook (/telegram)
    BE->>Gemini: Parses audio/text and extracts entities
    Gemini-->>BE: Returns structured JSON & confidence score
    alt Low Confidence / Missing Fields (Ambiguous)
        BE->>DB: Saves task as 'pending_verification'
        Caregiver->>BE: Reviews, corrects details, and approves
        BE->>UiPath: Triggers RPA Job
    else High Confidence
        BE->>UiPath: Instantly triggers RPA Job
    end
    UiPath->>Portal: Enters patient details & registers
    Portal-->>UiPath: Generates booking receipt & ticket
    UiPath->>BE: Calls callback (/complete) with receipt URL
    BE->>BE: Generates beautiful confirmation PDF with Barcode
    BE->>Bot: Delivers custom PDF ticket to User
    Bot-->>Elderly: Sends ticket PDF directly in chat

Loading

1. Intake & Clarification Session: The elderly user interacts with @yudhae_carecomp_bot on Telegram. If the user has an active task in pending_verification status created in the last 15 minutes, the backend automatically treats the message as a conversational followup. It calls Gemini to merge the new information into the existing task context instead of creating a duplicate task.
2. Stateful Multi-Agent Graph (LangGraph): The intake parameters are fed into a compiled LangGraph state workflow composed of:
   • Intake Parser Agent: Transcribes audio/photos and parses entities.
   • Safety & History Agent: Checks database history to flag duplicate bookings or health warnings.
   • Clinical Validation Agent: Evaluates parameters and routes tasks either to caregiver dashboard (HITL) or straight to automation.
   • RPA Dispatcher Agent: Instantly dispatches clear tasks to the UiPath robot.
3. Human-in-the-Loop (HITL) Gatekeeping & Affirmation Recognition:
   • If the task requires caregiver review, it is saved as pending_verification.
   • Conversational Follow-up: If a user sends a quick positive affirmation (e.g. "yes", "ya", "baik", "oke", "confirm", "setuju"), the Intake node immediately skips LLM processing, confirms the gathered details, and updates the task status to queued_rpa.
   • The family caregiver can also open the Caregiver Dashboard to review, correct, and manually approve tasks.
4. UiPath Automation & Simulation Callback:
   • The robot navigates the hospital portal (/demo-portal), registers the booking, and downloads the receipt.
   • If UiPath credentials are not configured or expired, the backend catches the token exception gracefully and marks the task as queued_rpa for manual caregiver intervention without crashing the Telegram bot workflow.
   • Task records are pre-created inside the RPA node so a valid task UUID is passed to background simulation threads, preventing callback sequence crashes.
5. Dynamic Document Delivery: Upon completion, the backend dynamically generates a beautifully formatted digital ticket PDF using ReportLab (with patient details, queue number, and a scannable barcode) and sends it directly back to the user's Telegram chat.

---

🛠️ Technology Stack

• Frontend: React (Vite) static dashboard styled with custom Vanilla CSS variables, deployed to Google Cloud Storage (GCS) static website hosting.
• Backend: FastAPI (Python 3.11) web server deployed to Google Cloud Run container registry.
• Database: PostgreSQL hosted on Google Cloud SQL for secure storage of profiles, messages, task queues, and audit trails.
• Cognitive Engine: Google Gemini 3.5 Flash using the modern google-genai SDK for structured JSON extraction (supporting datetime-relative parsing and smart follow-up merges).
• Automation Engine: UiPath Orchestrator API for triggering unattended automation jobs.
• Delivery Gateway: Telegram Bot API for conversational Zero-UI messaging.

---

📊 Database Schema (Cloud SQL Postgres)

1. users (Patient Profiles)

Holds elderly user records. The whatsapp_number field is repurposed to store the Telegram Chat ID.

• id (UUID, Primary Key)
• full_name (VARCHAR)
• whatsapp_number (VARCHAR, Unique)
• bpjs_number (VARCHAR, Unique)
• home_address (TEXT)
• medical_history (TEXT)
• created_at (TIMESTAMP)

2. conversations (Communication Logs)

Tracks incoming and outgoing chat interactions.

• id (UUID, Primary Key)
• user_id (UUID, Foreign Key referencing users.id)
• direction (VARCHAR: incoming, outgoing)
• message_type (VARCHAR: text, audio, image)
• media_url (TEXT)
• transcription (TEXT)
• created_at (TIMESTAMP)

3. tasks (Automation Queue)

Saves requests dispatched to the RPA robots.

• id (UUID, Primary Key)
• user_id (UUID, Foreign Key referencing users.id)
• task_type (VARCHAR: doctor_booking, medicine_order, bpjs_check)
• status (VARCHAR: pending_verification, queued_rpa, running_rpa, completed, failed, cancelled)
• extracted_data (JSONB)
• uipath_job_id (VARCHAR)
• result_document_url (TEXT)
• created_at (TIMESTAMP)
• updated_at (TIMESTAMP)

4. explainability_logs (AI Confidence Auditing)

Logs confidence scores, explanations, and flagged fields that triggered low confidence.

• id (UUID, Primary Key)
• task_id (UUID, Foreign Key referencing tasks.id)
• confidence_score (NUMERIC)
• is_ambiguous (BOOLEAN)
• explanation_text (TEXT)
• flagged_fields (VARCHAR[])

---

⚙️ Local Development & Setup

Prerequisites

• Python 3.11+
• Node.js & npm (for React Dashboard)
• Google Cloud SDK (gcloud CLI)

1. Backend Setup

1. Navigate to the backend/ directory:

   cd backend

2. Create and activate a virtual environment:

   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install dependencies:

   pip install -r requirements.txt

4. Set up environment variables in a .env file (or set on Cloud Run):

   DATABASE_URL="postgresql://postgres:PASSWORD@IP:5432/carecompanion"
   GEMINI_API_KEY="AIzaSy..."
   GEMINI_MODEL="gemini-2.5-flash"
   TELEGRAM_BOT_TOKEN="8624757720:..."
   UIPATH_CLIENT_ID="..."
   UIPATH_USER_KEY="..."
   UIPATH_FOLDER_ID="..."

5. Run the backend development server:

   python main.py

2. Frontend Dashboard Setup

1. Navigate to the frontend/ directory:

   cd frontend

2. Install Node packages:

   npm install

3. Start the Vite hot-reloading development server:

   npm run dev

---

🔮 Future Enhancements: Next-Gen Agentic AI

To scale CareCompanion into a fully autonomous healthcare system, we plan to implement:

1. Self-Healing RPA (Vision & Tool Use): Training the AI agent to interact directly with web browsers (using Playwright) to read screen layouts, navigate medical forms, and dynamically handle OTPs or layout shifts.
2. Conversational Voice Streaming (Gemini Live API): Allowing elderly users to speak naturally to CareCompanion in real-time phone calls using low-latency WebRTC streams.
3. IoT Smart Dispenser Integration: Linking the platform to physical smart pillboxes to automatically trigger refill orders when pill levels are low.