Draft list of Epics

Of course. Here is a complete list of all the Epics required to implement the finalized architecture.

This list acts as a high-level project roadmap. Each Epic represents a significant, valuable chunk of the project that can be broken down further into User Stories and Tasks.

---

Epic 1: Foundational Infrastructure & Environment Setup

• Goal: To create a consistent, containerized, and reproducible local development environment where all subsystems can be built and tested. This is the mandatory first step before any feature development can begin.
• Key User Stories/Features:
  • Containerize the Quarkus application.
  • Containerize the Python FastAPI application.
  • Orchestrate all services (Quarkus, FastAPI, MariaDB, Elasticsearch) for local development using docker-compose.
  • Establish database schemas and search index mappings.

Epic 2: Asynchronous Embedding Generation Pipeline

• Goal: To implement the core "write path" of the system, where newly created action labels are processed asynchronously to generate and store their embeddings without impacting API responsiveness.
• Key User Stories/Features:
  • Modify the "Create Label" API endpoint to trigger a non-blocking background task.
  • Implement the asynchronous worker logic within Quarkus.
  • Integrate the REST client call from Quarkus to the Python Embedding Service.
  • Ensure embeddings are correctly saved back to MariaDB and the full document is indexed in Elasticsearch.

Epic 3: Real-time Semantic Search API

• Goal: To implement the core "read path" of the system, providing the end-user with a real-time, natural language search endpoint.
• Key User Stories/Features:
  • Create a new search API endpoint in Quarkus that accepts a plain text query.
  • Implement the logic to call the Python Embedding Service to get a vector for the user's query.
  • Implement the logic to perform a k-Nearest Neighbor (knn) search in Elasticsearch using the query vector.
  • Implement response filtering based on a similarity score threshold (min_score).

Epic 4: Historical Data Backfill

• Goal: To process all existing action labels in the database that were created before this new system was implemented. This ensures the entire dataset is searchable, not just new entries.
• Key User Stories/Features:
  • Create a standalone, one-off script (can be Python or Java).
  • Implement logic in the script to read all labels from MariaDB where the embedding column is NULL.
  • Implement logic to process these labels in batches, generate embeddings, and update both MariaDB and Elasticsearch.

Epic 5: System Reliability & Recovery

• Goal: To make the system robust and production-ready by ensuring that any jobs that fail during the asynchronous process are not permanently lost.
• Key User Stories/Features:
  • Implement a scheduled cron job within Quarkus that runs periodically (e.g., hourly).
  • The cron job will query the database to find any records that are still missing an embedding after a certain amount of time.
  • The cron job will trigger the asynchronous embedding process for any unprocessed records it finds.

Epic 6: CI/CD & Deployment to Production

• Goal: To automate the building, testing, and deployment of all new components of this subsystem into a production environment.
• Key User Stories/Features:
  • Create a CI/CD pipeline (e.g., using GitHub Actions) for the Quarkus application.
  • Create a CI/CD pipeline for the Python FastAPI application.
  • Manage environment-specific configurations (database URLs, service addresses, secrets) for staging and production.
  • Document the deployment and rollback strategy.