Approach for project (WATER)

[XxSURYANSHxX]

@pradeeban and all the mentors, Since my prevs work/contribution already involve good experience for DWiM project KathiraveluLab/DWiM#1 (comment) and DICOM, i would love to contribute to this too to further learn and sharpen my skills. I have proposed an initial plan for this which is divided into multiple TASKS so that it is easy for all the mentors to review my work and keep all the PRs clean. I would love to have the feedback on this approach and if you have any suggestions, i would love to implement those also. This is phase 1 of this project and i aslo divided into weeks which i will cover quickly this month. Thankyou.

Module A: Edge Data Ingestion & Harmonization (Weeks 1–4)
Focus: Establishing the local data pipeline using Niffler to extract, flatten, and scrub DICOM metadata directly from a localized clinical environment.

Week 1: Local PACS & Staging Environment Setup
Task: Deploy an isolated, localized DICOM server and populate it with simulated clinical imaging data.
Technical Detail: Run the Orthanc DICOM server via a Docker container. Write a Python script utilizing REST APIs to automatically download anonymized radiological sequences (e.g., from The Cancer Imaging Archive) and push them into the Orthanc staging directory.
Goal: Establish a functional, high-throughput local clinical data source to build against without risking exposure to real Protected Health Information (PHI).

Week 2: SCP Listener & Asynchronous Extraction
Task: Configure Niffler to interface directly with your local Orthanc instance and extract imaging pipelines.
Technical Detail: Clone the Niffler repository and modify the configuration to deploy Service Class Provider (SCP) listeners. Execute automated DICOM network commands (C-FIND to query, C-MOVE to asynchronously transfer) to pull images from Orthanc to your local environment.
Goal: Successfully capture simulated real-time radiological data streams locally.

Week 3: Recursive Metadata Flattening
Task: Implement the Niffler Metadata Extractor module to parse deeply nested DICOM headers.
Technical Detail: Execute Niffler's flattening engine in Python to extract complex tags and map them into a structured, single-level format. Configure a local MongoDB instance to ingest and store this flattened metadata.
Goal: Transform unstructured DICOM headers into clean, queryable feature matrices ready for machine learning consumption.

Week 4: The Privacy Enforcement Engine
Task: Build the non-negotiable, HIPAA-compliant anonymization module for the edge node.
Technical Detail: Implement customizable scrubbing profiles natively in Python. Map highly identifying tags (like PatientID and StudyInstanceUID) to cryptographically secure, 128-bit surrogate UUIDs while meticulously tracking the mapping to maintain longitudinal study consistency.
Goal: Guarantee that no residual PHI leaks into the flattened metadata tables before any network transmission occurs.

Module B: Orchestration Scaffolding & Telemetry (Weeks 5–7)
Focus: Deploying the CONTROL-CORE components and establishing ultra-low-latency, brokerless message passing.

Week 5: Mediator Initialization & API Gateway
Task: Deploy the central brain of the distributed execution system and secure its access points.
Technical Detail: Initialize the Python 3.7 Flask-based CONTROL-CORE Mediator application via Docker. Configure a Kong API Gateway to establish multitenant authentication, ensuring only permitted nodes can connect.
Goal: Create a secure, centralized cloud routing director capable of authenticating incoming edge nodes.

Week 6: ZeroMQ Brokerless Telemetry Hub
Task: Establish real-time state synchronization between the edge node and the Mediator.
Technical Detail: Integrate the osparc-control Python module, utilizing ZeroMQ PUB/SUB sockets. Write scripts to continuously stream local resource utilization metrics (CPU/RAM) from your edge node directly to the Mediator without relying on a centralized message broker.
Goal: Achieve sub-5ms ping-pong message latency for real-time edge node monitoring.

Week 7: Microservice Wrappers & Remote Triggers
Task: Expose the Niffler edge extraction and anonymization functions as RESTful microservices.
Technical Detail: Wrap the core Python scripts using the FastAPI framework. This natively generates Swagger UI documentation and creates non-blocking endpoints that the Mediator can trigger asynchronously.
Goal: Provide a highly standardized, modern API interface for the orchestration layer to control edge operations.

Module C: Workflow Allocation & Secure Routing (Weeks 8–10)
Focus: Coding the mathematical logic for hybrid-cloud routing and implementing fault-tolerant data transmission.

Week 8: Allocation Engine Logic Core
Task: Develop the Python script that evaluates the dynamic hybrid-cloud cost function.
Technical Detail: Write the decision-making algorithm that continuously compares edge node hardware utilization and network bandwidth against an incoming workflow's required data payload size and computational cost.
Goal: Automate the routing decision: execute the workflow locally if optimal, or flag it for cloud offloading if edge resources are starved.

Week 9: Encrypted Payload Transmission & Caching
Task: Implement the secure data routing mechanism from the edge to the cloud.
Technical Detail: Utilize AES-256-GCM symmetric encryption with unique 32-byte keys for data payload transfer. Implement a stateful halt-and-resume protocol backed by Python pickle files to queue data locally if the network connection drops, resuming automatically upon reconnection.
Goal: Ensure zero data corruption and uncompromisable privacy during cloud offloading, even in unstable networks.

Week 10: Containerized Inference Execution
Task: Package a baseline ML pipeline and trigger it dynamically via the allocation engine.
Technical Detail: Write a Dockerfile for a simple Python model script. Program the Mediator to dynamically spin up this Docker container based on the Week 8 allocation logic and pass the securely routed, decrypted DICOM payload as an input volume.
Goal: Achieve an end-to-end, zero-touch distributed execution loop orchestrated entirely by your newly built WATER framework scaffolding.

This will have further PHASES to complete this project end to end by keeping 350 hours GSoC timeline in mind.

[pradeeban]

@XxSURYANSHxX you are exceeding our expectations. Impressive progress so far. @Chali-healthy, other mentors of HEALTHY, and I appreciate your effectiveness and thoughtfulness on the WATER project! Please make sure to include all the listed mentors and me in your proposal draft when it is ready.

[XxSURYANSHxX]

Sure @pradeeban, I will keep this accordingly and will follow/implementation every suggestions by all the mentors. Thankyou

[XxSURYANSHxX]

Heyy @pradeeban, @Chali-healthy. I have been exploring the core vision for the WATER project, specifically the need to build a unified, platform-agnostic framework that facilitates workflow allocation towards edge resources using cutting-edge, decentralized technologies. I would like to propose an updated architecture built entirely on Declarative Intent-Based Orchestration (IBO).

What I am doing and why:
To explain this simply, think of this approach like a decentralized ride sharing app, but for healthcare computing. Here researchers simply submit a "Declarative Intent." This is just a basic text file saying something like: "I need to run this medical imaging task on a highly secure, HIPAA-compliant edge node with at least 8GB of RAM."

The WATER framework then acts purely as a "Matchmaker." It uses a decentralized, peer-to-peer network where all the edge and cloud computers continuously broadcast their available hardware capacity and security clearance to each other using ZeroMQ. The matchmaker reads the researcher's intent, scores the available computers, and automatically sends the task to the perfect match.

I am choosing this route because it guarantees 100% tool-agnosticism. The framework doesn't need to know if it's running a Niffler DICOM extraction, a CONTROL-CORE neuromodulation simulation, or a custom AI model. It just receives a packaged Docker container, finds the best node, and executes it. This entirely eliminates the need for a central server bottleneck and prevents us from having to write custom integrations for every new medical tool a hospital might want to use.

Here is my detailed 10-week initial implementation plan to build a fully functional proof-of-concept from scratch before the GSoC coding period officially begins:

Module A: The Decentralized State Mesh (Weeks 1–3)
Focus: Building the brokerless peer-to-peer network so edge and cloud nodes can dynamically share their live hardware status without relying on a central server.

• Week 1: Node Daemon Setup. I will set up the Python environment and build a lightweight daemon using the psutil library. This script will track the local machine's live CPU/RAM usage and assign it infrastructure tags (e.g., clearance: edge-only, type: hospital-server).
• Week 2: Peer-to-Peer Broadcasting. I will integrate ZeroMQ PUB/SUB sockets into the daemon. By running multiple local Python terminals, I will prove that these simulated nodes can broadcast their live hardware states to each other asynchronously without a message broker.
• Week 3: The Live Resource Map. I will write the logic so every node continuously listens to the ZeroMQ network, building and maintaining a localized, real-time "map" of the entire computing continuum's available power.

Module B: The Intent Resolution Engine (Weeks 4–7)
Focus: Coding the "Matchmaker" algorithm that binds workloads to the right nodes based purely on their privacy and compute needs.

• Week 4: The Intent Schema. I will draft the universal YAML schema that defines a workload's Intent. It will simply declare the required Docker image, the minimum memory needed, and the strict privacy level required for the task.
• Week 5: The Scoring Algorithm. I will build the core Python matchmaker. It will take an incoming YAML Intent, compare it against the live ZeroMQ resource map from Module A, and score the nodes based on how perfectly they satisfy the request.
• Week 6: Hard Privacy Constraints. I will code the absolute healthcare compliance filters. If an Intent demands privacy: strict-local, the algorithm will instantly disqualify any node tagged as a public cloud, ensuring patient data is anchored to the hospital edge.
• Week 7: Resolution Stress Testing. I will simulate a high-traffic environment by feeding the engine dozens of different mock Intents (some requiring heavy cloud computing, others requiring local edge privacy) to verify it reliably selects the optimal target node ID every time.

Module C: Universal Container Execution (Weeks 8–10)
Focus: Physically executing the matched workloads using standard containers to prove the framework can run any agnostic tool seamlessly.

• Week 8: Docker SDK Integration. I will integrate the Python Docker SDK (docker-py) directly into the node daemon. This gives the node the physical ability to pull container images, mount local medical data volumes, and run isolated environments.
• Week 9: End-to-End Orchestration. I will link the whole lifecycle: A user submits an Intent -> The engine scores and matches it -> The execution command is sent via ZeroMQ -> The target node receives the command and spins up the Docker container.
• Week 10: Verification and Teardown. I will add the final lifecycle hooks. Once a container finishes its task, the node will securely log the exit status, update its available resources in the global registry, and cleanly spin down the environment.

I believe this Intent-Based approach will make WATER incredibly resilient, scalable, and easy to maintain for remote edge environments. I would love to hear your thoughts on this architecture! If you have any suggestions, please let me know i'll be glad to implement those. Thankyou

[XxSURYANSHxX]

@pradeeban, Should i make a proposal directly with my full plan or should i first setup some initial structure with this week wise plan. Whatever you suggest. :)

[pradeeban]

@XxSURYANSHxX A proposal draft makes sense at this point. The actual coding could wait for the coding period.

[XxSURYANSHxX]

Got it @pradeeban, will do exactly that.