Carrot-Mapper Agnostic Solutions

[CodeByKarthik]

Background & Problem Statement

🔺➤ Why are we using Azure in Carrot Mapper?

We are using Azure Services for 3 main reasons which are mentioned below:

• Azure functions
• Azure Blob Storage
• Deployment

🔺➤ Role of Azure Services in Carrot Mapper?

Azure Functions serve as a serverless computing platform that enables event-driven execution of tasks in Carrot Mapper. They are used for:

• We are using Azure Functions to automatically process, modify, and store mapping rules in Azure Blob Storage whenever a new request or event is triggered.

• When a user requests a mapping rules export, an Azure Function is triggered and the function fetches the mapping rules from the database automatically.

  It then generates a JSON, CSV, or SVG file and the function uploads the generated file to Azure Blob Storage for download.

• Azure Functions automate background data processing tasks like Transforming scan report data, Extracting mapping rules from datasets and rules generation.

Summary
- - - - - - -

• Carrot Mapper is hosted on Azure as a cloud-native application.
• Azure Functions are used to modify the mapping stored in Azure Blob Storage.
• Used for processing and exporting mapping rules.
• Used for scan report transformations and background tasks.

🔺➤ Problem Statement - Carrot Mapper is a Native Azure Application !

Currently, Carrot Mapper is not cloud-agnostic because it is locked into Azure services.

• Carrot Mapper relies on Azure Functions for event-driven processing.
• Carrot Mapper stores mapping data and scan report files in Azure Blob Storage.

⚠️ Deployment is tied to Azure infrastructure, making it difficult to use in AWS or GCP.

Now, we need Carrot Mapper to be deployed as a Cloud-Agnostic than a Native Application.

🔺➤ Why is this a Problem Statement?

The issue is that Carrot Mapper is not cloud-agnostic as it is locked into Azure.

• We need to make Carrot Mapper deployable on multiple cloud providers (AWS, GCP, etc.) instead of being limited to Azure.

What could be the potential Blockers?

Since Carrot Mapper’s core logic relies on the Azure Ecosystem, it is challenging to shift from Native Driven Application to Cloud Agnostic Solution.

• Strong reliance on Azure services → Azure Functions and Blob Storage are deeply embedded in the application logic.
• Refactoring complexity → Moving from an Azure-native solution to a cloud-agnostic architecture requires significant changes.
• Maintaining feature parity → Ensuring that features work the same way across AWS, GCP, or on-premise deployments.

---

Any Solutions?

Yes, there are multiple solutions which are briefly discussed below.

Approach 1 - Serverless Frameworks instead of Azure Functions

Use Serverless Frameworks to replace Azure Functions and make Carrot Mapper cloud-agnostic so that it can work on Azure, AWS, GCP, or even on-premise environments.

The azure Functions can be removed and replaced with Serverless Framework for event-driven execution of tasks in Carrot Mapper.

https://www.serverless.com/

There are other optios for function as a service

• OpenFaaS (Open-Source Serverless) - https://www.openfaas.com/
• Knative (Kubernetes-Native Serverless) - https://knative.dev/
• Nuclio (High-Performance Serverless) - https://nuclio.io/

But there is a problem!
- - - - - - - - - - - - - - - -

Even if we remove Azure functions, we need to solve the storage problem separately.

Use MinIO (Agnostic Storage) instead of Blob Storage

MinIO is an S3-compatible storage solution (MinIO’s S3-compatible SDK) instead of Azure Blob Storage (Azure Blob Storage SDK).

• MinIO is an open-source, S3-compatible object storage solution
• MinIO can be deployed on-premise or in AWS, GCP, or Azure.

https://min.io/

Other Multi-Cloud Storage Services

• [Cloudflare R2](https://www.cloudflare.com/products/r2/): S3-compatible storage that works across clouds.
• [Wasabi](https://wasabi.com/): S3-compatible storage that is cloud-agnostic.
• [backblaze:](https://www.backblaze.com/cloud-storage) Another comaptible storage with low-cost objectives.

🔺➤ Will it entirely solve the issue ?

While using Serverless Framework and MinIO (or other S3-compatible storage) almost solves the cloud-agnostic issue, there are still a few things that need to be considered:

• Setup deployment logic for each cloud.
• Setup separate documentation for users to deploy in each cloud (most preferably Azure, AWS and GCP).

🔺➤ What would be the solution?

1. Use Kubernetes

Write infrastructure as a code for kubernetes set up

{ optional } Use Terraform for writing the infrastructure code.

2. Multi-Cloud Docker Setup (Separate Docker Images for Each Cloud)

Approach 2 - Remove Azure Functions & Convert to a Python Package with Cloud Provider Endpoints

To achieve cloud-agnostic deployment, we remove Azure Functions and convert its logic into a standalone Python package.

This package will be responsible for executing Carrot Mapper’s core functionalities while integrating with different cloud providers dynamically.

Step 1:

• Move all Azure Functions (__init__.py) logic into a separate Python package.
• Remove Azure-specific dependencies and define generic Python functions for event-driven execution.
• Structure the logic into a reusable and installable Python package that can work across different cloud environments.

Step 2:

Reorganize the logic into a Python package that can be installed and used in different environments.

• Replace Azure Functions with a Django API-based event-handling mechanism to trigger and execute functions dynamically.
• The Django API will act as a centralized trigger service, replacing the need for AWS Lambda, GCP Cloud Functions, or Azure Functions.
• This approach eliminates cloud provider Vendor lock-in while ensuring seamless integration with different cloud environments.

Step 3:

Setting up the storage for triggers and uploads.

• Approach 1 - Cloud-Specific Storage:

Use different storage solutions depending on the selected cloud provider (AWS S3, GCP Cloud Storage, Azure Blob Storage).

• Approach 2 - Agnostic Storage Solution:

Implement MinIO or Cloudflare R2, which are S3-compatible and work across all cloud providers.

{ Recommended Approach }

- - - - - - - - - - - - - - - - - - - -

Using MinIO or other fully cloud-agnostic storage while keeping the option open for cloud-native storage services.

Setp 4:

Setting up deployment

• Approach 1: Dockerized Deployments for Each Cloud Provide

Create separate Docker configurations for Azure, AWS, and GCP and each cloud provider will have its own optimized Docker image.

• Approach 2: Kubernetes for Unified Deployment

Use Kubernetes (K8s) to deploy the application across different cloud environments. It allows for multi-cloud deployments without cloud-specific configurations.

🔺➤ How it works?

The Python package will contain the core logic, but we still use cloud-specific serverless functions only to trigger the package.

• Instead of writing business logic in Azure Functions, AWS Lambda, or GCP Cloud Functions, these functions will just call our Python package.

The Python package will contain the core logic and Django API will be used for triggers and executes functions.

In this approach, the Python package will encapsulate the core business logic, making it independent of any cloud provider.

However, we will still use cloud-specific serverless functions (such as AWS Lambda, GCP Cloud Functions, or Azure Functions) only for triggering the package.

• Instead of embedding business logic inside Azure Functions, AWS Lambda, or GCP Cloud Functions, these serverless functions will serve as lightweight triggers that simply invoke the Python package.
• The Python package will handle all processing, ensuring that the application remains cloud-agnostic while still leveraging cloud-native serverless capabilities for event-driven execution.

Additionally, the Django API will act as a centralized event handler, responsible for managing triggers, executing functions dynamically, and handling cloud-specific interactions.

This eliminates the need for hard-coded cloud dependencies inside the package while maintaining flexibility for deployment across different cloud environments.

Approach 3 - Routing Wrapper Model

This approach introduces a Routing Wrapper Model that determines which cloud provider (Azure, AWS, GCP, or Local) is being used and dynamically configures the system accordingly.

By performing this, Carrot Mapper can work across multiple cloud providers without requiring major code changes.

🔺➤ How it works?

Instead of directly calling Azure Functions and Blob Storage, we introduce a placeholder routing model that selects the correct cloud provider at runtime.

This is controlled via environment flags or configuration files.

• If Azure → Use Azure Blob Storage & Azure Functions
• If AWS → Replace with AWS S3 & AWS Lambda
• If GCP → Replace with Google Cloud Storage & Google Cloud Functions
• If Local → Use Local File System & Local Execution

Replace the Azure Functions and Blob Storage into the Routing Model placeholder. We can use flags to control what cloud provider which we want to use and based on the cloud provider, the configurations will be set up and imports will be executed.

This ensures the system can work with Azure, AWS, GCP, or even locally without major code changes.

When configuring the routing model, there are two approaches

1. Using a conditional based Approach

We use a simple condition to determine the cloud provider and execute the relevant logic.

Example Psudocode: Conditional Routing for Function Execution

import os

CLOUD_PROVIDER = os.getenv("CLOUD_PROVIDER", "local").lower()

def execute_function(function_name, *args, **kwargs):
    if CLOUD_PROVIDER == "azure":
        from cloud_providers.azure_handler import AzureFunctionHandler
        return AzureFunctionHandler().execute(function_name, *args, **kwargs)
    
    elif CLOUD_PROVIDER == "aws":
        from cloud_providers.aws_handler import AWSLambdaHandler
        return AWSLambdaHandler().execute(function_name, *args, **kwargs)

    elif CLOUD_PROVIDER == "gcp":
        from cloud_providers.gcp_handler import GCPFunctionHandler
        return GCPFunctionHandler().execute(function_name, *args, **kwargs)

    elif CLOUD_PROVIDER == "local":
        from cloud_providers.local_handler import LocalFunctionHandler
        return LocalFunctionHandler().execute(function_name, *args, **kwargs)

    else:
        raise ValueError(f"Unsupported cloud provider: {CLOUD_PROVIDER}")

2. Using YAML or JSON configuration files.

Instead of hardcoding cloud-specific logic, we define a configuration file (YAML/JSON) where the user can set the cloud provider.

Example Psuedocode: Configuration for Cloud-Agnostic Setup

cloud_provider: aws

providers:
  azure:
    function_handler: cloud_providers.azure_handler.AzureFunctionHandler
    storage_handler: cloud_providers.azure_storage.AzureStorage
  aws:
    function_handler: cloud_providers.aws_handler.AWSLambdaHandler
    storage_handler: cloud_providers.aws_s3_storage.S3Storage
  gcp:
    function_handler: cloud_providers.gcp_handler.GCPFunctionHandler
    storage_handler: cloud_providers.gcp_storage.GCPStorage
  local:
    function_handler: cloud_providers.local_handler.LocalFunctionHandler
    storage_handler: cloud_providers.local_storage.LocalStorage

Then, load the config dynamically as follows:

import yaml

with open("config.yaml", "r") as file:
    config = yaml.safe_load(file)

cloud_provider = config["cloud_provider"]
function_handler_class = config["providers"][cloud_provider]["function_handler"]

Steps for implementation

Step 1 :

Locate the Azure Functions that are responsible for:

• Processing scan report transformations
• Modifying mapping rules
• Uploading rules to Blob Storage
• Handling event-driven execution

These functions need to be abstracted into a routing model.

Step 2:

We introduce a Cloud Function Wrapper that routes the correct cloud provider function dynamically.

Example Pseudocode: Cloud Function Handler

class AzureFunctionHandler:
    def __init__(self):
        self.azure_function_url = os.getenv("AZURE_FUNCTION_URL", "http://localhost:7071")

    def execute(self, function_name, payload):
        """Trigger Azure Function via HTTP request."""
        url = f"{self.azure_function_url}/api/{function_name}"
        response = requests.post(url, json=payload)
        return response.json()



class AWSLambdaHandler:
    def __init__(self):
        self.client = boto3.client("lambda")

    def execute(self, function_name, payload):
        """Invoke AWS Lambda Function."""
        response = self.client.invoke(
            FunctionName=function_name,
            Payload=json.dumps(payload)
        )
        return json.load(response["Payload"])


class GCPFunctionHandler:
    def __init__(self):
        self.gcp_function_url = os.getenv("GCP_FUNCTION_URL")

    def execute(self, function_name, payload):
        """Trigger GCP Cloud Function via HTTP request."""
        url = f"{self.gcp_function_url}/{function_name}"
        response = requests.post(url, json=payload)
        return response.json()

Step 3:

Implement Cloud-Specific Methods

• Azure Functions Wrapper
• AWS Lambda Wrapper
• GCP Cloud Functions Wrapper
• Local Execution Wrapper (For Development)

Instead of calling Azure functions directly, all cloud function calls will go thorugh CloudFunctionRouter which makes carrot cloud-agnostic.

Step 4:

Currently, Carrot Mapper uses Azure Blob Storage to store exported mapping rules and files. Instead, we replace it with a unified storage interface that can use AWS S3, GCP Storage, Azure Blob, or local storage.

Example Pseudocode: Cloud Storage Handler

class CloudStorageRouter:
    def __init__(self):
        self.cloud_provider = os.getenv("CLOUD_PROVIDER", "local").lower()
        
        self.storage_map = {
            "azure": AzureStorage(),
            "aws": S3Storage(),
            "gcp": GCPStorage(),
            "local": LocalStorage()
        }

    def upload_file(self, file_path, destination):
        """Uploads a file to the respective cloud storage."""
        if self.cloud_provider in self.storage_map:
            return self.storage_map[self.cloud_provider].upload(file_path, destination)
        else:
            raise ValueError(f"Unsupported cloud provider: {self.cloud_provider}")

Final step:

Now that we removed Azure Functions & Blob Storage, we can deploy Carrot on any cloud!

• Modify docker-compose.yml to support different clouds.
• Or create a separate docker for each cloud provider and create a shell script to execute it based on the cloud provider flag which we used.
• Run the flag and everything will be executed.

Optional:

Configure the deployement process and add it in the logic so that the user will only needs to run the flag of the cloud provider and everything will be executed automatically.

🔺➤ Advantages of this approach

By using conditional logic + dynamic imports + a separate parsing model, you are essentially decoupling cloud-specific logic from the core application. This optimizes performance because:

• Only the required cloud SDKs are loaded (no unnecessary imports).
• Easier debugging since each cloud provider's logic is isolated.
• Faster Development - Requires less effort and quicker development.

Other Hybrid Approaches

💡Approach 1: Cloud Routing Technique for a Python Package

• Convert the entire cloud-related logic into a Python package.
• Introduce a Cloud Routing System that dynamically selects the right cloud provider based on configuration.
• Each cloud service (Azure, AWS, GCP, Local) will have its own module inside the package.
• The core logic stays intact, and the correct cloud module is loaded at runtime.

💡Approach 2: Hybrid Approach (Combining Python Package + Routing)

• Convert the cloud-related logic into a Python package.
• Introduce a routing layer using flags that load the correct cloud provider dynamically.
• The package handles cloud logic, while the main app only interacts with the routing system.

💡Approach 3: Python Package + Agnostic Storage + Deployment Setup using Flags

This approach builds upon Approach 2 but adds support for storage independence and automated deployment configuration using flags.

• Instead of being locked into Azure Blob Storage, we use MinIO (an S3-compatible storage solution) that works across AWS, GCP, Azure, or even local setups.
• We also introduce flags for deployment automation, meaning users can simply set a flag and deploy without making manual changes.

💡Approach 4: Python Package + Agnostic Storage + Kubernetes Configuration

This approach expands Approach 3 by introducing Kubernetes as a deployment solution.

• Instead of manually setting up cloud services, Kubernetes ensures uniform deployment across all providers.
• By containerizing the application and deploying it in Kubernetes clusters, Carrot Mapper can run seamlessly across Azure, AWS, or GCP without needing cloud-specific configurations.

🔺➤ Deployment Complexity ?

Approach 3 and 4 will be preferred methodologies 12 Factor App Principles and our use case.

• Approach 3 is slightly less in complexity.
• Approach 4 is slightly more in complexity.

🔺➤ Cloud Portability and Latency?

💡Approach 3 (Python package & Deployment Routing + Agnostic Storage + Flags)

• Best for minimal complexity and flexibility across cloud providers.

• Fast and easy deployment using flags.

• Good balance between maintainability and automation.

• {Optional} - Use Serverless Functions instead of relying on Django Core functionalities for Triggers and Actions.

💡Approach 4 (Python package + Agnostic Storage + Kubernetes)

• Best for large-scale, enterprise-grade deployments.

• Highly scalable and fully automated but more complex setup.

• Ideal for multi-cloud and hybrid cloud environments.

• {Optional} - Use Serverless Functions instead of relying on Django Core functionalities for Triggers and Actions.

Preferred Methodology

• If we want a quick and flexible solution with minimal complexity, go for Approach 3 (Python package & Deployment Routing + Flags).
• If we need enterprise-grade, highly scalable, and fully automated deployments, go for Approach 4 (Kubernetes-based).

---

For the Carrot-Agnostic Solution Discussions in PDF, I have attached and please find it below:
Carrot-Mapper-Agnostics-solution.pdf