🚀 Scalable Data Platform

A comprehensive, production-ready data platform that ingests diverse data sources, transforms data for analytics, and exposes secure REST APIs.

📋 Features

• Data Ingestion: JSON, CSV, XML, Excel, SQL sources
• Incremental Loading: Watermarking techniques for efficient updates
• Data Transformation: Cleansing, normalization, and enrichment
• Data Warehouse: Star schema design in Azure Synapse Analytics
• APIs: Python FastAPI + .NET Core Web API
• Security: Azure Entra ID authentication and RBAC
• Infrastructure: Azure Bicep templates for IaC
• CI/CD: Automated deployment pipelines

🏗️ Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Data Sources  │───▶│  Data Pipeline  │───▶│  Data Warehouse │
│  (JSON/CSV/XML) │    │  (ADF + Python) │    │  (Synapse SQL)  │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                                │
                                ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   .NET API      │◀───│   Load Balancer │───▶│   Python API    │
│  (Swagger UI)   │    │   (App Gateway) │    │  (FastAPI Docs) │
└─────────────────┘    └─────────────────┘    └─────────────────┘

🚀 Quick Start

Prerequisites

• Python 3.8+
• .NET 8.0 SDK (for .NET API)
• Azure CLI (for deployment)
• Git

1. Clone the Repository

git clone <repository-url>
cd GIS_takehome

2. Set Up Python Environment

# Create virtual environment
python3 -m venv venv

# Activate virtual environment
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

3. Generate Sample Data

# Generate sample data files
python3 scripts/generate_sample_data.py

This creates sample data in the sample_data/ directory:

• customers.json, customers.csv, customers.xml
• products.json, products.csv, products.xml
• sales.json, sales.csv, sales.xml
• weather.json, weather.csv
• stocks.json, stocks.csv
• news.json, news.csv

4. Start the APIs

Python FastAPI

# Navigate to API directory
cd api

# Start the server
uvicorn main:app --reload --host 0.0.0.0 --port 8000

Access Python API:

• API: http://localhost:8000
• Documentation: http://localhost:8000/docs
• Alternative Docs: http://localhost:8000/redoc

.NET Core API

# Navigate to .NET API directory
cd dotnet-api

# Start the server
dotnet run

Access .NET API:

• API: https://localhost:7001
• Swagger: https://localhost:7001/swagger

5. Test the Platform

Quick Test Script

# Run comprehensive test
python3 scripts/test_with_sample_data.py

Manual Testing

# Test health endpoints
curl http://localhost:8000/health
curl https://localhost:7001/health

# Test data endpoints
curl http://localhost:8000/api/data/customers
curl https://localhost:7001/api/data/sales

📊 API Endpoints

Python FastAPI Endpoints

Endpoint	Method	Description
/health	GET	Health check
/docs	GET	Interactive API docs
/api/data/customers	GET	Customer data
/api/data/products	GET	Product data
/api/data/sales	GET	Sales data
/api/analytics/summary	GET	Analytics summary
/api/monitoring/status	GET	System status
/api/auth/login	POST	User authentication

.NET Core API Endpoints

Endpoint	Method	Description
/health	GET	Health check
/swagger	GET	Swagger UI
/api/data/sales	GET	Sales data
/api/analytics/summary	GET	Analytics summary
/api/monitoring/status	GET	System status
/api/auth/login	POST	User authentication

🔧 Configuration

Environment Variables

Create a .env file in the project root:

# API Settings
API_SECRET_KEY=your-super-secret-key-with-at-least-32-characters
API_ALGORITHM=HS256
API_ACCESS_TOKEN_EXPIRE_MINUTES=30

# Database
DATABASE_URL=sqlite:///./dataplatform.db

# Azure Settings (for deployment)
AZURE_STORAGE_ACCOUNT_NAME=your-storage-account
AZURE_STORAGE_ACCOUNT_KEY=your-storage-key
AZURE_SYNAPSE_SERVER=your-synapse-server.database.windows.net
AZURE_SYNAPSE_DATABASE=your-synapse-database
AZURE_SYNAPSE_USERNAME=your-username
AZURE_SYNAPSE_PASSWORD=your-password

# CORS
ALLOWED_ORIGINS=["http://localhost:3000", "http://localhost:8000"]
ALLOWED_HOSTS=["localhost", "127.0.0.1"]

🚀 Azure Deployment

1. Prerequisites

# Install Azure CLI
brew install azure-cli  # macOS
# or download from: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli

# Login to Azure
az login

2. Deploy Infrastructure

# Create resource group
az group create --name rg-data-platform --location "East US"

# Deploy infrastructure
az deployment group create \
  --resource-group rg-data-platform \
  --template-file infrastructure/main.bicep \
  --parameters \
    resourceGroupName=rg-data-platform \
    location="East US" \
    dataPlatformName=your-data-platform \
    synapseAdminUsername=admin \
    synapseAdminPassword="SecurePassword123!" \
    enablePrivateEndpoints=true

3. Deploy APIs

# Deploy Python API
cd api
az webapp up --name your-python-api --resource-group rg-data-platform --runtime "PYTHON:3.9"

# Deploy .NET API
cd ../dotnet-api
az webapp up --name your-dotnet-api --resource-group rg-data-platform --runtime "DOTNETCORE:8.0"

🧪 Testing

Automated Testing

# Run all tests
python3 scripts/test_with_sample_data.py

# Run smoke tests
python3 scripts/smoke-tests.py

# Run performance tests
k6 run infrastructure/performance-tests.js

Manual Testing

1. Health Checks

   curl http://localhost:8000/health
   curl https://localhost:7001/health

2. API Documentation

   • Python: http://localhost:8000/docs
   • .NET: https://localhost:7001/swagger

3. Data Endpoints

   curl http://localhost:8000/api/data/customers
   curl https://localhost:7001/api/data/sales

📁 Project Structure

GIS_takehome/
├── api/                          # Python FastAPI
│   ├── main.py                   # Main application
│   ├── config.py                 # Configuration
│   ├── auth.py                   # Authentication
│   ├── models.py                 # Data models
│   ├── services.py               # Business logic
│   └── middleware.py             # Custom middleware
├── dotnet-api/                   # .NET Core API
│   ├── Controllers/              # API controllers
│   ├── Models/                   # Data models
│   ├── Services/                 # Business services
│   └── Program.cs                # Main entry point
├── data_processing/              # Data pipeline
│   ├── ingestion.py              # Data ingestion
│   ├── transformation.py         # Data transformation
│   └── sample_data_sources.py    # Sample data generation
├── infrastructure/               # Azure infrastructure
│   ├── main.bicep                # Bicep template
│   └── data-factory-pipeline.json # ADF pipeline
├── scripts/                      # Automation scripts
│   ├── generate_sample_data.py   # Sample data generation
│   ├── test_with_sample_data.py  # Comprehensive testing
│   └── deploy.py                 # Deployment script
├── sample_data/                  # Generated sample data
├── tests/                        # Test files
├── requirements.txt              # Python dependencies
└── README.md                     # This file

🔒 Security

• Authentication: JWT tokens with Azure Entra ID
• Authorization: Role-based access control (RBAC)
• Secrets: Azure Key Vault integration
• Network: Private endpoints and network security groups
• Data: Encryption at rest and in transit

📈 Monitoring

• Application Insights: Performance monitoring
• Azure Monitor: Infrastructure monitoring
• Health Checks: API health monitoring
• Logging: Structured logging with Serilog/structlog

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request

📄 License

This project is licensed under the MIT License.

🆘 Troubleshooting

Common Issues

1. Import Errors

   # Ensure virtual environment is activated
   source venv/bin/activate
   
   # Reinstall dependencies
   pip install -r requirements.txt

2. Port Already in Use

   # Kill process using port 8000
   lsof -ti:8000 | xargs kill -9

3. Database Connection Issues

   • Check .env file configuration
   • Ensure SQLite file is writable
   • For Azure: verify connection strings

4. Azure Deployment Issues

   # Check Azure CLI login
   az account show
   
   # Verify resource group exists
   az group show --name rg-data-platform

Getting Help

• Check the documentation in /docs directory
• Review the API documentation at /docs endpoints
• Check logs for detailed error messages
• Ensure all prerequisites are installed

🎯 Next Steps

1. Test the APIs using the provided endpoints
2. Explore the documentation at /docs and /swagger
3. Deploy to Azure using the provided scripts
4. Customize the platform for your specific needs
5. Extend with additional data sources and transformations

---

Happy coding! 🚀