Skip to content

hillcallum/stars_backend_new

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

98 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Star Map Backend Service

A microservice for managing star objects in a star map application. This service provides APIs for managing stars, users, and user-star relationships, with real-time event streaming.

Architecture

  • Backend: FastAPI (Python)
  • Database: Azure Table Storage
  • Cache: Azure Redis Cache
  • Real-time Events: Server-Sent Events (SSE)
  • Deployment: Azure Container Apps
  • CI/CD: GitHub Actions

Key Features

  • REST API for managing stars and users
  • Real-time updates via Server-Sent Events
  • Caching and rate limiting with Redis
  • Modular and maintainable code structure
  • Health checks and debug endpoints for monitoring
  • Comprehensive configuration system
  • Docker-based development and production environments

Project Structure

├── src/                    # Source code directory
│   ├── api/                # API modules
│   │   ├── admin.py        # Admin-related endpoints
│   │   ├── debug.py        # Debug endpoints
│   │   ├── health.py       # Health check endpoints
│   │   ├── sse.py          # Server-Sent Events endpoints
│   │   ├── sse_publisher.py # SSE publisher utilities
│   │   ├── stars.py        # Stars API endpoints
│   │   └── users.py        # Users API endpoints
│   ├── config/             # Configuration management
│   │   └── settings.py     # Application settings
│   ├── db/                 # Database connections
│   │   ├── azure_tables.py # Azure Table Storage client
│   │   └── redis_cache.py  # Redis cache client
│   ├── dependencies/       # FastAPI dependencies
│   │   └── providers.py    # Dependency injection providers
│   ├── models/             # Pydantic models
│   │   ├── star.py         # Star model
│   │   └── user.py         # User model
│   ├── utils/              # Utility functions
│   ├── main.py             # FastAPI application setup
│   └── run.py              # Application entry point
├── tests/                  # Test directory
│   ├── api/                # API tests
│   ├── db/                 # Database tests
│   ├── models/             # Model tests
│   ├── util/               # Utility tests
│   └── conftest.py         # Test fixtures and configuration
├── scripts/                # Utility scripts
│   ├── migrate.py          # Migration utility
│   ├── cleanup.sh          # Cleanup script
│   └── setup_dev.sh        # Development setup script
├── config/                 # Configuration files
│   ├── .env.example        # Example environment file
│   ├── .env.development    # Development environment variables
│   └── pytest.ini          # pytest configuration
├── docker/                 # Docker-related files
│   ├── Dockerfile          # Main Dockerfile
│   ├── docker-compose.yml  # Main Docker Compose configuration
│   ├── docker-compose.dev.yml # Development Docker Compose
│   ├── docker-compose.prod.yml # Production Docker Compose
│   └── nginx/              # Nginx configuration
├── docs/                   # Documentation
│   ├── AZURE_DEPLOYMENT.md # Azure deployment documentation
│   └── api/                # API documentation
├── .github/                # GitHub workflows and actions
├── .vscode/                # VS Code configuration (optional)
├── probes.yaml             # Container health check configuration for Azure
└── run.py                  # Application entry point (root level for convenience)

Project Reorganization Note

The project has been fully reorganized with a cleaner directory structure. All configuration files and Docker-related files are now in their respective directories:

Configuration Files

  • Environment files (.env.example, .env.development) are now in the config/ directory
  • Test configuration (pytest.ini) is now in the config/ directory

Docker Files

  • All Docker-related files are now in the docker/ directory:
    • docker/Dockerfile
    • docker/docker-compose.yml
    • docker/docker-compose.dev.yml
    • docker/docker-compose.prod.yml

Running with New Structure

  • Use python run.py start to start the application (unchanged)
  • Use python -m pytest -c config/pytest.ini to run tests
  • Use docker compose -f docker/docker-compose.yml up to run with Docker
  • Use docker compose -f docker/docker-compose.dev.yml up for development

CI/CD

  • CI/CD workflows have been updated to use the new file locations

All symlinks that previously provided backward compatibility have been removed to fully commit to the new, cleaner organization structure.

Files Remaining in Root Directory

A few essential files remain in the root directory for specific purposes:

  • run.py - Main entry point for running the application
  • requirements.txt - Python dependencies
  • probes.yaml - Container health check configuration for Azure Container Apps

Local Development

Prerequisites

  • Docker and Docker Compose
  • Python 3.10+
  • Azure Storage Account (or use Azurite for local development)
  • Azure Redis Cache (optional for local development)

Setup

  1. Clone the repository

    git clone <your-repo-url>
    cd stars_backend
  2. Set up your environment variables

    cp config/.env.example config/.env.development
    # Edit config/.env.development with your credentials
  3. Start the application with Docker Compose

    docker compose -f docker/docker-compose.yml up
  4. Access the API at http://localhost:8080

Running without Docker

# Install dependencies
pip install -r requirements.txt

# Run the application in development mode
python run.py dev

Configuration Management

The application uses a comprehensive settings management system based on Pydantic:

  1. Configuration Files:

    • config/.env.example - Template with all available settings
    • config/.env.development - Your local development settings
    • config/.env.{environment} - Environment-specific settings (development, staging, production, test)
  2. Environment-Specific Settings: The system automatically applies appropriate defaults based on your environment:

    • development - Debugging enabled, localhost services
    • staging - Production-like with staging endpoints
    • production - Optimized for production use
    • test - Configuration for running tests

Running Tests

# Install dependencies
pip install -r requirements.txt

# Run tests
pytest -c config/pytest.ini

Deployment

The application is deployed to Azure Container Apps using GitHub Actions.

CI/CD Pipeline

  1. The GitHub Action workflow builds and pushes the Docker image to GitHub Container Registry
  2. The Azure Container App is configured to pull the latest image

Manual Deployment

# Build and push the Docker image
docker buildx build -f docker/Dockerfile --platform linux/amd64 -t ghcr.io/<your-username>/astro-app-db:latest --push .

# Update the Azure Container App
az containerapp update \
  --name starmap-service \
  --resource-group starmap-rg \
  --image ghcr.io/<your-username>/astro-app-db:latest

Environment Variables

The application uses a hierarchical configuration system with prefixed environment variables:

Application Settings

Variable Description Required Default
ENVIRONMENT Deployment environment (development, staging, production, test) No development
PORT Application port No 8080
DEBUG Enable debugging features No false
PROJECT_NAME API name for documentation No Star Map API
VERSION API version No 1.1.0

Azure Storage Settings (prefix: AZURE_STORAGE_)

Variable Description Required Default
CONNECTION_STRING Connection string for Azure Table Storage Yes* -
ACCOUNT_URL URL for Azure Storage account (used with managed identity) Yes* -
USE_MANAGED_IDENTITY Whether to use Azure Managed Identity No false

*Either CONNECTION_STRING or both USE_MANAGED_IDENTITY and ACCOUNT_URL must be provided.

Redis Settings (prefix: REDIS_)

Variable Description Required Default
HOST Redis hostname Yes -
PORT Redis port No 6379
PASSWORD Redis access key No -
SSL Whether to use SSL for Redis No false
CACHE_TTL Default cache TTL in seconds No 300
POPULAR_CACHE_TTL Cache TTL for popular items in seconds No 3600
POPULARITY_THRESHOLD Threshold for considering an item popular No 50
POPULARITY_WINDOW Time window for popularity calculation in seconds No 3600

API Settings (prefix: API_)

Variable Description Required Default
CORS_ORIGINS List of allowed CORS origins in JSON format No ["http://localhost:3000"]
RATE_LIMIT_TIMES Number of requests allowed in the time window No 5
RATE_LIMIT_SECONDS Time window for rate limiting in seconds No 60

Logging Settings (prefix: LOG_)

Variable Description Required Default
LEVEL Logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL) No INFO
FORMAT Log format string No %(asctime)s - %(name)s - %(levelname)s - %(message)s

API Endpoints

Health Checks

  • GET /health - Overall health status
  • GET /health/liveness - Container liveness check
  • GET /health/readiness - Application readiness check

Stars

  • GET /stars - List all stars
  • POST /stars - Create a new star
  • GET /stars/{star_id} - Get a specific star
  • POST /stars/{star_id}/like - Like a star
  • DELETE /stars/{star_id} - Delete a star
  • GET /stars/active - Get all active stars
  • GET /stars/popular - Get popular stars

Users

  • GET /users - List all users
  • POST /users - Create a new user
  • GET /users/{user_id} - Get a specific user
  • PUT /users/{user_id} - Update a user
  • DELETE /users/{user_id} - Delete a user
  • GET /users/{user_id}/stars - Get stars for a user

Debug Endpoints

  • GET /debug/table-info - Get information about Azure Tables
  • GET /debug/active-stars - Diagnose issues with active stars
  • POST /debug/add-test-star - Add a test star for debugging

Real-time Events (SSE)

  • GET /events/stars/stream - Stream of real-time star events
  • GET /events/users/stream - Stream of real-time user events

Server-Sent Events

The application supports real-time updates through Server-Sent Events (SSE) for both stars and users:

Star Events

// Connect to the stars stream
const evtSource = new EventSource('/events/stars/stream');

// Handle incoming events
evtSource.onmessage = (event) => {
  const data = JSON.parse(event.data);
  console.log(data);
  // data.type will be 'create', 'update', or 'delete'
  // data.data contains the star information
};

User Events

// Connect to the users stream
const evtSource = new EventSource('/events/users/stream');

// Handle incoming events
evtSource.onmessage = (event) => {
  const data = JSON.parse(event.data);
  console.log(data);
  // data.type will be 'create', 'update', or 'delete'
  // data.data contains the user information
};

Troubleshooting

Common Issues

  1. Authentication Failures

    • Verify your connection strings and credentials in environment variables
    • Check if managed identity is properly configured (if using)
  2. Container Startup Issues

    • Check logs with az containerapp logs show --name starmap-service --resource-group starmap-rg
    • Verify environment variables are properly set
  3. Local Development Issues

    • Ensure Docker is running
    • Check .env.development file contains valid credentials
    • Make sure ports 8080, 6379, and 10000-10002 are available
  4. SSE Connection Issues

    • Verify your client supports SSE
    • Check for CORS configuration if accessing from a different domain
    • Ensure the server can maintain long-lived connections

License

[Your License]

codecov

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Packages