User Sync Service

A Go service that performs user ID synchronization between an ad server and third-party buyers. It supports both iframe and chained redirect sync methods.

Features

• Supports both iframe and chained redirect sync methods
• Redis-based storage for user ID mappings
• Cookie-based internal user ID management
• Concurrent-safe operations with optimized locking
• Graceful shutdown
• JSON-based configuration
• Performance optimizations for high throughput
• URL caching for reduced latency
• Connection pooling for better resource utilization
• Built-in test handler for buyer endpoint simulation

Prerequisites

• Go 1.16 or later
• Redis server (required for storage)

Installing Redis

macOS (using Homebrew):

brew install redis
brew services start redis

Ubuntu/Debian:

sudo apt update
sudo apt install redis-server
sudo systemctl start redis-server

Windows:

1. Download Redis for Windows from https://github.com/microsoftarchive/redis/releases
2. Extract and run redis-server.exe

Verify Redis Installation:

redis-cli ping
# Should return PONG

Installation

1. Clone the repository:

git clone <repository-url>
cd ow_user_sync

2. Install dependencies:

go mod tidy

Configuration

The service is configured using a JSON file (config.json) in cmd/server folder:

{
    "server": {
        "port": "8080",
        "base_url": "http://localhost:8080",
        "read_timeout": 10,
        "write_timeout": 10,
        "idle_timeout": 120
    },
    "redis": {
        "address": "localhost:6379",
        "password": "",
        "db": 0,
        "pool_size": 10
    }
}

Configuration Options

• Server Settings

  • port: Server port (default: "8080")
  • base_url: Base URL for callbacks (default: "http://localhost:8080")
  • read_timeout: Read timeout in seconds (default: 10)
  • write_timeout: Write timeout in seconds (default: 10)
  • idle_timeout: Idle timeout in seconds (default: 120)

• Redis Settings

  • address: Redis server address (default: "localhost:6379")
  • password: Redis password (default: "")
  • db: Redis database number (default: 0)
  • pool_size: Connection pool size (default: 10)

Buyers Configuration

The buyers configuration is located in internal/config/buyers.go. You can modify this file to:

• Add new buyers
• Change buyer URLs
• Test different buyer behaviors

Example buyer configuration:

var buyers = map[string]string{
    "buyer1": "http://localhost:8081",
    "buyer2": "http://localhost:8082",
    "buyer3": "http://localhost:8083",
}

Usage

1. Start the server:

go run cmd/server/main.go

2. The service exposes the following endpoints:

• GET /adserver/sync?type=iframe: Returns HTML with iframes for each buyer
• GET /adserver/sync?type=img: Starts chained redirects through buyers' sync URLs
• GET /adserver/sync-callback: Stores buyer UIDs and continues redirect chain
• GET /adserver/get-uids: Returns JSON of all buyer UIDs for the current user

Test Handler

The service includes a test handler at /test/ that simulates buyer endpoints for testing:

• /test/buyer1: Simulates buyer1's sync endpoint
• /test/buyer2: Simulates buyer2's sync endpoint
• /test/buyer3: Simulates buyer3's sync endpoint

Each test endpoint:

• Accepts a callback URL
• Generates a mock buyer UID
• Redirects to the callback URL with the UID

Important Note: When using the img endpoint, if the user's cookies are deleted, you must add a cache busting parameter (cb) to the next request. This ensures the browser doesn't use a cached response. Example:

# After deleting cookies, use:
curl "http://localhost:8080/adserver/sync?type=img&cb=1234567890"

Example test flow:

# Start with iframe sync
curl "http://localhost:8080/adserver/sync?type=iframe"

# Or start with image sync
curl "http://localhost:8080/adserver/sync?type=img"

Example Response

{
  "userId": "abc123",
  "buyerUids": {
    "ttd": "xyz456",
    "rubicon": "rubi888"
  }
}

Performance Optimizations

1. Concurrency Handling

   • Uses RWMutex for better concurrent read performance
   • Implements connection pooling for Redis
   • Optimizes lock granularity for better throughput

2. Redis Optimizations

   • Uses Redis pipelines for batch operations
   • Implements connection pooling
   • Configures timeouts and retries
   • Uses TTL for automatic cleanup

Scaling Options

Microservices Architecture

The service can be split into microservices for better scalability:

1. User Sync Service

   • Handles sync requests and redirects
   • Manages user ID generation
   • Can be scaled independently based on sync traffic

2. UID Storage Service

   • Manages Redis operations
   • Handles user ID mappings
   • Can be scaled based on storage needs

3. Test Service

   • Provides mock buyer endpoints
   • Can be deployed separately for testing

Benefits of microservices:

• Independent scaling of components
• Better resource utilization
• Easier maintenance and updates
• Improved fault isolation

Edge Caching

For global deployment, consider implementing edge caching:

1. CDN Integration

   • Deploy CDN for static content
   • Cache generated URLs at the edge
   • Implement edge-side includes for dynamic content

2. Regional Deployment

   • Deploy services in multiple regions
   • Use regional Redis instances
   • Implement geo-based routing

3. Load Balancing

   • Use load balancers for traffic distribution
   • Implement health checks
   • Configure auto-scaling based on load

Scaling and Regionalization Proposals

Horizontal Scaling

1. Load Balancing

   • Deploy multiple instances behind a load balancer
   • Use sticky sessions based on user ID for consistent routing
   • Implement health checks for instance monitoring

2. Redis Cluster

   • Use Redis Cluster for data sharding
   • Implement read replicas for better read performance
   • Configure automatic failover

3. Caching Layer

   • Add a distributed cache (e.g., Memcached) for frequently accessed data
   • Implement cache warming strategies
   • Use cache invalidation patterns

Regionalization

1. Multi-Region Deployment

   • Deploy instances in multiple regions
   • Use geo-based DNS routing
   • Implement region-aware Redis replication

2. Data Locality

   • Store user data in the region closest to the user
   • Implement cross-region data synchronization
   • Use eventual consistency for better performance

3. Edge Caching

   • Deploy CDN for static content
   • Cache generated URLs at the edge
   • Implement edge-side includes for dynamic content

Monitoring and Observability

1. Metrics Collection

   • Track request latency
   • Monitor Redis performance
   • Measure cache hit rates

2. Logging

   • Implement structured logging
   • Use log aggregation
   • Track error rates and types

3. Alerting

   • Set up performance thresholds
   • Monitor error rates
   • Track resource utilization

Development

The project follows a clean architecture with the following structure:

ow_user_sync/
├── cmd/
│   └── server/          # Server entry point
├── internal/
│   ├── handlers/        # HTTP handlers
│   ├── storage/         # Storage interface and Redis implementation
│   ├── config/          # Configuration
│   ├── cookies/         # Cookie management
│   └── utils/           # Utility functions
├── go.mod              # Go module file
└── README.md           # This file