Laravel Google Pub/Sub

A comprehensive Google Cloud Pub/Sub integration for Laravel that goes beyond basic queue functionality. This package provides a complete toolkit for building event-driven architectures, microservice communication, and real-time data pipelines.

Features

• Full Laravel Queue Driver - Seamless integration with Laravel's queue system
• Publisher/Subscriber Services - Direct publishing with compression, metadata, and batch support
• Event Integration - Bidirectional event flow between Laravel and Pub/Sub
• Webhook Support - Handle push subscriptions with built-in security
• Schema Validation - JSON Schema validation for message contracts
• Streaming Support - Real-time message processing with StreamingPull
• Multi-Service Architecture - Built for microservice communication
• CloudEvents Support - Industry-standard event formatting with v1.0 compatibility
• Enterprise Ready - Dead letter topics, retry policies, monitoring
• Emulator Support - Local development with Google Cloud Pub/Sub emulator
• Laravel Octane Compatible - Optimized for high-performance applications
• Comprehensive CLI - Rich set of Artisan commands for management

Table of Contents

• Requirements
• Installation
• Basic Configuration
• Quick Start
  • Basic Queue Usage
  • Direct Publishing
  • Event Integration
  • Subscribing to Messages
• Full Documentation
• Performance Tips
• Troubleshooting
• Contributing
• License

Requirements

• PHP 8.4+
• Laravel 11.0+
• Google Cloud Pub/Sub PHP library

Installation

Install the package via Composer:

composer require offload-project/laravel-google-pubsub

Publish the configuration file:

php artisan vendor:publish --provider="OffloadProject\GooglePubSub\PubSubServiceProvider" --tag="config"

Basic Configuration

Environment Variables

Add the following to your .env file:

# Basic Configuration
QUEUE_CONNECTION=pubsub
GOOGLE_CLOUD_PROJECT_ID=your-project-id

# Authentication (choose one method)
# Method 1: Service Account Key File
PUBSUB_AUTH_METHOD=key_file
GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json

# Method 2: Application Default Credentials
PUBSUB_AUTH_METHOD=application_default

# Optional Settings
PUBSUB_DEFAULT_QUEUE=default
PUBSUB_AUTO_CREATE_TOPICS=true
PUBSUB_AUTO_CREATE_SUBSCRIPTIONS=true

Queue Configuration

Update your config/queue.php:

'connections' => [
    'pubsub' => [
        'driver' => 'pubsub',
        'project_id' => env('GOOGLE_CLOUD_PROJECT_ID'),
        'queue' => env('PUBSUB_DEFAULT_QUEUE', 'default'),
        'auth_method' => env('PUBSUB_AUTH_METHOD', 'application_default'),
        'key_file' => env('GOOGLE_APPLICATION_CREDENTIALS'),
        
        // Optional overrides
        'auto_create_topics' => true,
        'auto_create_subscriptions' => true,
        'subscription_suffix' => '-laravel',
        'enable_message_ordering' => false,
    ],
],

Quick Start

1. Basic Queue Usage

Use it exactly like any other Laravel queue:

// Dispatch jobs as normal
ProcessPodcast::dispatch($podcast);

// Dispatch to specific queue (Pub/Sub topic)
ProcessPodcast::dispatch($podcast)->onQueue('audio-processing');

// Your Go microservices can subscribe to the same topic
// Subscription name: audio-processing-go-service

2. Direct Publishing

use OffloadProject\GooglePubSub\Facades\PubSub;

// Publish directly to a topic
PubSub::publish('orders', [
    'order_id' => 123,
    'total' => 99.99,
    'customer_id' => 456
]);

// With attributes and ordering
PubSub::publish('orders', $data, [
    'priority' => 'high',
    'source' => 'api'
], [
    'ordering_key' => 'customer-456'
]);

3. Event Integration

use OffloadProject\GooglePubSub\Attributes\PublishTo;
use OffloadProject\GooglePubSub\Contracts\ShouldPublishToPubSub;

#[PublishTo('orders')]
class OrderPlaced implements ShouldPublishToPubSub
{
    public function __construct(
        public Order $order
    ) {}
    
    public function pubsubTopic(): string
    {
        return 'orders';
    }
    
    public function toPubSub(): array
    {
        return [
            'order_id' => $this->order->id,
            'total' => $this->order->total,
            'customer_id' => $this->order->customer_id,
        ];
    }
}

// This event automatically publishes to the 'orders' topic
event(new OrderPlaced($order));

4. Subscribing to Messages

use OffloadProject\GooglePubSub\Facades\PubSub;

// Create a subscriber
$subscriber = PubSub::subscribe('orders-processor', 'orders');

// Add message handler
$subscriber->handler(function ($data, $message) {
    // Process the order
    processOrder($data);
});

// Start listening
$subscriber->listen();

Full Documentation

• Installation
• Configuration (comprehensive)
• Queue Driver
• Publisher & Subscriber
• Event Integration
• Webhooks (Push Subscriptions)
• Message Schemas and Validation
• CloudEvents
• Artisan Command
• Monitoring & Debugging
• Testing
• Examples

Performance Tips

1. Use Streaming Subscribers for Real-time Processing

Streaming subscribers provide lower latency and better throughput:

// config/pubsub.php
'use_streaming' => true,

// Real-time processing
$subscriber = PubSub::subscribe('high-volume-subscription');
$subscriber->stream(['max_messages_per_pull' => 1000]);

2. Enable Message Ordering Only When Necessary

Ordering reduces throughput, use it selectively:

// Only for specific topics that require order
'topics' => [
    'financial-transactions' => [
        'enable_message_ordering' => true,
    ],
    'analytics' => [
        'enable_message_ordering' => false, // Better performance
    ],
],

3. Set Appropriate Timeouts for Job Processing

Match acknowledgment deadlines to your processing time:

// Quick jobs (< 30 seconds)
'subscriptions' => [
    'quick-jobs' => ['ack_deadline' => 30],
],

// Slow jobs (up to 10 minutes)  
'subscriptions' => [
    'data-processing' => ['ack_deadline' => 600],
],

4. Monitor Dead Letter Topics for Failed Messages

Set up automated monitoring:

# Check dead letter queue size
php artisan pubsub:dead-letters orders --summary

# Reprocess dead letters
php artisan pubsub:dead-letters orders --process

5. Use Compression for Large Payloads

Automatic compression for messages over 1KB:

'message_options' => [
    'compress_payload' => true,
    'compression_threshold' => 1024, // bytes
],

6. Batch Publishing for High Volume

Reduce API calls with batch publishing:

$messages = collect($items)->map(fn($item) => [
    'data' => $item->toArray(),
    'attributes' => ['type' => 'bulk-import'],
]);

PubSub::publishBatch('imports', $messages->toArray());

7. Connection Pooling with Octane

Laravel Octane automatically reuses connections:

// config/octane.php
'warm' => [
    'pubsub', // Warm the Pub/Sub manager
],

Troubleshooting

Connection Errors

Verify your Google Cloud project ID

# Check current project
gcloud config get-value project

# Verify in your .env
grep GOOGLE_CLOUD_PROJECT_ID .env

Check service account permissions

# List current service account permissions
gcloud projects get-iam-policy your-project-id \
  --flatten="bindings[].members" \
  --filter="bindings.members:serviceAccount:your-service-account@*.iam.gserviceaccount.com"

# Required roles:
# - roles/pubsub.publisher (to publish)
# - roles/pubsub.subscriber (to subscribe)
# - roles/pubsub.admin (to create topics/subscriptions)

Ensure Pub/Sub API is enabled

# Check if API is enabled
gcloud services list --enabled | grep pubsub

# Enable if needed
gcloud services enable pubsub.googleapis.com

Message Delivery Issues

Check subscription acknowledgment settings

# View subscription details
gcloud pubsub subscriptions describe orders-laravel

# Update ack deadline if needed
gcloud pubsub subscriptions update orders-laravel --ack-deadline=120

Verify topic and subscription names

# List all topics
php artisan pubsub:topics:list

# List all subscriptions
php artisan pubsub:subscriptions:list

# Test specific subscription
php artisan pubsub:listen orders-laravel --max-messages=1

Monitor dead letter topics

# Check if dead letter is configured
gcloud pubsub subscriptions describe orders-laravel \
  --format="value(deadLetterPolicy)"

# Monitor dead letter messages
php artisan pubsub:listen orders-dead-letter-inspector \
  --topic=orders-dead-letter \
  --max-messages=10

Performance Issues

Adjust max_messages and ack_deadline settings

// In config/queue.php or config/pubsub.php
'pubsub' => [
    'max_messages' => 100,        // Increase for batch processing
    'ack_deadline' => 120,        // Increase for slow jobs
    'wait_time' => 0,            // Reduce for lower latency
],

Use streaming subscribers for high throughput

// For real-time, high-volume processing
$subscriber = PubSub::subscribe('high-volume');
$subscriber->stream([
    'max_messages_per_pull' => 1000,
]);

Consider message batching for publishing

// Instead of individual publishes
collect($events)->each(fn($e) => PubSub::publish('events', $e));

// Use batch publishing
$messages = collect($events)->map(fn($e) => ['data' => $e])->toArray();
PubSub::publishBatch('events', $messages);

Memory Issues

Large Message Handling

// Monitor memory usage
$subscriber->handler(function ($data, $message) {
    $before = memory_get_usage();
    
    // Process large message
    $this->processLargeFile($data);
    
    // Force garbage collection if needed
    if (memory_get_usage() - $before > 50 * 1024 * 1024) { // 50MB
        gc_collect_cycles();
    }
});

Queue Worker Memory Limits

# Set appropriate memory limit
php artisan queue:work pubsub --memory=512 --timeout=300

Authentication Issues

Application Default Credentials

# Set up ADC locally
gcloud auth application-default login

# Verify credentials
gcloud auth application-default print-access-token

Service Account Key File

# Verify key file exists and is valid
ls -la $GOOGLE_APPLICATION_CREDENTIALS

# Test authentication
php artisan pubsub:topics:list

Debugging Tools

Enable Debug Logging

# In .env
PUBSUB_LOG_PUBLISHED=true
PUBSUB_LOG_CONSUMED=true
PUBSUB_LOG_FAILED=true
PUBSUB_LOG_WEBHOOKS=true

Message Inspector

# Inspect messages without processing
php artisan pubsub:inspect orders-laravel --limit=5

Test Publishing

# Test with a simple message
php artisan pubsub:publish orders '{"test":true,"timestamp":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'"}'

Health Check

# Run health check
curl http://your-app.com/health/pubsub

Common Error Messages

"Permission denied"

• Check service account has required Pub/Sub roles
• Verify project ID is correct
• Ensure API is enabled

"Resource not found"

• Topic or subscription doesn't exist
• Enable auto-creation or create manually

"Deadline exceeded"

• Increase ack_deadline for slow processing jobs
• Consider breaking large jobs into smaller tasks

"Invalid message format"

• Check schema validation if enabled
• Verify JSON encoding of messages
• Check for compression issues with large payloads

Testing

./vendor/bin/pest

License

The MIT License (MIT). Please see License File for more information.