FireSync - GitHub to Firestore Sync Agent

A Java-based data synchronization agent that automatically pulls GitHub issues from repositories and stores them in Firebase Firestore. Built with scalability, maintainability, and error handling in mind.

Features

• Automated GitHub Issue Sync: Fetches the most recent issues from any GitHub repository
• Firebase Firestore Integration: Stores issues as documents in Firestore collections
• Incremental Sync: Only syncs new/updated issues since the last sync to minimize API calls
• Duplicate Prevention: Automatically detects and skips existing issues
• Configurable Scheduling: Optional scheduled sync with configurable intervals
• Environment Support: Multi-environment configuration (dev/staging/prod)
• Robust Error Handling: Retry logic for network failures and API rate limits
• Comprehensive Logging: Detailed logging for monitoring and debugging

Requirements

• Java 11+
• Maven 3.6+
• Firebase Project with Firestore enabled
• GitHub Personal Access Token (optional, for private repositories)

🛠️ Installation

1. Clone the Repository

git clone <repository-url>
cd firesync

2. Build the Project

mvn clean package

This creates a standalone JAR file at target/github-firestore-sync-1.0.0.jar.

Configuration

1. Firebase Setup

1. Create a Firebase project at Firebase Console
2. Enable Firestore Database
3. Generate a service account key:
   • Go to Project Settings → Service Accounts
   • Click "Generate new private key"
   • Save the JSON file as src/main/resources/firebase-service-account.json

2. Application Configuration

The application uses environment-specific configuration files:

Base Configuration (application.properties)

# GitHub Configuration
github.api.base.url=https://api.github.com
github.token=

# Repository Configuration
github.repo.owner=facebook
github.repo.name=react

# Firebase Configuration
firebase.project.id=your-firebase-project-id
firebase.service.account.path=src/main/resources/firebase-service-account.json

# Sync Configuration
sync.issues.limit=5

# Scheduler Configuration
sync.schedule.enabled=false
sync.schedule.interval.minutes=10

Environment-Specific Overrides

• application-dev.properties - Development environment
• application-staging.properties - Staging environment
• application-prod.properties - Production environment

3. Environment Variables

Set the environment using the ENV variable:

export ENV=prod  # or dev, staging

Usage

Command Line Options

java -jar target/github-firestore-sync-1.0.0.jar [options]

Options:

• -o, --repo-owner <owner> - GitHub repository owner
• -n, --repo-name <name> - GitHub repository name
• -h, --help - Show help message

Examples:

# Sync Facebook React repository
java -jar target/github-firestore-sync-1.0.0.jar --repo-owner facebook --repo-name react

# Sync Microsoft VSCode repository
java -jar target/github-firestore-sync-1.0.0.jar -o microsoft -n vscode

# Use production environment
ENV=prod java -jar target/github-firestore-sync-1.0.0.jar

Scheduled Sync

Enable scheduled sync by setting sync.schedule.enabled=true in your configuration:

sync.schedule.enabled=true
sync.schedule.interval.minutes=10

When enabled, the application runs continuously and syncs at the specified interval.

Data Structure

GitHub Issue Model

Each GitHub issue is stored with the following fields:

{
  "id": 123456789,
  "title": "Issue title",
  "state": "open|closed",
  "html_url": "https://github.com/owner/repo/issues/123",
  "created_at": "2023-10-15T10:30:00Z",
  "updated_at": "2023-10-15T14:45:00Z"
}

Firestore Collection

Issues are stored in collections named: {owner}_{repo}_issues

Example: facebook_react_issues

Sync State Model

The application tracks sync progress in a sync_state collection. Each repository has its own sync state document with the following structure:

{
  "lastSyncTimestamp": "2023-10-15T14:45:00Z",
  "lastSyncedIssueId": 123456789,
  "lastSyncedIssueUpdatedAt": "2023-10-15T12:30:00Z",
  "updatedAt": "2023-10-15T14:45:00Z"
}

Fields:

• lastSyncTimestamp - When the last sync operation completed
• lastSyncedIssueId - ID of the most recently synced issue
• lastSyncedIssueUpdatedAt - When the most recent issue was last updated on GitHub
• updatedAt - When this sync state document was last modified

Firestore Collections

Issues Collection: {owner}_{repo}_issues

• Example: facebook_react_issues

Sync State Collection: sync_state

• Document ID: {owner}_{repo}_sync_state
• Example: facebook_react_sync_state

Architecture

Core Components

• GitHubFirestoreSync - Main application class and orchestrator
• GitHubService - Handles GitHub API interactions
• FirestoreService - Manages Firestore operations
• SyncStateService - Tracks sync state for incremental updates
• SyncScheduler - Handles scheduled execution
• RetryUtil - Implements retry logic for failed operations

Key Features

1. Incremental Sync: Only fetches issues updated since the last sync
2. Duplicate Prevention: Checks existing documents before inserting
3. Error Recovery: Automatic retries with exponential backoff
4. Connection Testing: Validates GitHub and Firebase connections before sync
5. State Management: Tracks sync progress and last processed issue

Testing

Run the test suite:

mvn test

Test Coverage

• GitHubServiceTest - Tests GitHub API integration
• RetryUtilTest - Tests retry mechanism

Logging

The application uses SLF4J with Logback for logging. Logs are written to:

• logs/firesync.log - Main application log
• logs/firesync.{date}.log - Daily rotated logs

Log Levels

• INFO - General application flow and sync progress
• WARN - Non-critical issues and configuration warnings
• ERROR - Sync failures and critical errors
• DEBUG - Detailed debugging information

Development

Project Structure

src/
├── main/
│   ├── java/com/firesync/
│   │   ├── config/          # Configuration classes
│   │   ├── exceptions/      # Custom exceptions
│   │   ├── models/          # Data models
│   │   ├── scheduler/       # Scheduling logic
│   │   ├── services/        # Core business logic
│   │   └── utils/           # Utility classes
│   └── resources/
│       ├── application*.properties  # Configuration files
│       ├── firebase-service-account.json
│       └── logback.xml      # Logging configuration
└── test/
    └── java/com/firesync/   # Test classes

Adding New Sync Targets

The application is designed to support multiple sync targets. To add a new target:

1. Implement the SyncTarget interface
2. Add configuration for the new target
3. Update the main application to use the new target

Extending Issue Fields

To sync additional GitHub issue fields:

1. Update the GitHubIssue model
2. Modify the FirestoreService to handle new fields
3. Update tests accordingly

Troubleshooting

Common Issues

1. Firebase Authentication Error

   • Verify service account JSON file path
   • Check Firebase project ID configuration
   • Ensure Firestore is enabled in Firebase console

2. GitHub API Rate Limiting

   • Add GitHub personal access token for higher rate limits
   • Check current rate limit status in logs

3. Network Connectivity

   • Verify internet connection
   • Check firewall settings
   • Review proxy configuration if applicable

Debug Mode

Enable debug logging by setting the log level:

<!-- In logback.xml -->
<logger name="com.firesync" level="DEBUG"/>

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Support

For issues and questions:

1. Check the troubleshooting section
2. Review application logs
3. Open an issue on GitHub with detailed information