Skip to content

Releases: lukaszj321/github-data-sync-service

Release list

GitHub Data Sync Service v0.2.0

GitHub Data Sync Service v0.2.0 Latest
Latest

Choose a tag to compare

View all tags
@lukaszj321 lukaszj321 released this 17 Jul 21:13
v0.2.0
19dd85a

GitHub Data Sync Service v0.2.0

v0.2.0 turns the project from a repository-registration foundation into a working issues synchronization service.

Highlights

  • Synchronizes GitHub issues for registered public repositories.
  • Runs synchronization in a separate worker process backed by PostgreSQL jobs.
  • Reads GitHub Issues API pages through validated rel="next" pagination.
  • Filters pull requests returned by the issues endpoint.
  • Stores issues idempotently using stable GitHub identifiers.
  • Distinguishes created, updated, unchanged, skipped, and fetched records.
  • Reschedules rate-limited jobs without holding database locks or sleeping for long periods.
  • Recovers stale running jobs after an expired worker heartbeat.
  • Isolates unexpected failures to a single job so the worker process remains available.
  • Preserves previously committed pages when a later page fails.
  • Avoids duplicate active jobs for the same repository and resource.

API

New endpoints in 0.2.0:

  • POST /repositories/{repository_id}/sync
  • GET /sync-jobs
  • GET /sync-jobs/{job_id}
  • GET /repositories/{repository_id}/issues

Existing repository and health endpoints remain available.

Synchronization Flow

  1. A client creates an issues synchronization job.
  2. The worker claims the job from PostgreSQL.
  3. The worker requests one GitHub API page without holding a database transaction.
  4. Issues and job statistics for that page are committed atomically.
  5. Pull requests are counted as skipped and are not stored as issues.
  6. The process continues through the validated rel="next" URL.
  7. The job finishes as completed, failed, or rate_limited.

Reliability

  • PostgreSQL partial unique index prevents duplicate active jobs.
  • Each page is stored in a short transaction.
  • Retried jobs restart from the first page and rely on idempotent upserts.
  • Rate limits use Retry-After, X-RateLimit-Reset, or a configured fallback.
  • Unexpected job failures are rolled back and safely recorded.
  • Stale worker locks can be recovered.
  • Worker and API containers run as non-root users.

Quality

The release is validated with:

  • Ruff linting and formatting checks.
  • Strict mypy checks.
  • Unit tests with branch coverage above 85%.
  • PostgreSQL integration tests.
  • Alembic upgrade and downgrade validation.
  • Optional live GitHub API tests.
  • A complete Docker end-to-end issue synchronization smoke test.
  • Wheel and source distribution builds.
  • Installation and CLI version checks from the built wheel.
  • Docker non-root runtime verification.
  • GitHub Actions for both branch and release tag pushes.

Limitations

  • Public GitHub repositories only.
  • No ETag or incremental synchronization.
  • No persistent page checkpoint or resume from a page number.
  • No automatic pruning of locally stored issues.
  • No dedicated synchronization for pull requests, commits, releases, or workflow runs.
  • No labels, comments, milestones, or assignee normalization.
  • No OAuth.
  • No frontend.
  • No Redis, Celery, Kafka, Kubernetes, or cloud deployment.
  • Not intended as a complete GitHub analytics platform.
Assets 2
Loading

GitHub Data Sync Service v0.1.0

GitHub Data Sync Service v0.1.0

Choose a tag to compare

View all tags
@lukaszj321 lukaszj321 released this 17 Jul 17:06
v0.1.0
0db47d4

GitHub Data Sync Service 0.1.0

GitHub Data Sync Service 0.1.0 introduces the backend foundation for reliable synchronization of public GitHub engineering data.

This milestone focuses on repository registration and validation rather than full resource synchronization.

Highlights

  • Validates public repositories through the live GitHub REST API.
  • Stores repository metadata idempotently in PostgreSQL.
  • Uses the stable GitHub repository ID to support repository renames and transfers.
  • Provides FastAPI endpoints for registering, listing, and reading repositories.
  • Includes a PostgreSQL-backed job queue foundation using FOR UPDATE SKIP LOCKED.
  • Runs the API, database migrations, worker, and PostgreSQL as separate Docker Compose services.
  • Protects optional GitHub tokens from logs and user-facing errors.
  • Includes deterministic unit tests, PostgreSQL integration tests, and an opt-in live GitHub API test.

API

Implemented endpoints:

  • POST /repositories
  • GET /repositories
  • GET /repositories/{repository_id}
  • GET /health
  • GET /ready

Quality

The release is validated with:

  • Ruff linting and formatting checks.
  • Strict mypy checks.
  • Unit tests with branch coverage above 85%.
  • PostgreSQL integration tests.
  • Alembic migration checks.
  • Wheel and source distribution builds.
  • Installation from the built wheel.
  • Docker image build and non-root runtime verification.
  • GitHub Actions.

Current Scope

Version 0.1.0 does not yet synchronize issues, pull requests, commits, releases, or workflow runs.

The worker and sync_jobs table provide the execution foundation for the next milestone, but resource synchronization is deliberately outside this release.

Limitations

  • Public GitHub repositories only.
  • No issues synchronization yet.
  • No pagination or ETag support yet.
  • No OAuth.
  • No frontend.
  • No Redis, Celery, Kafka, Kubernetes, or cloud deployment.
  • Not intended as a production GitHub analytics platform.
Loading