Skip to content
This repository was archived by the owner on Jul 16, 2025. It is now read-only.
This repository was archived by the owner on Jul 16, 2025. It is now read-only.

Implement Repositories Table and Background Job for GitHub Data Sync #46

@codersquirrelbln

Description

@codersquirrelbln

Issue Summary
We need to refactor our application to include a repositories table in our database. This table will store GitHub repositories associated with users, allowing us to maintain a consistent and reliable dataset. Additionally, we will implement a background job to ensure that the data in this table stays up-to-date with the latest information from the GitHub API.

Reason for Change
Currently, we fetch repository data directly from the GitHub API every time we need it. While this works, it creates several challenges:

  1. API Rate Limits: Frequent API calls can cause us to hit GitHub's rate limits, leading to delays or failures in fetching data.
  2. Performance: Fetching data from the API on every request can slow down the application and negatively impact user experience.
  3. Consistency: Storing repository data in the database ensures consistency across different parts of the application and allows us to build more robust features.

Proposed Changes
1. Create a repositories Table:

  • Store essential data about each repository, such as:
    id: The unique identifier for the repository.
    full_name: The full name of the repository (e.g., user/repo-name).
    name: The name of the repository.
    owner_login: The GitHub username of the repository owner.
    html_url: The URL to the repository on GitHub.
    description: A brief description of the repository.
    private: A boolean indicating whether the repository is private.
    fork: A boolean indicating whether the repository is a fork.
    created_at, updated_at, pushed_at: Timestamps related to the repository.
    Additional fields like stargazers_count, watchers_count, language, etc.
  1. Modify Associations:
  • Update the User model to have many repositories.
  • Update the Campaign model to belong to a repository.
  • Ensure the Contribution model's relationships remain consistent.
  • Craft associations carefully, as they might have to be destroyed if dependent...
  1. Background Job for Syncing Data:
  • Implement a background job that periodically syncs the repository data in our database with the latest data from GitHub. Question: When should the sync happen and with what frequency? In addition, it might also be a good option to let the user sync manually, in case they added a new repository that they want to include immediately.
    This job should:
  • Fetch the latest repository data for each user.
  • Update existing records in the repositories table.
  • Add new repositories that the user may have created.
  • Handle deletions or changes to repositories.
  1. Refactor Existing Code:
  • Update controllers and helpers to use the repositories table instead of fetching data directly from the GitHub API.
    Example files that will need updates:
  • HomeController: Modify how repository data is fetched and displayed.
  • GithubApiHelper: Refactor to support both direct API calls and database interactions.
  • User: Update to reflect the new association with repositories.
  • Any other relevant models, controllers, or helpers that interact with repository data.

Tasks

  • Create migration for repositories table.
  • Update User, Campaign, and Contribution models with appropriate associations.
  • Implement the background job for syncing GitHub data.
  • Refactor the HomeController, GithubApiHelper, and other related files.
  • Test all changes thoroughly to ensure the application functions as expected.

Expected Outcome

  • Improved performance by reducing API calls.
  • Better consistency and reliability of repository data within the application.
  • Easier extension of features related to repositories and campaigns.

This approach will lay a solid foundation for further development and ensure that our application can handle GitHub data more effectively.

Metadata

Metadata

Labels

databaseimplications for database - structure and otherwiseenhancementNew feature or requestperformance

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions