Skip to content

feat(scrapper): add PostgreSQL storage and scheduled link updates#4

Merged
666mxvbee merged 7 commits into
mainfrom
scheduler-updates
May 30, 2026
Merged

feat(scrapper): add PostgreSQL storage and scheduled link updates#4
666mxvbee merged 7 commits into
mainfrom
scheduler-updates

Conversation

@666mxvbee
Copy link
Copy Markdown
Owner

No description provided.

@666mxvbee 666mxvbee changed the title Scheduler updates feat(scrapper): add PostgreSQL storage and scheduled link updates May 30, 2026
@666mxvbee
Copy link
Copy Markdown
Owner Author

Summary

This pull request completes the Scrapper persistence and scheduled update-processing stage for LinkTracker.

It adds PostgreSQL-backed storage, SQL migrations, raw SQL and EF Core repository implementations, scheduled GitHub/StackOverflow link checks, batched and parallel processing, and tests based on Testcontainers and WireMock.

Changes

  • Added PostgreSQL persistence for chats, links, subscriptions, and tags.
  • Added SQL migrations executed programmatically with DbUp.
  • Added Docker Compose PostgreSQL setup and environment-based database configuration.
  • Added raw SQL repositories using Npgsql.
  • Added ORM repositories using EF Core.
  • Added configurable repository selection through Database:AccessType.
  • Added tag CRUD API and database-level tag filtering.
  • Added scheduled link update processing.
  • Added configurable batch size and parallelism.
  • Added GitHub update detection for new issues and pull requests.
  • Added StackOverflow update detection for new answers and comments.
  • Added notification sender abstraction from Scrapper to Bot.
  • Added HTTP notification sender implementation.
  • Added README run instructions and project badges.

Testing

  • dotnet restore
  • dotnet build
  • dotnet test

The test suite covers migrations, SQL and ORM repositories, tag CRUD, update preview formatting, GitHub and StackOverflow API handling, batch processing, parallel processing, and isolated failure handling.

@666mxvbee 666mxvbee merged commit 2e7a835 into main May 30, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant