Skip to content

stricter tests for broken scraper check and better query#1188

Merged
fbacall merged 2 commits into
ElixirTeSS:masterfrom
DaanVanVugt:bugfix/broken_scraper_check
Dec 4, 2025
Merged

stricter tests for broken scraper check and better query#1188
fbacall merged 2 commits into
ElixirTeSS:masterfrom
DaanVanVugt:bugfix/broken_scraper_check

Conversation

@mikesndrs

@mikesndrs mikesndrs commented Dec 4, 2025

Copy link
Copy Markdown
Contributor

Summary of changes
Fix broken scraper check.
Add more tests for query function.

Checklist

  • I have read and followed the CONTRIBUTING guide.
  • I confirm that I have the authority necessary to make this contribution on behalf of its copyright owner and agree
    to license it to the TeSS codebase under the
    BSD license.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a broken scraper check query that was incorrectly filtering content providers with stale events or materials, and adds comprehensive test coverage for the new query logic. The old query used WHERE clauses with LEFT JOIN which would incorrectly handle providers with only events or only materials, or providers with no content at all. The new implementation correctly identifies providers that have at least one event or material, but all of them have an updated_at timestamp before the cutoff time.

Key Changes

  • Extracted the broken scraper detection logic into a new ContentProvider.with_broken_scrapers class method
  • Fixed the SQL query to properly handle edge cases using HAVING clauses with conditional COUNT checks
  • Added 8 comprehensive tests covering various scenarios including empty providers, events-only, materials-only, mixed content, and title filtering

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
test/mailers/curation_mailer_test.rb Adds new test class with comprehensive test coverage for the with_broken_scrapers method
app/models/content_provider.rb Implements new with_broken_scrapers class method with corrected SQL query logic
app/mailers/curation_mailer.rb Refactors to use the new model method instead of inline query

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread test/mailers/curation_mailer_test.rb Outdated
Comment thread test/mailers/curation_mailer_test.rb Outdated
@fbacall
fbacall merged commit f49cac1 into ElixirTeSS:master Dec 4, 2025
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants