Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(MovieScraper): Move to job-based movie loading #1163

Merged
merged 2 commits into from May 7, 2023

Conversation

bugwelle
Copy link
Collaborator

@bugwelle bugwelle commented Jan 7, 2021

Just like for TV scrapers. This will allow us to have parallel scraping, parallel searches and fewer race conditions that would be possible if we move scraping to a background process (like changing the scraper's language in the settings during scraping).

Should fix:

@bugwelle bugwelle added this to the v2.8.6 milestone Jan 7, 2021
@bugwelle bugwelle added this to In progress in New Scraper Interfaces via automation Jan 7, 2021
@bugwelle bugwelle added this to In progress in More Scraper API details via automation Jan 7, 2021
@bugwelle bugwelle force-pushed the movie-load branch 2 times, most recently from 50199a7 to daff284 Compare January 16, 2021 17:23
@bugwelle bugwelle modified the milestones: v2.8.6, v2.8.8 Jan 21, 2021
@bugwelle
Copy link
Collaborator Author

bugwelle commented Jan 31, 2021

TODO:

  • Rework CustomMovieScraper
    • Let it create multiple scrape jobs
  • Rework CustomMovieScraper handling in MovieDialog -> if a scraper needs search first, let the user select correct movie
  • Save select scraper language when movie is scraped
  • Scrape Dialog: List scrapers that need title search (doesn't match currently; also on first dialog being shown: no item listed; when changing to other scraper, text not removed)
  • Custom Single Movie Scraper
  • Custom Multi Movie Scraper

@bugwelle bugwelle force-pushed the movie-load branch 2 times, most recently from 43fdd62 to 68ac996 Compare February 3, 2021 16:56
@bugwelle bugwelle mentioned this pull request Feb 24, 2021
@bugwelle bugwelle modified the milestones: v2.8.8, v2.8.10 Apr 26, 2021
@bugwelle bugwelle modified the milestones: v2.8.10, v2.8.14 May 6, 2021
@bugwelle bugwelle modified the milestones: v2.8.14, v2.8.16 Feb 6, 2022
@bugwelle bugwelle modified the milestones: v2.8.16, v2.6.18 Apr 10, 2022
@bugwelle bugwelle force-pushed the movie-load branch 4 times, most recently from 2b300d7 to f992df9 Compare June 6, 2022 12:07
@bugwelle bugwelle force-pushed the movie-load branch 3 times, most recently from ee18a68 to 8d56634 Compare July 21, 2022 16:28
@bugwelle
Copy link
Collaborator Author

bugwelle commented Mar 25, 2023

Ok, so far I think I've implemented everything. Now I need to do some testing:

  • Add scraper tests
  • Manual test
    • Single movie scraping
      • Load all details
        • from the same scraper
        • from different scrapers (custom movie scraper)
      • Load no details (via settings)
    • Multi movie scraping
      • Load all details
        • from the same scraper
        • from different scrapers (custom movie scraper)
      • Load no details (via settings);
        • TODO: Currently still loads items? --> was the case if only one scraper was selected --> fixed

@bugwelle bugwelle force-pushed the movie-load branch 4 times, most recently from c4c321e to 48c0db6 Compare April 27, 2023 19:12
@bugwelle bugwelle force-pushed the movie-load branch 3 times, most recently from 0ab84ad to 62e7e6e Compare May 2, 2023 17:47
@bugwelle bugwelle force-pushed the movie-load branch 3 times, most recently from 9cf82dc to 87546b9 Compare May 7, 2023 11:55
@bugwelle bugwelle marked this pull request as ready for review May 7, 2023 11:55
Just like for TV scrapers, move to a job based movie scraper interface.
This will make it possible (in the future) to have preview windows in
movie search dialogs or parallel scraping (which previously didn't work
properly).
@bugwelle bugwelle merged commit ecdee92 into Komet:master May 7, 2023
New Scraper Interfaces automation moved this from In progress to Done May 7, 2023
More Scraper API details automation moved this from In progress to Done May 7, 2023
@bugwelle bugwelle deleted the movie-load branch May 7, 2023 11:57
@bugwelle bugwelle changed the title [MovieScraper] Refactor and move to job based scrapers refactor(MovieScraper): Move to job-based movie loading May 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

Successfully merging this pull request may close these issues.

None yet

1 participant