[Scrapers] Introduce new TV show scraper interface

Komet · Dec 6, 2020 · 9fa8c84 · 9fa8c84
1 parent 6fdff3d
commit 9fa8c84
Show file tree

Hide file tree

Showing 227 changed files with 11,993 additions and 4,462 deletions.
diff --git a/.gitignore b/.gitignore
@@ -6,6 +6,7 @@ MediaInfoDLL/
 MediaInfo/
 ffmpeg
 *.7z
+*.zip
 
 # Deployment
 bintray.json

diff --git a/.travis.yml b/.travis.yml
@@ -1,7 +1,7 @@
 language: generic
 
 os: linux
-dist: xenial
+dist: bionic
 
 branches:
   only:
@@ -101,7 +101,7 @@ jobs:
   - name: "Linux Qt 5.12 build"
     env: QT=qt512 QT_PPA=qt-5.12.3 CXX=g++ CC=gcc
     os: linux
-    dist: xenial
+    dist: bionic
     install:
       - "travis-ci/install_dependencies.sh"
     before_script:
@@ -119,7 +119,7 @@ jobs:
   - name: "Linux Qt 5.6 build"
     env: QT=qt56 QT_PPA=qt563 CXX=g++ CC=gcc
     os: linux
-    dist: xenial
+    dist: bionic
     install:
       - "travis-ci/install_dependencies.sh"
     before_script:

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,10 +1,14 @@
 # Changelog
 
-## 2.6.8 - *tbd*
+## 2.7.0 - *tbd*
+
+This is the next "big" MediaElch version. This version brings fully rewritten TV scrapers including
+improved user interfaces, better user experience and a new TV show scraper.
 
 MediaElch now requires Qt 5.6 or later. Qt 5.6 was released in 2016 and we highly recommend to update
 to the latest version if your system supports it (this only affects MediaElch's version for Linux
-distributions).
+distributions).  
+This means that Ubuntu 16.04 is no longer supported!
 
 Note: You may need to set "DVD order" in your settings again as the internal settings-key changed.
 
@@ -36,6 +40,16 @@ Note: You may need to set "DVD order" in your settings again as the internal set
  - Movie: Add buttons that take you to the movie's IMDb/TMDb page (#684)
  - TMDb ID field added to UI for movies (#1022)
  - TMDb ID filter: Add "Has TMDb ID"/"No TMDb ID" movie filters (#684)
+ - New TV show search dialog  
+   You can now distinguish between episode and TV show details that you want to load using the
+   selected scraper.
+ - New TV scraper settings  
+   The settings dialog for TV scrapers has been completely redesigned.  It now features the
+   scraper's website, description, terms of service and more so that you know how MediaElch
+   uses the scraper.
+ - New Custom TV scraper  
+   The custom TV scraper has got a new look and feel.  You can select the scrapers you want for
+   episode and TV show details.
  - Advanced Settings: You can now specify a custom stylesheet for MediaElch theme development (#1040)
  - Movie Filter: Add filter by original title (#1057)
 

diff --git a/MediaElch.pro b/MediaElch.pro
@@ -276,14 +276,47 @@ SOURCES += src/main.cpp \
     src/scrapers/movie/VideoBuster.cpp \
     src/scrapers/music/TvTunes.cpp \
     src/scrapers/music/UniversalMusicScraper.cpp \
-    src/scrapers/tv_show/TheTvDb.cpp \
-    src/scrapers/tv_show/thetvdb/ApiRequest.cpp \
-    src/scrapers/tv_show/thetvdb/Cache.cpp \
-    src/scrapers/tv_show/thetvdb/EpisodeLoader.cpp \
-    src/scrapers/tv_show/thetvdb/EpisodeParser.cpp \
-    src/scrapers/tv_show/thetvdb/Search.cpp \
-    src/scrapers/tv_show/thetvdb/ShowLoader.cpp \
-    src/scrapers/tv_show/thetvdb/ShowParser.cpp \
+    src/scrapers/tv_show/ShowIdentifier.cpp \
+    src/scrapers/tv_show/EpisodeIdentifier.cpp \
+    src/scrapers/tv_show/TvScraper.cpp \
+    src/scrapers/tv_show/EpisodeScrapeJob.cpp \
+    src/scrapers/tv_show/ShowScrapeJob.cpp \
+    src/scrapers/tv_show/ShowSearchJob.cpp \
+    src/scrapers/tv_show/SeasonScrapeJob.cpp \
+    src/scrapers/tv_show/ShowMerger.cpp \
+    src/scrapers/tv_show/empty/EmptyTvScraper.cpp \
+    src/scrapers/tv_show/custom/CustomTvScraper.cpp \
+    src/scrapers/tv_show/custom/CustomTvScraperConfig.cpp \
+    src/scrapers/tv_show/custom/CustomEpisodeScrapeJob.cpp \
+    src/scrapers/tv_show/custom/CustomSeasonScrapeJob.cpp \
+    src/scrapers/tv_show/custom/CustomShowScrapeJob.cpp \
+    src/scrapers/tv_show/imdb/ImdbTv.cpp \
+    src/scrapers/tv_show/imdb/ImdbTvApi.cpp \
+    src/scrapers/tv_show/imdb/ImdbTvEpisodeParser.cpp \
+    src/scrapers/tv_show/imdb/ImdbTvEpisodeScrapeJob.cpp \
+    src/scrapers/tv_show/imdb/ImdbTvShowSearchJob.cpp \
+    src/scrapers/tv_show/imdb/ImdbTvShowParser.cpp \
+    src/scrapers/tv_show/imdb/ImdbTvShowScrapeJob.cpp \
+    src/scrapers/tv_show/imdb/ImdbTvSeasonParser.cpp \
+    src/scrapers/tv_show/imdb/ImdbTvSeasonScrapeJob.cpp \
+    src/scrapers/tv_show/thetvdb/TheTvDb.cpp \
+    src/scrapers/tv_show/thetvdb/TheTvDbApi.cpp \
+    src/scrapers/tv_show/thetvdb/TheTvDbEpisodeScrapeJob.cpp \
+    src/scrapers/tv_show/thetvdb/TheTvDbEpisodeParser.cpp \
+    src/scrapers/tv_show/thetvdb/TheTvDbEpisodesParser.cpp \
+    src/scrapers/tv_show/thetvdb/TheTvDbShowSearchJob.cpp \
+    src/scrapers/tv_show/thetvdb/TheTvDbShowScrapeJob.cpp \
+    src/scrapers/tv_show/thetvdb/TheTvDbSeasonScrapeJob.cpp \
+    src/scrapers/tv_show/thetvdb/TheTvDbShowParser.cpp \
+    src/scrapers/tv_show/tmdb/TmdbTv.cpp \
+    src/scrapers/tv_show/tmdb/TmdbTvApi.cpp \
+    src/scrapers/tv_show/tmdb/TmdbTvEpisodeScrapeJob.cpp \
+    src/scrapers/tv_show/tmdb/TmdbTvShowSearchJob.cpp \
+    src/scrapers/tv_show/tmdb/TmdbTvShowParser.cpp \
+    src/scrapers/tv_show/tmdb/TmdbTvEpisodeParser.cpp \
+    src/scrapers/tv_show/tmdb/TmdbTvSeasonParser.cpp \
+    src/scrapers/tv_show/tmdb/TmdbTvShowScrapeJob.cpp \
+    src/scrapers/tv_show/tmdb/TmdbTvSeasonScrapeJob.cpp \
     src/ui/movie_sets/MovieListDialog.cpp \
     src/ui/movie_sets/SetsWidget.cpp \
     src/settings/AdvancedSettings.cpp \
@@ -307,13 +340,15 @@ SOURCES += src/main.cpp \
     src/ui/settings/NetworkSettingsWidget.cpp \
     src/ui/settings/ScraperSettingsWidget.cpp \
     src/ui/settings/TvScraperSettingsWidget.cpp \
+    src/ui/settings/CustomTvScraperSettingsWidget.cpp \
     src/ui/settings/TvShowSettingsWidget.cpp \
     src/ui/small_widgets/AlphabeticalList.cpp \
     src/ui/small_widgets/Badge.cpp \
     src/ui/small_widgets/ClosableImage.cpp \
     src/ui/small_widgets/FilterWidget.cpp \
     src/ui/small_widgets/ImageGallery.cpp \
     src/ui/small_widgets/ImageLabel.cpp \
+    src/ui/small_widgets/LanguageCombo.cpp \
     src/ui/small_widgets/LoadingStreamDetails.cpp \
     src/ui/small_widgets/MediaFlags.cpp \
     src/ui/small_widgets/MessageLabel.cpp \
@@ -336,16 +371,18 @@ SOURCES += src/main.cpp \
     src/ui/tv_show/TvShowFilesWidget.cpp \
     src/ui/support/SupportDialog.cpp \
     src/scrapers/trailer/HdTrailers.cpp \
+    src/ui/tv_show/TvShowCommonWidgets.cpp \
     src/ui/tv_show/TvShowMultiScrapeDialog.cpp \
     src/ui/tv_show/TvShowSearch.cpp \
-    src/ui/tv_show/TvShowSearchEpisode.cpp \
+    src/ui/tv_show/TvShowSearchWidget.cpp \
     src/tv_shows/TvShowUpdater.cpp \
     src/tv_shows/TvShowUtils.cpp \
     src/ui/tv_show/TvShowWidget.cpp \
     src/ui/tv_show/TvShowWidgetEpisode.cpp \
     src/ui/tv_show/TvShowWidgetSeason.cpp \
     src/ui/tv_show/TvShowWidgetTvShow.cpp \
     src/ui/tv_show/TvTunesDialog.cpp \
+    src/tv_shows/EpisodeMap.cpp \
     src/tv_shows/TvShowModel.cpp \
     src/tv_shows/TvShowProxyModel.cpp \
     src/tv_shows/model/TvShowModelItem.cpp \
@@ -397,7 +434,6 @@ HEADERS  += Version.h \
     src/scrapers/concert/ConcertScraperInterface.h \
     src/scrapers/music/MusicScraperInterface.h \
     src/scrapers/movie/MovieScraperInterface.h \
-    src/scrapers/tv_show/TvScraperInterface.h \
     src/scrapers/ScraperInterface.h \
     src/data/Locale.h \
     src/data/Rating.h \
@@ -548,14 +584,47 @@ HEADERS  += Version.h \
     src/scrapers/movie/VideoBuster.h \
     src/scrapers/music/TvTunes.h \
     src/scrapers/music/UniversalMusicScraper.h \
-    src/scrapers/tv_show/TheTvDb.h \
-    src/scrapers/tv_show/thetvdb/ApiRequest.h \
-    src/scrapers/tv_show/thetvdb/Cache.h \
-    src/scrapers/tv_show/thetvdb/EpisodeLoader.h \
-    src/scrapers/tv_show/thetvdb/EpisodeParser.h \
-    src/scrapers/tv_show/thetvdb/Search.h \
-    src/scrapers/tv_show/thetvdb/ShowLoader.h \
-    src/scrapers/tv_show/thetvdb/ShowParser.h \
+    src/scrapers/tv_show/ShowIdentifier.h \
+    src/scrapers/tv_show/EpisodeIdentifier.h \
+    src/scrapers/tv_show/TvScraper.h \
+    src/scrapers/tv_show/EpisodeScrapeJob.h \
+    src/scrapers/tv_show/ShowScrapeJob.h \
+    src/scrapers/tv_show/ShowSearchJob.h \
+    src/scrapers/tv_show/SeasonScrapeJob.h \
+    src/scrapers/tv_show/ShowMerger.h \
+    src/scrapers/tv_show/empty/EmptyTvScraper.h \
+    src/scrapers/tv_show/custom/CustomTvScraper.h \
+    src/scrapers/tv_show/custom/CustomTvScraperConfig.h \
+    src/scrapers/tv_show/custom/CustomSeasonScrapeJob.h \
+    src/scrapers/tv_show/custom/CustomEpisodeScrapeJob.h \
+    src/scrapers/tv_show/custom/CustomShowScrapeJob.h \
+    src/scrapers/tv_show/imdb/ImdbTv.h \
+    src/scrapers/tv_show/imdb/ImdbTvApi.h \
+    src/scrapers/tv_show/imdb/ImdbTvEpisodeParser.h \
+    src/scrapers/tv_show/imdb/ImdbTvEpisodeScrapeJob.h \
+    src/scrapers/tv_show/imdb/ImdbTvShowSearchJob.h \
+    src/scrapers/tv_show/imdb/ImdbTvShowParser.h \
+    src/scrapers/tv_show/imdb/ImdbTvShowScrapeJob.h \
+    src/scrapers/tv_show/imdb/ImdbTvSeasonScrapeJob.h \
+    src/scrapers/tv_show/imdb/ImdbTvSeasonParser.h \
+    src/scrapers/tv_show/thetvdb/TheTvDb.h \
+    src/scrapers/tv_show/thetvdb/TheTvDbApi.h \
+    src/scrapers/tv_show/thetvdb/TheTvDbEpisodeScrapeJob.h \
+    src/scrapers/tv_show/thetvdb/TheTvDbEpisodeParser.h \
+    src/scrapers/tv_show/thetvdb/TheTvDbEpisodesParser.h \
+    src/scrapers/tv_show/thetvdb/TheTvDbShowSearchJob.h \
+    src/scrapers/tv_show/thetvdb/TheTvDbShowScrapeJob.h \
+    src/scrapers/tv_show/thetvdb/TheTvDbSeasonScrapeJob.h \
+    src/scrapers/tv_show/thetvdb/TheTvDbShowParser.h \
+    src/scrapers/tv_show/tmdb/TmdbTv.h \
+    src/scrapers/tv_show/tmdb/TmdbTvApi.h \
+    src/scrapers/tv_show/tmdb/TmdbTvEpisodeScrapeJob.h \
+    src/scrapers/tv_show/tmdb/TmdbTvShowSearchJob.h \
+    src/scrapers/tv_show/tmdb/TmdbTvShowParser.h \
+    src/scrapers/tv_show/tmdb/TmdbTvEpisodeParser.h \
+    src/scrapers/tv_show/tmdb/TmdbTvSeasonParser.h \
+    src/scrapers/tv_show/tmdb/TmdbTvShowScrapeJob.h \
+    src/scrapers/tv_show/tmdb/TmdbTvSeasonScrapeJob.h \
     src/ui/movie_sets/MovieListDialog.h \
     src/ui/movie_sets/SetsWidget.h \
     src/settings/AdvancedSettings.h \
@@ -579,13 +648,15 @@ HEADERS  += Version.h \
     src/ui/settings/NetworkSettingsWidget.h \
     src/ui/settings/ScraperSettingsWidget.h \
     src/ui/settings/TvScraperSettingsWidget.h \
+    src/ui/settings/CustomTvScraperSettingsWidget.h \
     src/ui/settings/TvShowSettingsWidget.h \
     src/ui/small_widgets/AlphabeticalList.h \
     src/ui/small_widgets/Badge.h \
     src/ui/small_widgets/ClosableImage.h \
     src/ui/small_widgets/FilterWidget.h \
     src/ui/small_widgets/ImageGallery.h \
     src/ui/small_widgets/ImageLabel.h \
+    src/ui/small_widgets/LanguageCombo.h \
     src/ui/small_widgets/LoadingStreamDetails.h \
     src/ui/small_widgets/MediaFlags.h \
     src/ui/small_widgets/MessageLabel.h \
@@ -609,21 +680,23 @@ HEADERS  += Version.h \
     src/ui/support/SupportDialog.h \
     src/scrapers/trailer/HdTrailers.h \
     src/scrapers/trailer/TrailerProvider.h \
+    src/ui/tv_show/TvShowCommonWidgets.h \
     src/ui/tv_show/TvShowMultiScrapeDialog.h \
     src/ui/tv_show/TvShowSearch.h \
-    src/ui/tv_show/TvShowSearchEpisode.h \
     src/ui/tv_show/TvShowWidget.h \
     src/ui/tv_show/TvShowWidgetEpisode.h \
     src/ui/tv_show/TvShowWidgetSeason.h \
     src/ui/tv_show/TvShowWidgetTvShow.h \
     src/ui/tv_show/TvTunesDialog.h \
+    src/tv_shows/EpisodeMap.h \
     src/tv_shows/TvShowModel.h \
     src/tv_shows/TvShowProxyModel.h \
     src/tv_shows/model/TvShowModelItem.h \
     src/tv_shows/model/TvShowBaseModelItem.h \
     src/tv_shows/model/TvShowRootModelItem.h \
     src/tv_shows/model/EpisodeModelItem.h \
     src/tv_shows/model/SeasonModelItem.h \
+    src/ui/tv_show/TvShowSearchWidget.h \
     src/tv_shows/TvShowUpdater.h \
     src/tv_shows/TvShowUtils.h \
     src/ui/media_centers/KodiSync.h \
@@ -691,6 +764,7 @@ FORMS    += src/ui/main/MainWindow.ui \
     src/ui/settings/NetworkSettingsWidget.ui \
     src/ui/settings/ScraperSettingsWidget.ui \
     src/ui/settings/TvScraperSettingsWidget.ui \
+    src/ui/settings/CustomTvScraperSettingsWidget.ui \
     src/ui/settings/TvShowSettingsWidget.ui \
     src/ui/small_widgets/FilterWidget.ui \
     src/ui/small_widgets/ImageLabel.ui \
@@ -702,7 +776,7 @@ FORMS    += src/ui/main/MainWindow.ui \
     src/tv_shows/ItemWidgetShow.ui \
     src/ui/tv_show/TvShowMultiScrapeDialog.ui \
     src/ui/tv_show/TvShowSearch.ui \
-    src/ui/tv_show/TvShowSearchEpisode.ui \
+    src/ui/tv_show/TvShowSearchWidget.ui \
     src/ui/tv_show/TvShowWidget.ui \
     src/ui/tv_show/TvShowWidgetEpisode.ui \
     src/ui/tv_show/TvShowWidgetSeason.ui \

diff --git a/docs/contributing/continuous-integration.md b/docs/contributing/continuous-integration.md
@@ -1,5 +1,8 @@
 # Continuous Integration (CI) and Continuous Delivery/Deployment (CD)
 
+TODO:  
+TravisCI has switched to a paid model. We therefore may switch to Jenkins or GitHub actions.
+
 ## What is CI?
 See [Wikipedia](https://en.wikipedia.org/wiki/Continuous_integration). For MediaElch this means that we have one main code branch (`master`) and each code change (i.e. Git commit) is build and tested automatically by a CI service.
 Furthermore every pull request is tested and will only be merged if all tests succeed.

diff --git a/docs/contributing/dependencies.md b/docs/contributing/dependencies.md
@@ -1,18 +1,32 @@
 # MediaElch's Dependencies
 
-*This document is work-in-progress*
-
 MediaElch has a few dependencies. This document describes what and why
 dependencies are used.
 
- - [Qt][qt]
- - [ffmpeg][ffmpeg]
- - [MediaInfoLib][mediainfolib]
- - [Zenlib][zenlib]
- - [Catch2][catch]
+Note that on Linux (not the AppImage, though) dependencies are downloaded
+through the system's package manager so the used versions may be older.
+
+ - [Qt > 5.6][qt]  
+   Qt is the GUI framework upon which MediaElch is build.
+ - [ffmpeg][ffmpeg]  
+   ffmpeg is used to create screenshots of movies. That's it!
+   For all systems except Linux we use the latest version available.
+ - [MediaInfoLib][mediainfolib]  
+   MediaInfo is used to gather video file details like the resolution, etc.
+   For all systems except Linux we use the latest version available.
+ - [Zenlib][zenlib]  
+   ZenLib is a dependency of MediaInfoLib. MediElch uses it for some string
+   conversions that are related to MediaInfo.
+ - [QuaZip][quazip]  
+   A library which allows us to use Minizip with Qt.
+   The latter is also part of Qt so we don't explicitly install it.
+ - [Catch2][catch]  
+   Catch2 is our testing framework.
+   Please refer to our [Testing document](./testing.md) for more details.
 
 [qt]: https://qt.io
 [catch]: https://github.com/catchorg/Catch2
 [ffmpeg]: https://ffmpeg.org/
 [mediainfolib]: https://github.com/MediaArea/MediaInfoLib
 [zenlib]: https://github.com/MediaArea/ZenLib
+[quazip]: https://github.com/stachenov/quazip
diff --git a/docs/contributing/introduction.md b/docs/contributing/introduction.md
@@ -47,8 +47,8 @@ And while I was at it, I created build instructions for different platforms. You
 you know all of your dependencies and don't have to do this but is really helpful for new users.
 
 If you want to ensure that even with newer versions of your software all dependencies are written
-down somewhere, you can create docker images. I've done that for MediaElch as well, see:
-https://github.com/Komet/MediaElch/tree/master/travis-ci/docker
+down somewhere, you can create docker images. I've done that for MediaElch as well, see
+[our Dockerfiles](../../travis-ci/docker/README.md)
 
 
 ## How do I build MediaElch on X?
@@ -71,6 +71,17 @@ That includes unit and integration tests (see [`test/README.md`](../../test/READ
 ## What's the CI setup?
 Please refer to [`continous-integration.md`](continous-integration.md).
 
-...
+
+## When do you require a more recent Qt release?
+From time to time we require a more recent Qt release.  For example
+MediaElch v2.8 requires 5.6 and no longer supports Qt 5.5.  There may be new
+features that make the development a lot easier. In the case of Qt 5.6 this
+was related to network requests.  Previously we had to handle redirections
+(`3xx` responses) manually.  Qt 5.6 does this automatically which made my
+life a lot easier.
+
+On the downside we want to support Ubuntu LTS versions. As of 2020-12-01,
+Ubuntu 16.04 is still supported by Ubuntu but no longer by MediaElch's PPA.
+But please try to support all LTS versions of Ubuntu if possible.
 
 [qt]: https://www.qt.io/
diff --git a/docs/contributing/project-structure.md b/docs/contributing/project-structure.md
@@ -10,3 +10,9 @@
 ## What is this?
 This document describes important design decision that
 were set in the development process of MediaElch.
+
+## Separation of UI and logic
+We aim to separate MediaElch's user interface from the internal logic.
+The latter includes the scraping logic, file searcher and more.
+
+By separating both, we can easier test MediaElch without instantiating a GUI.
diff --git a/docs/contributing/sanitizers.md b/docs/contributing/sanitizers.md
@@ -11,8 +11,8 @@ To build MediaElch with the AddressSanitizer enabled, you can use either
 CMake or QMake.
 
 Note that MediaElch has quite a few memory leaks due to how filters are
-handled. Furthermore there are many false positives. I suggest to disable
-leak detection alltogether for now.
+handled.  Furthermore there are many false positives due to Qt.
+I suggest to disable leak detection altogether for now.
 
 ## CMake