Standardize providers #665

medariox · 2016-06-04T18:26:53Z

Goal of this PR is to bring the existing providers to more standardized form, without rewriting any of them.

Planned changes:

Use unicode_literals
Add tracebacks
Use format()
Rename variables to snake case
Clean up imports
Change migration leftovers

This will have to be merged after #658

fernandog · 2016-06-07T16:04:26Z

@medariox doc strings if you are interested:

    def search(self, search_strings, age=0, ep_obj=None):  # pylint: disable=too-many-locals, too-many-branches
+        """
+        Searches indexer using the params in search_strings, either for latest releases, or a string/id search
+        :param search_strings: Search to perform
+        :param age: Not used for this provider
+        :param ep_obj: Not used for this provider / :param ep_obj: episode object (kat,newznab,rarbg, etc)
+
+        :return: A list of items found
         """
+

torrentday:

-    def search(self, search_params, age=0, ep_obj=None):  # pylint: disable=too-many-locals
+    def search(self, search_strings, age=0, ep_obj=None):  # pylint: disable=too-many-locals

p0psicles · 2016-06-09T08:52:37Z

sickbeard/providers/anizb.py

 from bs4 import BeautifulSoup


 class Anizb(NZBProvider):  # pylint: disable=too-many-instance-attributes
-    """Nzb Provider using the open api of anizb.org for daily (rss) and backlog/forced searches"""
+    '''Nzb Provider using the open api of anizb.org for daily (rss) and backlog/forced searches'''


Aah, @medariox docstrings are always in double quotes.

medariox · 2016-06-09T09:47:52Z

@fernandog @p0psicles
Thanks for the feedback. Will fix/add your suggestions in the coming commits.

* Added unicode_literals to GenericProvider * Also adapted all providers, to make use of the future import unicode_literals * Removed the decode()/encode() * Cleaned up some double to single quotes * Added proper exceptions for the provider results items * Some logging cleanup using format() * Now Really remove the .decodes() * Also removed the encodes. * Fixed after a search/replace * Fixed docstrings

p0psicles · 2016-06-10T09:14:45Z

With the new exception handling, this one pops up:

2016-06-10 11:13:16 SEARCHQUEUE-DAILY-SEARCH :: [TokyoToshokan] :: [7f7ad0c] Failed parsing provider. Traceback: 'Traceback (most recent call last):\n  File "/home/opt/medusa/sickbeard/providers/tokyotoshokan.py", line 89, in search\n    title = desc_top.get_text(strip=True)\nAttributeError: \'NoneType\' object has no attribute \'get_text\'\n'
Traceback (most recent call last):
  File "/home/opt/medusa/sickbeard/providers/tokyotoshokan.py", line 89, in search
    title = desc_top.get_text(strip=True)
AttributeError: 'NoneType' object has no attribute 'get_text'

p0psicles · 2016-06-10T13:56:56Z

sickbeard/providers/bitsnoop.py

            for search_string in search_strings[mode]:

                if mode != 'RSS':
-                    logger.log(u"Search string: {}".format(search_string.decode("utf-8")),
+                    logger.log('Search string: {0}'.format(search_string.decode('utf-8')),
                               logger.DEBUG)

                try:
                    search_url = (self.urls['rss'], self.urls['search'] + search_string + '/s/d/1/?fmt=rss')[mode != 'RSS']


shouldn't we do this using urljoin by default now?

Yes, we should, but since there are so many occurrences that would require urljoin and I can't test most of them, I decided to leave the urljoin to another PR (also, this PR is getting way too huge!)

* Remove redundant character escapes * Use augmented assignment * Fix indentation * Use six.iteritems for py3 compatibility

#658) * Moved seeders sorting out of providers code and into the sickbeard/search.py searchProviders() code. * Removed the lambda sort from all providers * Corrected bug introducted in TVCache * Removed bogus condition from GenericProvider. That condition can never be true. * Standardize providers (#665) * Standardize first 10 providers * Small anizb update * Small bluetigers update * Next 10 providers * Added unicode_literals to GenericProvider (#677) * Added unicode_literals to GenericProvider * Also adapted all providers, to make use of the future import unicode_literals * Removed the decode()/encode() * Cleaned up some double to single quotes * Added proper exceptions for the provider results items * Some logging cleanup using format() * Now Really remove the .decodes() * Also removed the encodes. * Fixed after a search/replace * Fixed docstrings * Next 11 providers * Next 11 providers, removed sceneelite * Last 9 providers * Remove sceneelite from init * Renamed all search_params to search_strings * Fix for GFTracker * Fix TNTVillage * Fix HDTorrents * Fix Extratorrent * Fix HDSpace * Use string in SQL with unicode_literals in GenericProvider * Fix BITHDTV * Fix TVChaosUK * Added flag to newznab, for torznab providers. If it's torznab then results are sorted by seeders in search.py. * Improve BitSnoop * Improve Anizb * Improve Bluetigers * Cleanup BTdigg * Improve Hounddawgs * Improve FreshOn * More improvements and cleanups * Fix ThePirateBay * Fix for omgwtfnzb, needed a default value, cause getattr doesn't do that by default. * Add size to freshon, cleanup, fix for tvchaosuk * Improve size parsing code Freshon * Fixes for ExtraTorrent and HDTorrents * Fixed bithdtv * For when it's not getting back the 750px tables. * Fix tokyotoshokan provider errors * Fixed properSearch. * listPropers does an sql, but accessing the row, requires it to use b'' * Added newznab search by search_query fallback, when search by tvdbid does not give back results. * Fix HDTorrents, use urljoin, partial rewrite * Fix rare Zooqle error * Improve HDTorrents, bring back ugly hack * Improve TNTVillage, fix daily search, much more * Fix BIT-HDTV * More standardization * More standardization * Bring back eng releases only option * small fixup * Small tnt change * Update daily search url * Remove freeleech option for MTV * Remove TypeError from connection time out * FIx repeated keyword in dict * More standardization * Standardize method names and order * FIx missed URL join * Standardize string formatting * Last small changes * Change TPB url, update cache to 20 min * More providers (#698) * Remove redundant character escapes * Use augmented assignment * Fix indentation * Use six.iteritems for py3 compatibility * Store hash for torrentproject

medariox added the In progress label Jun 4, 2016

medariox added this to the 0.1.2 milestone Jun 4, 2016

p0psicles force-pushed the move-seeder-sorting branch 2 times, most recently from 094ec1f to f998d60 Compare June 4, 2016 20:14

medariox added 3 commits June 4, 2016 22:20

Standardize first 10 providers

466d4c0

Small anizb update

6324889

Small bluetigers update

f74e486

p0psicles force-pushed the stand-providers branch from efe130c to f74e486 Compare June 4, 2016 20:22

Next 10 providers

54f3531

p0psicles reviewed Jun 9, 2016
View reviewed changes

medariox added 2 commits June 10, 2016 14:05

Next 11 providers

664a5b6

Next 11 providers, removed sceneelite

1b25a1c

medariox closed this Jun 10, 2016

medariox reopened this Jun 10, 2016

medariox added 3 commits June 10, 2016 15:16

Last 9 providers

141b973

Remove sceneelite from init

466cb73

Renamed all search_params to search_strings

afbf7fd

medariox added Concluded Needs review Needs testing Requires testing to make sure it's working as intended and removed In progress labels Jun 10, 2016

p0psicles reviewed Jun 10, 2016
View reviewed changes

medariox added 4 commits June 11, 2016 14:16

Fix for GFTracker

7f43627

Fix TNTVillage

c3727b2

Fix HDTorrents

09d7481

Fix Extratorrent

e84000d

p0psicles force-pushed the stand-providers branch from 8bf0079 to 447eece Compare June 13, 2016 14:08

medariox and others added 18 commits June 14, 2016 19:28

Fix HDTorrents, use urljoin, partial rewrite

297efec

Fix rare Zooqle error

d29244a

Improve HDTorrents, bring back ugly hack

2e1d145

Improve TNTVillage, fix daily search, much more

8a5c4f5

Fix BIT-HDTV

b087fcf

More standardization

2818a27

More standardization

c1e8463

Bring back eng releases only option

a8d94f4

small fixup

67092fc

Small tnt change

9f08ad9

Update daily search url

87d887c

Remove freeleech option for MTV

56289ef

Remove TypeError from connection time out

355a5d5

FIx repeated keyword in dict

973ebef

More standardization

9d12be5

Standardize method names and order

fc73fb6

FIx missed URL join

f9c35aa

Standardize string formatting

752d280

labrys force-pushed the stand-providers branch from c31053a to 752d280 Compare June 18, 2016 14:04

medariox and others added 4 commits June 18, 2016 16:20

Last small changes

51d0a34

Change TPB url, update cache to 20 min

fd80aec

More providers (#698)

c406b3f

* Remove redundant character escapes * Use augmented assignment * Fix indentation * Use six.iteritems for py3 compatibility

Store hash for torrentproject

403fdfd

p0psicles merged commit 4994e6b into move-seeder-sorting Jun 18, 2016

labrys deleted the stand-providers branch June 19, 2016 01:44

labrys restored the stand-providers branch June 19, 2016 14:44

fernandog removed the Needs review label Jan 20, 2017

fernandog removed the Needs testing Requires testing to make sure it's working as intended label Feb 9, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Standardize providers #665

Standardize providers #665

medariox commented Jun 4, 2016 •

edited

fernandog commented Jun 7, 2016 •

edited

p0psicles Jun 9, 2016

medariox commented Jun 9, 2016

p0psicles commented Jun 10, 2016 •

edited

p0psicles Jun 10, 2016

medariox Jun 10, 2016

Standardize providers #665

Standardize providers #665

Conversation

medariox commented Jun 4, 2016 • edited

fernandog commented Jun 7, 2016 • edited

p0psicles Jun 9, 2016

Choose a reason for hiding this comment

medariox commented Jun 9, 2016

p0psicles commented Jun 10, 2016 • edited

p0psicles Jun 10, 2016

Choose a reason for hiding this comment

medariox Jun 10, 2016

Choose a reason for hiding this comment

medariox commented Jun 4, 2016 •

edited

fernandog commented Jun 7, 2016 •

edited

p0psicles commented Jun 10, 2016 •

edited