Split parsing #788

labrys · 2016-07-19T07:29:33Z

PR is based on the DEVELOP branch
Don't send big changes all at once. Split up big PRs into multiple smaller PRs that are easier to manage and review
Read contribution guide

Separate the search and parsing functions in an effort to continue standardization of providers and enhancing of search functionality. This also subcategorizes the providers by type.

p0psicles · 2016-07-19T19:08:55Z

@labrys whats your plan of getting this in? As this can't be reviewed.
Only thing we can do, is make sure it's tested with the providers we have.

@duramato you can maybe give this a go for a week or so? Think that's the only way.
I'll remove the needs review, ast that's pretty much impossible.

fernandog · 2016-07-28T14:33:57Z

please respect rule!

Don't send big changes all at once. Split up big PRs into multiple smaller PRs that are easier to manage and review

+5,766 −4,939- Files changed 83

fernandog · 2016-07-29T14:19:52Z

sickbeard/helpers.py


-    except (SocketTimeout, TypeError) as e:


@medariox @labrys sockets timeout its not part of requests exception. you should known better

it will raise a traceback like this:

File "C:\SickRage\Python\lib\httplib.py", line 832, in connect self.timeout, self.source_address) File "C:\SickRage\Python\lib\socket.py", line 575, in create_connection raise err timeout: timed out

fernandog · 2016-07-29T17:13:39Z

need to change all logs from helpers.py

fernandog · 2016-07-30T13:54:54Z

sickbeard/helpers.py

@@ -1429,10 +1428,8 @@ def getURL(url, post_data=None, params=None, headers=None,  # pylint:disable=too

    try:
        resp.raise_for_status()
-    except requests.RequestException as e:
+    except requests.exceptions.RequestException as e:


@medariox shouldn't we add both? requests.RequestException and requests.exceptions ?

one exceptions is for requests itself. other is all other execeptions right?

like sockets, it's an exceptions but not part of Requests because its sockets

I think just requests.exceptions.RequestException should be fine.

fernandog · 2016-08-02T02:35:32Z

sickbeard/providers/__init__.py

@@ -107,4 +106,5 @@ def getProviderClass(provider_id):
        for x in sickbeard.providerList + sickbeard.newznabProviderList + sickbeard.torrentRssProviderList
        if x
    }
-    return providers[provider_id]
+    if providers:
+        return providers.values()[0]


@medariox this change is making all providers in hstory appear with NZB image.

Also why getProviderClass used to return a list and now returns a dict?!
this change will need to be one in all over Medusa code!

Should be fixed now.
getProviderClass never returned a list, it always returned the first element of the list. So that's just the same, it's the first value of the dict. Since it makes very little sense to make a list/dict and return only the first element, I made it just directly return the first object now.

…e reported as ERROR and change other logs formats

TD is a special provider that we are still testing their blocks to scripts

[Rarbg] :: [42fd472] Error 20: No results found to [Rarbg] :: [42fd472] No results found. Code: 20

coveralls · 2016-08-03T19:27:49Z

Coverage increased (+0.02%) to 28.874% when pulling 3d2a90a on split-parsing into cb3ec55 on develop.

ratoaq2 · 2016-08-03T19:29:53Z

Approved

labrys added Enhancement Concluded Needs review Needs testing Requires testing to make sure it's working as intended labels Jul 19, 2016

labrys force-pushed the split-parsing branch from 1954dbc to 61e3e1e Compare July 19, 2016 07:31

p0psicles removed the Needs review label Jul 19, 2016

labrys force-pushed the split-parsing branch 2 times, most recently from eb28292 to db455f2 Compare July 22, 2016 00:03

medariox force-pushed the split-parsing branch from 51ff7ba to 8b495f5 Compare July 29, 2016 13:24

fernandog reviewed Jul 29, 2016
View reviewed changes

medariox added this to the 0.1.3 milestone Jul 29, 2016

fernandog added In progress and removed Concluded labels Jul 29, 2016

fernandog force-pushed the split-parsing branch 2 times, most recently from a9a8580 to f2b2998 Compare July 30, 2016 12:22

fernandog reviewed Jul 30, 2016
View reviewed changes

fernandog force-pushed the split-parsing branch 2 times, most recently from 6c75475 to 3ea784a Compare July 31, 2016 16:54

fernandog modified the milestones: 0.1.3, 0.1.4 Jul 31, 2016

fernandog force-pushed the split-parsing branch from 3ea784a to f7fd9e7 Compare August 1, 2016 02:42

fernandog reviewed Aug 2, 2016
View reviewed changes

labrys and others added 5 commits August 3, 2016 20:43

Split parsing

ed75679

Clean up getURL

38d8e25

Fix #801: 'dict' object has no attribute 'content'

484f1ad

Fix #803: 'NoneType' object is not iterable

21b8a20

Fix #728: 'int' object has no attribute 'get'

60ff959

medariox and others added 21 commits August 3, 2016 20:43

one more

e3e6fda

one more

81c0ee6

Adds back SocketTimeout catch. Its not unknown error and should not b…

bb078f4

…e reported as ERROR and change other logs formats

fixup

d941d3b

fixup2

f919955

Allow user to submit error when response is not what we are expecting

144426a

TD is a special provider that we are still testing their blocks to scripts

fixup3

64982b2

Fix newpct

671547b

Removed SocketTimeout (handled by requests.exceptions.Timeout)

9496def

Fix custom url with ending slash (extratorrent)

ce6dc59

Fix rarbg, reorder imports

79e633c

Small ExtraTorrent improvement

0875d88

Remove unused import

a4990a4

Remove unused import

687aeab

Remove unused import

6420d17

Fix: String statement has no effect

a3afadd

Invert log to display message first. Not all code are errors

5eef4a8

[Rarbg] :: [42fd472] Error 20: No results found to [Rarbg] :: [42fd472] No results found. Code: 20

fixup

3de154c

Fix manual search

5325830

Don't add point to rarbg message, as most errors end with !

a186d70

Bring sense to getProviderClass, reorder imports

4f6c566

medariox force-pushed the split-parsing branch from a2a7a98 to 4f6c566 Compare August 3, 2016 18:43

Updating flake8 whitelist with new paths

3d2a90a

medariox merged commit 26c7e8a into develop Aug 3, 2016

medariox deleted the split-parsing branch August 3, 2016 19:31

fernandog removed the In progress label Jan 20, 2017

fernandog removed the Needs testing Requires testing to make sure it's working as intended label Feb 9, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Split parsing #788

Split parsing #788

labrys commented Jul 19, 2016

p0psicles commented Jul 19, 2016

fernandog commented Jul 28, 2016 •

edited

Loading

fernandog Jul 29, 2016 •

edited

Loading

fernandog commented Jul 29, 2016

fernandog Jul 30, 2016 •

edited

Loading

medariox Jul 30, 2016

fernandog Aug 2, 2016 •

edited

Loading

medariox Aug 2, 2016

coveralls commented Aug 3, 2016

ratoaq2 commented Aug 3, 2016 •

edited

Loading

Split parsing #788

Split parsing #788

Conversation

labrys commented Jul 19, 2016

p0psicles commented Jul 19, 2016

fernandog commented Jul 28, 2016 • edited Loading

fernandog Jul 29, 2016 • edited Loading

Choose a reason for hiding this comment

fernandog commented Jul 29, 2016

fernandog Jul 30, 2016 • edited Loading

Choose a reason for hiding this comment

medariox Jul 30, 2016

Choose a reason for hiding this comment

fernandog Aug 2, 2016 • edited Loading

Choose a reason for hiding this comment

medariox Aug 2, 2016

Choose a reason for hiding this comment

coveralls commented Aug 3, 2016

ratoaq2 commented Aug 3, 2016 • edited Loading

fernandog commented Jul 28, 2016 •

edited

Loading

fernandog Jul 29, 2016 •

edited

Loading

fernandog Jul 30, 2016 •

edited

Loading

fernandog Aug 2, 2016 •

edited

Loading

ratoaq2 commented Aug 3, 2016 •

edited

Loading