Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

imdb_watchlist plugin broken #2035

Closed
jeroen-s opened this issue Dec 8, 2017 · 25 comments

Comments

Projects
None yet
@jeroen-s
Copy link

commented Dec 8, 2017

Presumably due to IMDB making changes on their end.

Flexget version: 2.11.8

Stacktrace:

2017-12-08 19:34 CRITICAL task          populate_movies_list BUG: Unhandled error in plugin imdb_watchlist: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/flexget/task.py", line 486, in __run_plugin
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flexget/event.py", line 23, in __call__
    return self.func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flexget/utils/cached_input.py", line 148, in wrapped_func
    response = func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flexget/plugins/input/imdb_watchlist.py", line 106, in on_task_input
    total_item_count = int(soup.find('div', class_='desc').get('data-size'))
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
@Arghh

This comment has been minimized.

Copy link

commented Dec 9, 2017

Hi.

I have the same problem on 2.11.9 (had it on an older version aswell). Problems started yesterday. My config was working fine before, I did not making any changes. However I dont get any errors just:

2017-12-09 16:06 VERBOSE imdb_watchlist find-movies No movies were found in imdb list: watchlist 2017-12-09 16:06 WARNING input_cache find-movies Input imdb_watchlist did not return a list, cannot cache. 2017-12-09 16:06 VERBOSE details find-movies Task didn't produce any entries.

also the same with flexget execute --now --tasks "find-movies" --no-cache

Altho I have movies on the list. I also removed all movies and added 1. Still nothing.

Probly something to do with user created lists on imdb side?

@GMBridge

This comment has been minimized.

Copy link

commented Dec 17, 2017

looking at the plugin, and the returned HTML from imdb, it appears imdb changed how the page is structured, so the existing soup.find() in the plugin needs to be updated to reflect the new page layout.

@cvium cvium added the Bug label Dec 18, 2017

@jvlomax

This comment has been minimized.

Copy link

commented Dec 22, 2017

I tried having a look at the code for this yesterday. It looks like the watchlist entries are no longer in the HTML returned when doing the get request, but is injected by some javascript on the webpage. I have a gist of what the js looks like. I have no idea what would be the best way to make the imdb_watchlist plugin make use of this though.

@ranirahn

This comment has been minimized.

Copy link

commented Dec 24, 2017

So compact view is missing on IMDB site. So only way to get it working again is to get all data from a normal view.

thawn added a commit to thawn/Flexget that referenced this issue Dec 30, 2017

@thawn thawn referenced this issue Dec 30, 2017

Merged

imdb_watchlist plugin fixed issue #2035 #2050

0 of 1 task complete
@thawn

This comment has been minimized.

Copy link
Contributor

commented Dec 30, 2017

IMDB changed two things:

  1. on the watchlist page, the list is now dynamically generated using a react javascript widget. This widget uses a JSON api.
  2. on other list pages, the compact view is gone

Accordingly, I wrote a parser for the javascript list of imdb title ids and the JSON api, and updated the html parsing code to be able to handle the detailed list view.

@itsbebbo

This comment has been minimized.

Copy link

commented Dec 31, 2017

Thanks for the update thawn - this seems to be working well at the moment. I assume your parser manages to pull back all pages of the watchlist from IMDB as the javascript on their website is incremental in that it retrieves 40ish entries at a time?

@vkasotis

This comment has been minimized.

Copy link

commented Dec 31, 2017

Thanks Thawn for the update. I have updated flexget to version, 2.11.17, still, no luck. Is there something I can do manually?

Thanks in advance!
V.

@itsbebbo

This comment has been minimized.

Copy link

commented Dec 31, 2017

It isn’t in version .17, you will need to pull down the updated IMDb plugin python file and drop it into your flexget install...

@vkasotis

This comment has been minimized.

Copy link

commented Dec 31, 2017

Thanks itsbebbo! You see a very new to linux/osmc and flexget..... Is there a command I could execute or where I can find new IMDb plugin python file so as to copy it? More importantly, where should I copy it?
I mean, I have folder for flexget like "/home/osmc/.flexget" where I have the config.yml.

Probably these question should seem silly to you, but as I said I'm rather new a this.

Thanks for your time.
V.

@vkasotis

This comment has been minimized.

Copy link

commented Dec 31, 2017

Got it guys! Found a way....

Happy new year to all!

@thawn

This comment has been minimized.

Copy link
Contributor

commented Dec 31, 2017

For the impatient: you can already use the updated plugin by downloading the file imdb_watchlist.py and dropping it into a folder called plugins within the same directory where your config.yml file resides. (e.g. on linux the file will go into ~/.flexget/plugins/ ).

@thawn

This comment has been minimized.

Copy link
Contributor

commented Dec 31, 2017

@itsbebbo: yes, the parser pulls all entries even for very long watchlists (tested with a watchlist containing ~600 entries). All the imdb ids are stored on the page from the start. The parser uses that list of imdb ids to grab all entries at once using the new unofficial imdb API. The same API is also used by the widget on imdb.com to pull the entries ~ 50 at a time.

@itsbebbo

This comment has been minimized.

Copy link

commented Dec 31, 2017

liiight added a commit that referenced this issue Dec 31, 2017

[fix] imdb_watchlist plugin (#2050). Closes #2035
* imdb_watchlist plugin fixed issue #2035

* imdb_watchlist plugin fixed minor formatting issue

* imdb_watchlist plugin removed obsolete debug code

* imdb_watchlist plugin added some verbose log output

* imdb_watchlist plugin removed unused import datetime

* imdb_watchlist plugin: implemented changes suggested by cvium (thanks)

* imdb_watchlist plugin: implemented changes suggested by cvium (thanks)

* imdb_watchlist plugin: implemented changes suggested by cvium (thanks)

* imdb_watchlist plugin: implemented changes suggested by liiight (thanks)
@chewy74

This comment has been minimized.

Copy link

commented Jan 1, 2018

I just updated and imdb_watchlist is still returning with no results

pi@raspberrypi:~ $ flexget execute --tasks IMDB There is a FlexGet process already running for this config, sending execution there. 2018-01-01 11:26 VERBOSE utils.requests IMDB Waiting 4.44 seconds until next request to imdb.com 2018-01-01 11:26 VERBOSE details IMDB Task didn't produce any entries. 2018-01-01 11:26 VERBOSE details IMDB Summary - Accepted: 0 (Rejected: 0 Undecided: 0 Failed: 0)

@thawn

This comment has been minimized.

Copy link
Contributor

commented Jan 1, 2018

@chewy74 I think the fix will be in v2.11.19. Until then, put the fixed file into your plugins directory as I described above.

@chewy74

This comment has been minimized.

Copy link

commented Jan 1, 2018

@thawn thanks for the reply. I'm getting an error message that it can't be registered

pi@raspberrypi:~ $ flexget daemon start -d
2018-01-01 12:25 CRITICAL plugin Error while registering plugin imdb_watchlist. A plugin with the same name is already registered`

@ranirahn

This comment has been minimized.

Copy link

commented Jan 1, 2018

This error does not matter. I have it also but I get my list downloaded. After it will be integrated into new version it will be gone but just ignore that error for now.

@chewy74

This comment has been minimized.

Copy link

commented Jan 1, 2018

OK, but I still don't get my list... What's the cli command to clear the watchlist cache? I will try that

@thawn

This comment has been minimized.

Copy link
Contributor

commented Jan 1, 2018

@chewy74 did you restart the flexget process? I noticed the message flexget already running for this config in your earlier comment. I think you need to restart all flexget processes for the updated plugin to take effect.

@ranirahn

This comment has been minimized.

Copy link

commented Jan 1, 2018

Ah thats why i did not had that problem because i dont run daemon. I run it with cron.

@chewy74

This comment has been minimized.

Copy link

commented Jan 1, 2018

I did stop and restart the daemon @thawn

@liiight

This comment has been minimized.

Copy link
Member

commented Jan 1, 2018

The pr was merged, just update Flexget and restart daemon

@chewy74

This comment has been minimized.

Copy link

commented Jan 2, 2018

@liiight OK, I'm not sure what's going on because my watchlist is not being retrieved. I've tried adding new movies to my watchlist, but it's not retrieving and approving new, or existing entries to my movie list.

Log looks like it's pulling up the queue, but it's not doing anything with the queue.

Debug log
https://pastebin.com/ateXfqd4

@liiight

This comment has been minimized.

Copy link
Member

commented Jan 2, 2018

Then it's unrelated to this issue, please discuss in forums or chat

@cvium

This comment has been minimized.

Copy link
Member

commented Jan 2, 2018

@chewy74 imdb_list != imdb_watchlist.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.