Opensubtitles doesn't find any episodes subtitles released after 17 Oct #1168

enchained · 2020-10-23T15:19:40Z

Describe the bug
I turned on the debug mode for this. Logs: https://hastebin.com/weyegawenu.sql
In the logs look for the Star **** and The Spanish ******** show searches. It looks like opensubtitles does connect, but gives only movies results for the first show, and nothing for the second? That's strange cause bazarr downloaded previous episode subs of those shows from opensubtitles with no problem (before I updated the docker image), but now there's no opensubtitles subs in any language in the manual search. And this provider is usually the only option for my language. Please look into it. I hope the info I provided is enough.

To Reproduce
Steps to reproduce the behavior:

Enable opensubtitles provider
Make sure its status is Good
Do a manual search on any common show (chosen subs lang for the show must be both English and Russian)
See opensubtitles missing in the results

Expected behavior
It should show up in the manual search if the subs exist on the site and the same show was matched fine by bazarr to opensubtitles before.

Software (please complete the following information):

Bazarr: v0.9.0.5
Sonarr version v3.0.3.825
OS: docker

morpheus65535 · 2020-10-23T15:49:47Z

It's an issue with imdb number with a leading 0. It's fixed in dev branch. Can you confirm?

enchained · 2020-10-23T17:40:23Z

@morpheus65535 I cloned it in the portainer using linuxserver/bazarr:development instead of :latest, it's now v0.9.0.6. But the manual search for those 2 shows still doesn't have opensubtitles in it, so it's possibly a different issue (or I'm using the wrong dev version?). Also it worked for some other shows both in dev and stable. But not for the ones in the logs.

For the affected shows there's nothing in the logs related to the opensubtitles after those lines:

|urllib3.connectionpool          |Starting new HTTPS connection (1): api.opensubtitles.org:443|
|subliminal_patch.http           |DNS: Falling back to default DNS or IP on api.opensubtitles.org|
|urllib3.connectionpool          |https://api.opensubtitles.org:443 "POST /xml-rpc HTTP/1.1" 200 1858|

For unaffected shows it looks normal like this:

|urllib3.connectionpool          |Starting new HTTPS connection (1): api.opensubtitles.org:443|
|subliminal_patch.http           |DNS: Falling back to default DNS or IP on api.opensubtitles.org|
|urllib3.connectionpool          |https://api.opensubtitles.org:443 "POST /xml-rpc HTTP/1.1" 200 24046|
|subliminal_patch.providers.opensubtitles|Found subtitle <OpenSubtitlesSubtitle 'http://www.opensubtitles.org/en/subtitles/8375484/sid-d-yFx-misZnJ3VZ2qkReOPxvb8a/next-file-1-en' [en:None]> by fulltext|

and a lot of lines like the last one after

morpheus65535 · 2020-10-23T20:50:03Z

It means that OpenSubtitles returned subtitles but they get excluded because they don't really match for one of those reasons: https://github.com/morpheus65535/bazarr/blob/master/libs/subliminal_patch/providers/opensubtitles.py#L305-L322

enchained · 2020-10-24T11:48:28Z

But my settings for both shows are: English, Russian, Hearing-Impaired: False Forced: False
Both episodes have English subtitles on opensubtitles that doesn't look only_foreign or something, I don't see how the code you linked would filter those out (please correct me if it could):
https://www.opensubtitles.org/en/search/sublanguageid-eng/imdbid-10677378
https://www.opensubtitles.org/en/search/sublanguageid-eng/imdbid-10468124

Opensubtitles indeed doesn't say anywhere in my logs that "No subtitles found", that means that response is definitely not empty and is filtered out in the process? Then I can't really explain why those lines come up with nothing, and not sure how I could live-debug this:

23/10/2020 17:21:47|INFO    |subliminal_patch.providers.opensubtitles|Searching subtitles [{'moviehash': '24f4c7e40057be3a', 'moviebytesize': '1963504091', 'sublanguageid': 'eng,eng,rus,rus'}, {'query': 'Star Trek: Discovery', 'season': 3, 'episode': 2, 'sublanguageid': 'eng,eng,rus,rus'}]|
23/10/2020 16:52:32|INFO    |subliminal_patch.providers.opensubtitles|Searching subtitles [{'moviehash': '9d57a2b4641ce8ae', 'moviebytesize': '2367675088', 'sublanguageid': 'eng,eng,rus,rus'}, {'query': 'The Spanish Princess', 'season': 2, 'episode': 2, 'sublanguageid': 'eng,eng,rus,rus'}]|

Also, should the languages duplicate like that? I feel like some of my settings could be corrupted after the update, but I set a lot of things manually, so would like to avoid a reset if possible.

morpheus65535 · 2020-10-25T13:04:45Z

I've run debugger with Star Trek: Discovery S03E02 on OpenSubtitles and strangely, even if I ask for English subtitles, it returns me multiple Czech subtitles for different movies and series... Look like a bug on their side because it works fine for S03E01. The request is properly build with all the required criteria so there's not really anything that can be done on my side. I suggest you add another provider to avoid getting stuck without a subtitles.

Magikarplvl4 · 2020-10-25T21:57:49Z

@morpheus65535 maybe we can open a ticket at OS for investigation? --> https://trac.opensubtitles.org/projects/opensubtitles/report

morpheus65535 · 2020-10-25T23:36:54Z

I've sent a msg to the admin on Slack. Waiting for feedback.

enchained · 2020-10-26T00:35:48Z

Thank you for investigating! I hope they will fix it eventually, cause it is basically the only provider that has Russian subtitles, for the recent titles at least.

morpheus65535 · 2020-10-27T23:35:48Z

@opensubtitles told me that they're going to rerun the indexer to make sure this doesn't happen.

opensubtitles · 2020-10-28T11:43:42Z

I reindexed the db, but the problem persists. It is very strange and I really dont understand. If you can send more examples when this happen, maybe I will know more. (try with season and episode, without it etc etc).

enchained · 2020-10-28T15:19:22Z

@opensubtitles I could try to play around with the queries, but I never worked with XMLRPC before, and can't figure out if there's an easy way to access the api, like some public web sandbox, or a postman import? I found npm api wrapper though, will try setting it up later.
UPD: found the debugger, looks like what I needed.

enchained · 2020-10-29T16:40:50Z

@opensubtitles

I've tested a few of my shows in bazarr and I've experienced new examples of the error. In all of them, previous episodes (if there are any) work, but the one from example and the next ones do not. I think the common denominator is the sub upload date: everything up until (and including) 15 Oct works, everything after 17 Oct does not. Did anything change on your end around that time maybe?

Older shows seem not affected. Only my ongoing shows (every one of them) have that issue for every episode starting from this one:

Warrior, S02E03, upload date 17 Oct
https://www.opensubtitles.org/en/search/sublanguageid-eng/imdbid-10449078

The Spanish Princess, S02E02, upload date 18 Oct
https://www.opensubtitles.org/ru/search/sublanguageid-eng/imdbid-10468124

Pandora (2019), S02E03, upload date 19 Oct
https://www.opensubtitles.org/en/search/sublanguageid-eng/idmovie-975865

Star Trek: Discovery, S03E02, upload date 22 Oct
https://www.opensubtitles.org/ru/search/sublanguageid-eng/imdbid-10677378

Barbarians (2020), S01E02, upload date 23 Oct
https://www.opensubtitles.org/ru/search/sublanguageid-eng/idmovie-980547

The Undoing, S01E01, upload date 26 Oct
https://www.opensubtitles.org/ru/search/sublanguageid-eng/imdbid-9256866

This is Us, S05E01, upload date 28 Oct
https://www.opensubtitles.org/en/search/sublanguageid-eng/imdbid-10322342

I also did some test requests based on those lines from my initial logs that reproduced the error:

Searching subtitles [{'moviehash': '24f4c7e40057be3a', 'moviebytesize': '1963504091', 'sublanguageid': 'eng,eng,rus,rus'}, {'query': 'Star Trek: Discovery', 'season': 3, 'episode': 2, 'sublanguageid': 'eng,eng,rus,rus'}]
Searching subtitles [{'moviehash': '9d57a2b4641ce8ae', 'moviebytesize': '2367675088', 'sublanguageid': 'eng,eng,rus,rus'}, {'query': 'The Spanish Princess', 'season': 2, 'episode': 2, 'sublanguageid': 'eng,eng,rus,rus'}]

I've used npm api wrapper in the end, cause it was easier for me to make its response readable. It says it prioritizes by: Hash + filesize => Filename => IMDBid (+ Season and Episode for TV Series).

I can't try it without season+episode completely, cause the temp agent limits to 5 (I've sent a user agent request today).

My findings (works for both examples):
The search query starts working if I just add 'imdbid' of the series (not episode), '5171438' and '8417308' respectively, to it. And it still works if leave 'imdbid', but remove 'moviehash', 'moviebytesize' and 'query' from my search query. It also doesn't work if among those 4 I leave only one of those, if it's not 'imdbid'.

I have 720p versions, so it's reasonable if my hash and size don't find anything, but if I also try existing bytesize from the webpage, it also doesn't work in a search query for any episode actually, so I'm not sure if I'm doing it right. I'm taking it from here:
https://www.opensubtitles.org/ru/subtitles/8390975/the-spanish-princess-flodden-en 4375380650
https://www.opensubtitles.org/ru/subtitles/8384344/the-spanish-princess-camelot-en 4520051600

I'm not sure how npm api prioritize things (cut off or a chain of requests), but it looks like the 'query' parameter does not work for those new episodes? Can I do any other tests to help?

morpheus65535 · 2020-11-03T16:20:06Z

@opensubtitles any chance of getting this fixed?

enchained · 2020-11-04T19:21:53Z

@morpheus65535 probably they're still looking into it https://forum.opensubtitles.org/viewtopic.php?p=44590#p44590

By the way, I tried to find out if there is some specific reasoning behind bazarr using query instead of imdbid in my subliminal_patch.providers.opensubtitles requests in the first place.

Since the api issue is only with query, and bazarr added imdbId to the series in the latest release (v0.9.0.5), I checked why it could still use query for my searches. I noticed that in bazarr.db the imdbId was missing completely from the table_shows schema for some reason, even if I updated and restarted the container and double-checked the version. Then I updated to the dev image and the version changed to v0.9.0.6, the imdbId appeared in the schema and was filled with ids. But in the search logs it still uses query and finds nothing.

04/11/2020 21:59:40|DEBUG   |root                            |BAZARR is using these video object properties: {
'name': '/volume1/Series/The Undoing/The.Undoing.S01E01.The.Undoing.1080p.10bit.WEBRip.6CH.x265.HEVC-PSA.mkv', 
'source': 'Web', 
'release_group': 'PSA', 
'resolution': '1080p', 
'video_codec': 'H.265', 
'audio_codec': 'AAC', 
'imdb_id': None, 
'hashes': {'opensubtitles': '82d40e6a3bc54bf5'}, 
'size': 680471500, 
'subtitle_languages': set(), 
'original_name': 'The.Undoing.S01E01.The.Undoing.1080p.10bit.WEBRip.6CH.x265.HEVC-PSA.mkv', 
'plexapi_metadata': {}, 
'hints': {
    'title': 'The Undoing', 
    'type': 'episode', 
    'single_value': True, 
    'expected_title': ['The Undoing']}, 
'audio_languages': {'eng'}, 
'external_subtitle_languages': set(), 
'series': 'The Undoing', 
'season': 1, 
'episode': 1, 
'title': 'The Undoing', 
'year': None, 
'original_series': True, 
'tvdb_id': None, 
'series_tvdb_id': 369307, 
'series_imdb_id': 'tt8134470', 
'alternative_series': [], 
'used_scene_name': True, 
'original_path': '/volume1/Series/The Undoing/The.Undoing.S01E01.The.Undoing.1080p.10bit.WEBRip.6CH.x265.HEVC-PSA.mkv', 
'fps': 23.976}|
04/11/2020 21:59:40|INFO    |subliminal_patch.core           |Listing subtitles for <Episode ['The Undoing', 1x1]>|
04/11/2020 21:59:40|INFO    |subliminal_patch.core           |Listing subtitles with provider 'opensubtitles' and languages {<Language [en]>, <Language [en]>}|
04/11/2020 21:59:40|INFO    |subliminal_patch.core           |Initializing provider opensubtitles|
04/11/2020 21:59:40|INFO    |subliminal_patch.providers.opensubtitles|Searching subtitles [{
'moviehash': '82d40e6a3bc54bf5', 
'moviebytesize': '680471500', 
'sublanguageid': 'eng,eng'}, 
{'query': 'The Undoing', 'season': 1, 'episode': 1, 'sublanguageid': 'eng,eng'}]|

I suppose it's because subliminal_patch.providers.opensubtitles just tries to use imdb_id, but the required id is in the series_imdb_id? From my previous api testing, the opensubtitles search requests worked specifically with the series id for the imdbid, not an episode one. Can this be fixed on bazarr end?

morpheus65535 · 2020-11-05T01:42:32Z

I've just pushed some changes to OpenSubtitles provider in dev branch. Seems to be working for me. Can you test it?

enchained · 2020-11-05T14:17:16Z

I had to wait for linuxserver development image to update first, but after I pulled it, it's working now as intended, with imdbid in the request. Thanks for the fix!

opensubtitles · 2020-11-11T18:41:21Z

please check again, I think I have fixed it. It was caused by IDs (integer) overflowing.

morpheus65535 added the wontfix label Oct 25, 2020

TRaSH- mentioned this issue Oct 31, 2020

Bazarr doesn't find non English subs on OpenSubtitles #1176

Closed

GameProgramer mentioned this issue Oct 31, 2020

Opensubtitle will return CS while NL is being requested Diaoul/subliminal#1014

Open

enchained changed the title ~~Opensubtitles doesn't look for tv shows in manual search anymore after an update~~ Opensubtitles doesn't find any episodes subtitles released after 17 Oct Oct 31, 2020

morpheus65535 added awaiting feedback and removed wontfix labels Nov 3, 2020

morpheus65535 added a commit that referenced this issue Nov 5, 2020

Fix for #1168 and #1176

5b831f9

enchained closed this as completed Nov 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Opensubtitles doesn't find any episodes subtitles released after 17 Oct #1168

Opensubtitles doesn't find any episodes subtitles released after 17 Oct #1168

enchained commented Oct 23, 2020

morpheus65535 commented Oct 23, 2020

enchained commented Oct 23, 2020 •

edited

morpheus65535 commented Oct 23, 2020

enchained commented Oct 24, 2020

morpheus65535 commented Oct 25, 2020

Magikarplvl4 commented Oct 25, 2020

morpheus65535 commented Oct 25, 2020

enchained commented Oct 26, 2020

morpheus65535 commented Oct 27, 2020

opensubtitles commented Oct 28, 2020

enchained commented Oct 28, 2020 •

edited

enchained commented Oct 29, 2020 •

edited

morpheus65535 commented Nov 3, 2020

enchained commented Nov 4, 2020

morpheus65535 commented Nov 5, 2020

enchained commented Nov 5, 2020

opensubtitles commented Nov 11, 2020

Opensubtitles doesn't find any episodes subtitles released after 17 Oct #1168

Opensubtitles doesn't find any episodes subtitles released after 17 Oct #1168

Comments

enchained commented Oct 23, 2020

morpheus65535 commented Oct 23, 2020

enchained commented Oct 23, 2020 • edited

morpheus65535 commented Oct 23, 2020

enchained commented Oct 24, 2020

morpheus65535 commented Oct 25, 2020

Magikarplvl4 commented Oct 25, 2020

morpheus65535 commented Oct 25, 2020

enchained commented Oct 26, 2020

morpheus65535 commented Oct 27, 2020

opensubtitles commented Oct 28, 2020

enchained commented Oct 28, 2020 • edited

enchained commented Oct 29, 2020 • edited

morpheus65535 commented Nov 3, 2020

enchained commented Nov 4, 2020

morpheus65535 commented Nov 5, 2020

enchained commented Nov 5, 2020

opensubtitles commented Nov 11, 2020

enchained commented Oct 23, 2020 •

edited

enchained commented Oct 28, 2020 •

edited

enchained commented Oct 29, 2020 •

edited