Add subtitles support with Subliminal

[Diaoul]

Subtitles download:

• At post processing
• In the show page ("Download subtitles" button)
• In the manage page (multiple checkboxes)
• Per episode with a dedicated ajax button in the show page
• Periodically in background
  • wait 24h after the first search (which is at postprocessing) for old episodes (> 7 days) then give up
  • wait 4h, 8h, 16h, 24h, 24h and 24h between each search for new episodes (<= 7 days) then give up
• Can scan for subtitles on the show dir using "Re-scan files" button in show page
• Created files (subtitles) receive permissions of their parent folder

Options:

• Multiple languages support
• Signe file download in order of set languages (.srt) or multiple files (.en.srt, .fr.srt)
• Plugins selection and ordering

Limitations:

• Python 2.6+ (subtitles support will be disabled if this condition isn't met, everything else will still work fine)

Misc:

• This line: sys.path.append('lib') can be changed to sys.path.insert(0,"lib") to avoid conflict with installed libraries. You can even tweak all added libraries to add "lib." before every import. Bad hack IMO but if this is the only solution for some users...

Try it!

cd ~
git clone http://github.com/SickBeard-Team/SickBeard.git SickBeard-Diaoul
cd SickBeard-Diaoul
git checkout subliminal
./SickBeard.py

Bug report

Please help me improving this feature, if you encounter a bug, do the following:

1. Edit sickbeard/logger.py to have the following: https://gist.github.com/1256331 (you can even comment every lines that references logger 'sickbeard' in the gist so the log is more readable)
2. Run Sick-Beard in console and reproduce the error
3. Paste the logs somewhere on the web (not here!) like pastebin and report the bug here
4. Also provide python version (python -V)

For encoding issues, add the following informations:

• OS (Linux: uname -a)
• sys.getdefaultencoding()
• sys.getfilesystemencoding()
• locale.getpreferredencoding()

[neoflex]

great addition!

from what I saw, it works great but it didn't download the subtitles during the post processing. I guess that is because the subtitles are available only a few hours after the release of the episode. If it's the reason, maybe you could add an option to rescan automatically to check if the subtitle becomes available.

Thanks for your work.

[Diaoul]

Well, if no subtitles are available, subliminal won't write them ;)
What are the options we need for automatic subtitles check & download?

• Frequency (in the config file, same for every show, editable in configuration page)
• Checkbox in the "Add new show"
• Checkbox in the "Edit" page of the show

This will require interraction with SB's database and some modifications to it. I'll look into it

[no00wa]

As long as you have an frequency pane you can always have it recheck for subtitles so that's awesome.
Following options could be handy;

• Make sure there's a lot of engines to be included ( Opensubtitles.org, Subtitlesource.org, Subscene.com, Addic7ed.com, TheSubDB.com, BierDopje.com etc.) and an option to select which ones you want to use.
• Ability to write to the log if an download fails for subtitles / Sickbeard cant find subtitles
• Have a marker for each Episode ( small icon like or so or just +1 ) if Sickbeard has an subtitle for it and use a marker on the show "Status" on the home page to incidicate that this show uses subtitles. Also an option like the magnifying glass on "Manage" so you can instantly look for a subtitle.
• The ability to check per show if it needs subtitles or not ( I'm dutch for example and don't want subs on shows they make here if i would watch any ) and which language it needs subtitles in ( English default i guess ). Make sure there's an option too when you add the show.
• Would it be possible to get a "list" for available subtitles if you choose to manually search for one ? So for example, i have subtitles for every episode of a show but i'm missing just 1. When trying to download that specific one i always get the wrong one or so, then it would be handy to have a list from all the subtitles Sickbeard finds for the language you specified so you can just choose which one you want.

You might already have a few of these options but i'm just trying to help out :)

[tester22]

There is also the issue of load on the subtitle services, eg for some episodes subtitles will never be available, for those episodes it might make sense to automatically check once a day for x days but not longer.
If that kind of logic isn't implemented and SB tries to find subtitles for all episodes that doesn't have any it might result in hundreds of tries from each SB installation.

[Diaoul]

The major problem here will be as tester22 said: load.
Subtitles providers are doing that for free and downloading subtitles with subliminal doesn't help them (you don't see ads).

Maybe testers22's rule could be more elaborated:

• If recent episode (< 2 day old): check for subtitles every max(x, 24) hours during 10 days
  Where x starts at 1 and double after a failed attempt.
• If not recent episode: check once a day during 3 days

For example: check -> wait 1h -> check -> wait 2h -> check -> wait 4h -> check -> wait 8h -> check -> wait 16h -> check -> wait 24h -> check -> wait 24h -> ...

Maybe we could do x = x² instead of x = 2*x and start at 2h :
For example: check -> wait 2h -> check -> wait 4h -> check -> wait 16h -> wait 24h -> check -> wait 24h -> ...

[bjeanes]

+1

This is a fantastic addition!

[Diaoul]

I'll make another commit this week with the SUBTITLESFINDER thread that works as I said in my comment above or similarly.

Expect another one in at least two weeks which will allow users to search for subtitles manually for an episode and pick the subtitle they want in a list of results.

[Diaoul]

Here it is. Automatic subtitles download

[thezoggy]

well there is only 23 (if i recall correctly) total languages supported in the tvdb.. which are the only languages we support in sickbeard..

per tvdb api module:
# List of language from http://www.thetvdb.com/api/0629B785CE550C8D/languages.xml
# Hard-coded here as it is realtively static, and saves another HTTP request, as
# recommended on http://thetvdb.com/wiki/index.php/API:languages.xml
self.config['valid_languages'] = [
"da", "fi", "nl", "de", "it", "es", "fr","pl", "hu","el","tr",
"ru","he","ja","pt","zh","cs","sl", "hr","ko","en","sv","no"
]

[Diaoul]

OpenSubtitles uses the full list of ISO 639:
http://trac.opensubtitles.org/projects/opensubtitles/wiki/XMLRPC

[thezoggy]

so your saying that one can request languages independently of languages that sb supports?
http://www.opensubtitles.org/addons/export_languages.php

i thought you used the languages that the tvshow was set to...

[Diaoul]

I use languages set in the subtitles configuration page. Whatever the input is (2 characters long only though) I'll search for it in plugins (if available)
Once midgetspy merges your changes for unified UI i'll adapt my subliminal branch to fit.

[Diaoul]

Try it if you want :)

[tester22]

Isn't there a small typo here: ... support or subtitles ...

[Diaoul]

True. I'll correct that

Thanks

[Diaoul]

Well, in fact, it contains the answer to the question "are subtitles supported in Sick-Beard?" not "has user enabled subtitles?"
If you have a better name, i'll take it ;)

[Diaoul]

@midgetspy anything you want modified before merging this?

[neoflex]

having a bug with this:
500 Internal Server Error
The server encountered an unexpected condition which prevented it from fulfilling the request.
Traceback (most recent call last):
File "/usr/local/sbin/sickbeard/cherrypy/_cprequest.py", line 660, in respond
response.body = self.handler()
File "/usr/local/sbin/sickbeard/cherrypy/lib/encoding.py", line 193, in call
self.body = self.oldhandler(_args, *_kwargs)
File "/usr/local/sbin/sickbeard/cherrypy/_cpdispatch.py", line 25, in call
return self.callable(_self.args, *_self.kwargs)
File "/usr/local/sbin/sickbeard/sickbeard/webserve.py", line 1873, in index
return _munge(t)
File "/usr/local/sbin/sickbeard/sickbeard/webserve.py", line 99, in _munge
return unicode(string).encode('utf-8', 'xmlcharrefreplace')
File "/usr/lib/pymodules/python2.7/Cheetah/Template.py", line 1010, in unicode
return getattr(self, mainMethName)()
File "_usr_local_sbin_sickbeard_data_interfaces_default_home_tmpl.py", line 288, in respond
File "/usr/local/sbin/sickbeard/sickbeard/tv.py", line 652, in nextEpisode
sqlResults = myDB.select(query, params)
File "/usr/local/sbin/sickbeard/sickbeard/db.py", line 92, in select
sqlResults = self.action(query, args).fetchall()
File "/usr/local/sbin/sickbeard/sickbeard/db.py", line 71, in action
sqlResult = self.connection.execute(query, args)
File "/usr/lib/python2.7/sqlite3/dbapi2.py", line 66, in convert_timestamp
datepart, timepart = val.split(" ")
ValueError: need more than 1 value to unpack


This error disapears as soon as I go back to the normal, master version

[Diaoul]

Try cloning my fork and run it from scratch, not from another SB repository.
That works fine for me, I'll try an "upgrade" from master -> subliminal on my side

[Diaoul]

@neoflex also, send me the full debug log with that error please

[neoflex]

http://pastebin.com/63xe7Gy6

I'll try with a clean install.

[neoflex]

still the same bug with a clean install from your repo (but keeping my sickbeard.db and config.ini)

[thezoggy]

jesus man use pastebin next time. flooding my phone.

[Diaoul]

@neoflex can't reproduce the bug. If anyone else can please copy debug log and error message to pastebin and link here.
Thanks

[neoflex]

Did you try on linux? I am using ubuntu 11.04. Here are my .db files, so you can try :
http://www.mediafire.com/?4v17tbq36w5glha
http://www.mediafire.com/?f8825onrre3rwoz

I am not sure at all but I had the impression that the bug may happen with tv shows having a special character in their name (such as & )

If you can't reproduce this, I will guess that it is very specific of my configuration somehow.

[Diaoul]

@neoflex IMO here is what you did :

• You switched to my fork, sickbeard.db got updated
• You reverted back to another version of SB with the same sickbeard.db as in my fork and added new shows. As I added an option in database connection, default value for TIMESTAMP changed back to 0 which isn't a valid value
• You tried my fork again with a corrupted database: this causes the bug

Edit your sickbeard.db and remove all fields related to subtitles (3 in tv_episodes, 1 in tv_show). Also change the db version to 9 and run my fork again.

[Diaoul]

In fact, the bug appears after adding a new show. I'll look into it, thanks

[Nyaran]

I have a fork with subtitle support
https://github.com/Pistachitos/Sick-Beard

[jonaskp]

@Nyaran: That is fantastic. Works excactly as expected.
Couple of questions:
When I have downloaded a subtitle, it does not show a flag. It just shows an empty square. Do I need to do something for this to show up?
I have added two subtitle languages (danish and english). If I search for a subtitle, and it finds the english one, the search-button disappears, so I can no longer search manually for a danish one (if it is added at a later time). Is this a mistake?
Is there some way to have it search subtitles for all episodes that are missing subtitles?

And thank you so much for this, I have been missing it for so long.

[Nyaran]

Flags icons and download all missing subtitles are pending features.

About download multiple subtitles, I will check it when I can, but I think that works when I tried it.

[jonaskp]

I can manually search for multiple subtitles under a show, that works. What I meant was if it was possible to have it search all shows for episodes that is missing subtitles once and for all.

[Nyaran]

Yes, I want to add that feature soon ;)

[jonaskp]

Oh, cool.
Btw is there a discussion like this under your own fork?
Seems I might as well ask questions there if possible.
I am not that experienced with github.

[Nyaran]

I'm not sure if github has a section to create discussions, anyway I create this "issue" Pistachitos#1
You can ask there all that you want

[gaetan-petit]

Hey @midgetspy can you please give a status on this development ?
Like I said before, that would be too bad to drop this as @Diaoul himself is so committed to bring this feature to SickBeard.

[jonaskp]

Hi Gaetan,
I have been using Nyarans fork for the last month (aprox.) and it has been working fantastic. Of course it would be great to have subtitle download integrated in the original fork, but for now this is fine by me.

[lilp69]

Hi,
I also have a little trouble with the ascii.
I installed SickBeard on a machine Xubuntu.
And when post-processing must be carried out or looking for subtitles, I get the following message:
2012-11-30 21:04:27.115862 POSTPROCESSER :: Exception generated in thread POSTPROCESSER: ascii
2012-11-30 21:13:44.843274 FINDSUBTITLES :: Exception generated in thread FINDSUBTITLES: ascii

I try as specified above to know what encoding using Python but with the following command "sys.getfilesystemencoding ()" I get the following message: Traceback (most recent call last):
   File "", line 1, in
NameError: name 'sys' is not defined

Here is the link of Pasterbin, for info of Log and Python version
http://pastebin.com/gu4xqsHs

If someone will have a solution.
Thank you in advance.

[JeanMertz]

Can anyone tell me why this hasn't been merged yet? I searched this PR for @midgetspy comments, but couldn't find anything stating why this is closed but not merged?

[thezoggy]

this repo is pretty much dead, there is still code in it we need to put into the main trunk. just one of the many things to do on a very long todo list

[JeanMertz]

@thezoggy thank you for that. Although I have to say, when you talk about "a very long todo list", I can't really think of anything major currently wrong with Sickbeard, besides subtitles for foreign people. But obviously different people have different needs. I just hope we get first-party subtitle support in the near future 👍

[thezoggy]

right now the main things are adding new providers / removing the old ones that are invalid. fix any issues with the new xbmc launch.. finish up the 1080i/p (web-dl/aired/rawhd) stuff. get back to finishing up the custom_torrent branch.. cleanup more of the ui/update libs/etc. then there is a whole laundry list of other things that i personally have planned but will prob never get around to coding

[JeanMertz]

@thezoggy Like I said, different people with different needs 👍 Those features do look interesting indeed. Actually, I got things working using an external subtitle searched. Keep up the good work!

[jonaskp]

For others I can still recommend Nyarans fork for downloading subtitles. I just got back to watching Fringe, and started in the middle of season 3. One click on the "Search subtitles"-button and it downloaded all of the subtitles, and so far (aprox. 30 episodes later) all of them have been working great.

[JeanMertz]

@jonaskp does it differ in other ways besides the subtitle support? And did you try (and succeed) in merging the latest changes from origin back into the fork?

[gaetan-petit]

@thezoggy : I'm sorry but are you suggesting that debbuging for XBMC and minor flaws in the UI might be more important than working on the subtitle feature ?

That may sound impolite but are you, like @midgetspy, from an english speaking country?
What is wrong with you guys, have you ever heard about ethnocentrism?
I have to tell you that you acted like ass by letting @Diaoul working on 36K code lines and not even having the courtesy of having a look on his work.

BTW: @JeanMertz which external tool are you using ?

[JeanMertz]

Right now I use https://code.google.com/p/auto-sub/ it works fine for me for tv shows. Perhaps running https://github.com/Diaoul/subliminal in the background would be even better (especially if you also want movie subtitles)

[Skindred]

@gaetan-petit being rude helps nobody in this matter..

While I do agree I personally would like to see some features be added sooner than others (1080p/subtitle support);
we still have to respect the path the devs take, and we have to respect the priorities they set.

It is their hard work, it are their countless hours that are being poured into this.
Show some more respect, please.
I acknowledge the hard work being done by Diaoul (on subliminal and Synology in general), but the decision to pull lies with @midgetspy and is his choice.
For custom versions, there are forks.

It doesn't help to start flinging words like ethnocentrism around, because we could just aswell start throwing the word 'chauvinist' right back.

I am not a native English speaker, however, I have adapted.
I also speak French, German and Dutch.
I myself prefer English subtitles, because I want to master the language.

All of this being said;
if someone would like to merge all these features, feel free to create a fork, and do the hard work needed for this.

I myself am a long ways away from being a skilled Python dev..
Or I would take a swing at it myself.

[Diaoul]

Subliminal is under some work currently and 0.7.0 will fix some issues and add a lot of features like better caching by using dogpile.cache.
I have not much time lately so I'll do my best for 0.7.0

[jonaskp]

@JeanMertz:I don't know how it differs from the orginal fork of SickBeard. I have very little experience with Github, so merging it with the original fork is not something I have tried :-)
I have not been aware if any new features has been added to SickBeard since I changed to the other fork, but to me subtitles are way more important than a lot of other small changes.

[Greg-F]

Bonjour Diaoul, (puis-je m'exprimer en français ? Au besoin, je traduirai si cela peut-être utile à la communauté).

J'ai installé subliminal sur mon serveur unraid 5.0-rc8a (PLUS), mais j'ai constaté que cette installation n'est pas "persistante". De plus, il s'avère que - contrairement à Couchpotato ou Sabnzbd - je suis contraint de garder ma console telnet active. Dès la fermeture, je perds la connexion avec la page Sickbeard-Diaoul...

La console m'indique justement l'existence d'un dossier root/sickbeard-diaoul, que je suis incapable de retrouver physiquement.

Voilà, où j'en suis après une semaine d'utilisation la combinaison unraid-sabnzbd-couchpotato-sickbeard-Plex, ou plutôt de tentative d'utilisation... Peut-être pourriez-vous m'orienter ? Quoi qu'il en soit, merci de l'attention que vous avez porté à mon message.

[marcofranssen]

Does it also work after renaming the episodes?

[Greg-F]

Hello,

I dont' use Subliminal anymore. It never worked, so I gave up. I now use separately Sickbeard (on unraid) and Sublight (on Windows). It seems easier to me, and it
works fine.

Message du 15/07/13 à 22h47
De : "Marco Franssen"
A : "midgetspy/Sick-Beard"
Copie à : "Greg-F"
Objet : Re: [Sick-Beard] Add subtitles support with Subliminal (#185)

Does it also work after renaming the episodes?

---

Reply to this email directly or view it on GitHub:
#185 (comment)

---

Foot : les dernières infos sur le mercato sont sur la chaine Sports de Voila.fr http://sports.voila.fr/football/mercato-en-direct.html

[marcofranssen]

I want it because it runs on my Synology. At the moment I dropped the postprocessing. I search for subtitles using subliminal first and if I have the subtitles I do the manual postprocessing. However I want this to be automated.

[Diaoul]

You can have subtitles support with mr-orange's fork of Sick-Beard. It uses subliminal and gives me great results.

[marcofranssen]

Yeah, but does this one also support torrents, because I have a branch now
which support torrents.

On Thursday, July 18, 2013, Antoine Bertin wrote:

You can have subtitles support with mr-orange's fork of Sick-Beard. It
uses subliminal and gives me great results.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/185#issuecomment-21198885
.

[Diaoul]

Yes it does have torrent support

[junalmeida]

I have a branch with only midgetspy master and subliminal support at my repo.