Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Funimation Redesign #11773

Closed
GMAzrael opened this issue Jan 18, 2017 · 14 comments
Closed

Funimation Redesign #11773

GMAzrael opened this issue Jan 18, 2017 · 14 comments

Comments

@GMAzrael
Copy link

@GMAzrael GMAzrael commented Jan 18, 2017

Please follow the guide below

  • You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
  • Put an x into all the boxes [ ] relevant to your issue (like that [x])
  • Use Preview tab to see how your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2017.01.18. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

  • I've verified and I assure that I'm running youtube-dl 2017.01.18

Before submitting an issue make sure you have:

  • At least skimmed through README and most notably FAQ and BUGS sections
  • Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

  • Bug report (encountered problems with youtube-dl)
  • Site support request (request for adding support for a new site)
  • Feature request (request for a new functionality)
  • Question
  • Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

Funimation is going through a site refresh. While most of the Website seems unuseable, I managed to grab some URLs.

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add -v flag to your command line you run youtube-dl with, copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'-u', u'PRIVATE', u'-p', u'PRIVATE', u'https://www.funimation.com/shows/yu-yu-hakusho/koenma-appears/uncut']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.01.18
[debug] Python version 2.7.12 - Linux-4.4.0-59-generic-x86_64-with-Ubuntu-16.04-xenial
[debug] exe versions: avconv 2.8.10-0ubuntu0.16.04.1, avprobe 2.8.10-0ubuntu0.16.04.1, ffmpeg 2.8.10-0ubuntu0.16.04.1, ffprobe 2.8.10-0ubuntu0.16.04.1, rtmpdump 2.4
[debug] Proxy map: {}
[generic] uncut: Requesting header
[redirect] Following redirect to https://www.funimation.com/shows/yu-yu-hakusho/koenma-appears/uncut/
[generic] uncut: Requesting header
WARNING: Falling back on generic information extractor.
[generic] uncut: Downloading webpage
[generic] uncut: Extracting information
ERROR: Unsupported URL: https://www.funimation.com/shows/yu-yu-hakusho/koenma-appears/uncut/
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/generic.py", line 1720, in _real_extract
    doc = compat_etree_fromstring(webpage.encode('utf-8'))
  File "/usr/local/bin/youtube-dl/youtube_dl/compat.py", line 2526, in compat_etree_fromstring
    doc = _XML(text, parser=etree.XMLParser(target=_TreeBuilder(element_factory=_element_factory)))
  File "/usr/local/bin/youtube-dl/youtube_dl/compat.py", line 2515, in _XML
    parser.feed(text)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1653, in feed
    self._raiseerror(v)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1517, in _raiseerror
    raise err
ParseError: not well-formed (invalid token): line 6, column 409
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 694, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 359, in extract
    return self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/generic.py", line 2555, in _real_extract
    raise UnsupportedError(url)
UnsupportedError: Unsupported URL: https://www.funimation.com/shows/yu-yu-hakusho/koenma-appears/uncut/


If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):

https://www.funimation.com/shows/yu-yu-hakusho/koenma-appears/uncut/?a=1
https://www.funimation.com/shows/tales-of-zestiria-the-x/age-of-chaos/simulcast/?lang=english

Note that youtube-dl does not support sites dedicated to copyright infringement. In order for site support request to be accepted all provided example URLs should not violate any copyrights.


Description of your issue, suggested solution and other information

Parsing the URL for the extractor will need to change. Video ID's are now specified in meta property tags in the head of the video page (I think). The video ID's are now stripped from the URL. Addition of a new video subset (uncut/promotional/official). options are also parsed in via (?lang=english).

@starchivore
Copy link

@starchivore starchivore commented Jan 19, 2017

https://www.funimation.com/shows/yu-yu-hakusho/koenma-appears/uncut/?a=1

That's an easy one, just grab the link to their video player @ line 427:

https://www.funimation.com/player/119804

Line 33 gave us a little something called screenshots as follows:

https://d132fumi6di1wa.cloudfront.net/exp/FunimationStoreFront/1401842/English/1401842_English_8c27a8ba-3e83-e611-80c5-0017a4776c06.m3u8

Simply get rid of /exp first:

https://d132fumi6di1wa.cloudfront.net/FunimationStoreFront/1401842/English/1401842_English_8c27a8ba-3e83-e611-80c5-0017a4776c06.m3u8

Add anywhere between _Layer1 and _Layer9 right before .m3u8

https://d132fumi6di1wa.cloudfront.net/FunimationStoreFront/1401842/English/1401842_English_8c27a8ba-3e83-e611-80c5-0017a4776c06_Layer1.m3u8
.
.
.
https://d132fumi6di1wa.cloudfront.net/FunimationStoreFront/1401842/English/1401842_English_8c27a8ba-3e83-e611-80c5-0017a4776c06_Layer9.m3u8

Finally we could start downloading with ease:

ffmpeg -i https://d132fumi6di1wa.cloudfront.net/FunimationStoreFront/1401842/English/1401842_English_8c27a8ba-3e83-e611-80c5-0017a4776c06_Layer9.m3u8 -c copy yu-yu-hakusho-koenma-appears.mp4

BTW, now we've got geo-restrictions as show @ line 33:

"territories": ["IE", "GB", "US", "CA"]

@kaithar
Copy link

@kaithar kaithar commented Jan 24, 2017

Would be nice if this plugin could be made to support funimationnow.uk as well.

@Starsam80 Starsam80 mentioned this issue Feb 25, 2017
4 of 4 tasks complete
@GMAzrael
Copy link
Author

@GMAzrael GMAzrael commented Feb 27, 2017

@dstftw Funimation Extractor is still broken on this. FYI

@cmcnamara87
Copy link

@cmcnamara87 cmcnamara87 commented Mar 4, 2017

I would love this to be fixed

@julianrichen
Copy link
Contributor

@julianrichen julianrichen commented Mar 10, 2017

I'm looking into fixing this, no promises.

What I found so far, with help from the work done by streamlink, is as follows. We can get the video id either by scaning the page for this badly formated js:

var TITLE_DATA = {
    id: '<video_id>',
    title: '<title>',
    titleSlug: '<slug>',
    seasonNum: <season>,
    episodeNum: <episode>,
    seriesId: <series_id>,
    videoId: null,
    alpha: "<>",
    type: '<episode>'
}
KANE_customdimensions.contentType = 'Episodes and Movies';
KANE_customdimensions.videoType = 'Episode';
KANE_customdimensions.showName = '<series>';
KANE_customdimensions.videoTitle = '<title>';
KANE_customdimensions.videoNumber = '1.0';
KANE_customdimensions.showID = '<show_id>';
KANE_customdimensions.videoID = "<video_id>";
KANE_customdimensions.season = '<season>';
KANE_customdimensions.showLanguage =  '<language>';
KANE_customdimensions.externalAlphaId =  '<idk>';
KANE_customdimensions.showreviewaverageRating = '';

or use the metatags:

<meta property="og:video:url" content="/player/<video_id>">
<meta property="og:video:secure_url" content="/player/<video_id>">

or finding basically any instance of /player/<video_id>

You can then get the series/video info with:

https://prod-api-funimationnow.dadcdigital.com/api/source/catalog/title/experience/<video_id>/

The data is stored inside a nested node of seasons[] -> episodes[] -> (info about video)

You would need to loop it untill you found the slug == to the video url. That gives you all the video info.

You can then get the m3u8 or mp4 via:

https://prod-api-funimationnow.dadcdigital.com/api/source/catalog/video/<video_id>/signed/

m3u8 produces normal multi bitrate/video size stuff and mp4 seems to be 720p. The above signed url can sometimes return:

{
  "errors": [
    {
      "code": 2,
      "detail": "If you are not a member, sign up at www.funimation.com. Otherwise, log in.",
      "title": "Mature content blocked"
    }
  ]
}

If you are not logged and the video is TV-MA, so a parameter or cookie needs to be passed?

I've only looked at dubbed content so idk about getting the subtitles streams.

remitamine added a commit that referenced this issue Apr 1, 2017
@XyaoX
Copy link

@XyaoX XyaoX commented Apr 3, 2017

@starchivore is there anywhere we can use ffmpeg to download subtitles from funimation???

@chainikdn
Copy link

@chainikdn chainikdn commented Apr 26, 2017

Can't login anymore.
Probably because of another re-design, for example the actual login URL is now "https://www.funimation.com/log-in" while the one used in extractor is
_LOGIN_URL = 'http://www.funimation.com/login'
and the form fields names are different.

[Funimation] Logging in as ******
ERROR: Unable to log in; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpi8nwgst1\build\youtube_dl\YoutubeDL.py", line 760, in extract_info
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpi8nwgst1\build\youtube_dl\extractor\common.py", line 428, in extract
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpi8nwgst1\build\youtube_dl\extractor\common.py", line 386, in initialize
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpi8nwgst1\build\youtube_dl\extractor\funimation.py", line 93, in _real_initialize
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpi8nwgst1\build\youtube_dl\extractor\funimation.py", line 90, in _login

@remitamine remitamine closed this May 8, 2017
@kaithar
Copy link

@kaithar kaithar commented May 19, 2017

There's still something not quite right with the extractor... For some reason it's not honouring the language of the URL I give it. For example,
https://www.funimationnow.uk/shows/absolute-duo/blaze/simulcast/?lang=english is what I give it, but it ends up pulling down the Japanese audio version instead. Any ideas?

@kaithar
Copy link

@kaithar kaithar commented May 20, 2017

I've been digging around in the requests stream and it honestly looks like funimation are returning the initial language at random.

The missing bit in the extractor I think is something mentioned in the description provided by @julianrichen ... specifically, the extractor skips the step of querying https://prod-api-funimationnow.dadcdigital.com/api/source/catalog/title/experience/<video_id>/ and jumps straight to requesting the video/<>/signed url. What I think is going on internally, somewhere, is that the page:

  1. is loaded with a language that isn't always going to be right,
  2. loads the series data using title/experience,
  3. looks at the query string and/or user preference json to work out what language it should be playing
  4. loops the series data to find the correct episode's video id
  5. requests the video/<>/signed url for the video id found in step 4 instead of the one from step 2

I'm making progress fixing things. I'm noticing that there's some sources for subtitle files, can someone point me at a good example of how I could return those sources?

@kaithar
Copy link

@kaithar kaithar commented May 20, 2017

So, uh, I've ended up rewriting quite a bit of the code, and I've also got a prototype of a series extractor, but I've run in to two issues and could do with a little help.

Problem 1 is that I can see how separate subs are listed in the experience file, but it looks like most dubs might actually have subs burnt in. It would be handy if someone knows of a video with soft subs to test with. I know it's possible, AoT episode 1 in English has a CC track listed in the textTracks field (srt, vtt, dfxp formats), but it's the only one in the season that has one. Would be nice to fix but I guess it's minor.

Problem 2 is that the geofencing makes it pretty annoying to properly test ... it works for me on funimationnow.uk but I can't really run the test suite or, more of an issue, add tests for the playlist code. Don't suppose I could get some assistance with that?

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented May 21, 2017

Hello @kaithar, could you open a new issue for bugs/missing features for Funimation? Ideas pasted on closed issues is likely to be forgotten.

@kaithar
Copy link

@kaithar kaithar commented May 29, 2017

@yan12125 I was hoping some invested people would be subscribed to this, but yeah, looks like that's for the best.

@GMAzrael
Copy link
Author

@GMAzrael GMAzrael commented May 29, 2017

I'm a beginner at python and have no experience with extractors. :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
10 participants
You can’t perform that action at this time.