Support member-only videos on VTM.be #9974

MichaelDeBoey · 2016-07-01T13:51:55Z

I've verified and I assure that I'm running youtube-dl 2016.07.01
At least skimmed through README and most notably FAQ and BUGS sections
Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

Bug report (encountered problems with youtube-dl)
Site support request (request for adding support for a new site)
Question

$ youtube-dl -v http://vtm.be/video?aid=144670
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'http://vtm.be/video?aid=144670']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.07.01
[debug] Python version 2.7.10 - Darwin-15.5.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 3.0.1, ffprobe 3.0.1
[debug] Proxy map: {}
[generic] video?aid=144670: Requesting header
WARNING: Falling back on generic information extractor.
[generic] video?aid=144670: Downloading webpage
[generic] video?aid=144670: Extracting information
ERROR: Unsupported URL: http://vtm.be/video?aid=144670
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/generic.py", line 1481, in _real_extract
    doc = compat_etree_fromstring(webpage.encode('utf-8'))
  File "/usr/local/bin/youtube-dl/youtube_dl/compat.py", line 2524, in compat_etree_fromstring
    doc = _XML(text, parser=etree.XMLParser(target=_TreeBuilder(element_factory=_element_factory)))
  File "/usr/local/bin/youtube-dl/youtube_dl/compat.py", line 2513, in _XML
    parser.feed(text)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1642, in feed
    self._raiseerror(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1506, in _raiseerror
    raise err
ParseError: not well-formed (invalid token): line 70, column 17
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 676, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 343, in extract
    return self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/generic.py", line 2182, in _real_extract
    raise UnsupportedError(url)
UnsupportedError: Unsupported URL: http://vtm.be/video?aid=144670

Single video: http://vtm.be/video/volledige-afleveringen/id/256911617431000
Single video: http://vtm.be/video?aid=144670

Description of your issue, suggested solution and other information

When I try to run youtube-dl http://vtm.be/video?aid=137771 it works like expected.
When I try to run youtube-dl http://vtm.be/video?aid=144670it doesn't.

I think it has something to do with the fact that the first video can be watched by everyone.
It's just a sneak peak of an episode.
The second video is a full episode, so you can only watch it when you're logged in.

So I tried to run youtube-dl with the -u option and provided my username, and after that I provided my password when requested.
But that also didn't work unfortunately.
I keep getting the following log:

[generic] video?aid=144670: Requesting header
WARNING: Falling back on generic information extractor.
[generic] video?aid=144670: Downloading webpage
[generic] video?aid=144670: Extracting information
ERROR: Unsupported URL: http://vtm.be/video?aid=144670

Am I doing something wrong?

I allready found #4318, but that's not really the problem I think, because it works with videos that doesn't need an account to watch (ie: not full episodes)

himanshuraja · 2016-07-01T16:23:15Z

put whole url in quotes " " so that shell can understand .

MichaelDeBoey · 2016-07-01T16:26:55Z

That gives the same result @himanshuraja

TRox1972 · 2016-07-01T16:35:59Z

@MichaelDeBoey Since the site is not fully supported, try to use the --cookies option instead of -u.

MichaelDeBoey · 2016-07-01T16:41:41Z

Still the same result @TRox1972 😞

Is it because it has no specific .py-file within the youtube-dl source code that it doesn't work when you need to log in?

Maybe @yan12125, @dstftw, @jaimeMF or @remitamine can better answer that last question?

TRox1972 · 2016-07-01T17:22:19Z

@MichaelDeBoey Did you follow these instructions? Does the cookie-file contain the cookies you get after login with a web browser?

The site in question does not have an own extractor (.py-file if you will), and the videos and information collected is therefore collected by the generic extractor, which doesn't supported authentication by username. However, if you pass in the cookies you get when logging in to the site via your web browser, the generic extractor may be able to catch the video in question, as it does with 'free-access' videos on the site.

yan12125 · 2016-07-01T17:42:47Z

In some cases cookies works fine, while not always. If this is the case, writing extra codes is inevitable.

MichaelDeBoey · 2016-07-01T18:00:53Z

@TRox1972 No I didn't.
I now tried it the way explained in the instructions, but I keep getting the following error:

file does not look like a Netscape format cookies file

It's because cookies are storred in .sqlite format I think.

yan12125 · 2016-07-01T18:05:47Z

It's because cookies are storred in .sqlite format I think.

FYI: cookies as sqlite are not supported yet. Use cookies exporter to get a plain text file.

himanshuraja · 2016-07-01T18:06:14Z

in question website does not have an own extractor in format that we use as extension .

MichaelDeBoey · 2016-07-01T18:35:04Z

@TRox1972 & @yan12125 just tried it with the cookies (downloaded with the extention like you said) as an option, but it still doesn't work. :/

So I guess this can only be solved by adding a new extractor?

yan12125 · 2016-07-01T18:37:56Z

So I guess this can only be solved by adding a new extractor?

By your results, yes.

MichaelDeBoey · 2016-07-01T18:41:01Z

@yan12125: There's also some valuable info in the Session Cookies and Local Storage I think.
Is there a way to pass these or isn't that supported?

yan12125 · 2016-07-01T18:46:29Z

In terms of video downloading, implementation of session cookies is the same as other cookies. and local storage eventually goes into cookies/URL parameters/post data, etc. Anyway, a new extractor is necessary.

MichaelDeBoey · 2016-07-01T18:48:57Z

Anyway, a new extractor is necessary.

OK that's a pitty :/

I've been looking around and found some information to possibly get the .m3u8-file.
So I think someone can make a new extractor with that information like they did with #7079?
I'm not familiar with Python so... 😞

@yan12125: Do I make a new issue or do I paste it here?

yan12125 · 2016-07-01T19:12:00Z

The most difficult part is login. Public-available videos on vtm.be uses HTML5 <video>, and I guess it holds for private videos, too. The most difficult part is login.

MichaelDeBoey · 2016-07-01T19:36:25Z

@yan12125: I found a website where it explains (in Dutch) how you can get the .m3u8-file for the specific video, when you have login credentials.

You first need to make an Medialaan-account.
Then go to the following link, which will provide you an JSON-file
https://accounts.eu1.gigya.com/accounts.login?APIKey=3_HZ0FtkMW_gOyKlqQzW5_0FHRC7Nd5XpXJZcDdXY4pk5eES2ZWmejRW5egwVm4ug-&sdk=js_6.1&format=json&loginID=[INSERT E-MAIL HERE]&password=[INSERT PASSWORD HERE]
This JSON-file will give you an UID, an UIDSignature & a signatureTimestamp
Then go to the following link, which will provide you with a second JSON-file http://vod.medialaan.io/api/1.0/item/[INSERT VIDEO ID HERE]/video?app_id=vtm_watch&user_network=vtm-sso&UID=[INSERT ENCODED UID HERE]&UIDSignature=[INSERT ENCODED UIDSIGNATURE HERE]&signatureTimestamp=[INSERT SIGNATURETIMESTAMP HERE]
The video-id can be found by taking the id from the link:
http://vtm.be/video/volledige-afleveringen/id/256911617431000 gives 256911617431000 as video-id
I haven't figured out yet how to get the video-id if the shortlink (http://vtm.be/video?aid=144670) is used
The UIDSignature needs to be encoded before using it in the link.
The second JSON-file has an uri, which should be the URL to the .m3u8 playlist-file for that video.

Maybe @TomGijselinck, @lodev, @dstftw or @remitamine can help with implementing this?

MichaelDeBoey · 2016-07-02T19:06:19Z

@yan12125, @dstftw, @remitamine: Is this enough information or do I need to try and get some more?

MichaelDeBoey · 2016-07-02T19:33:11Z

Public-available videos on vtm.be uses HTML5 <video>, and I guess it holds for private videos, too.

@yan12125: about the HTML5 video-tags: I just tried to look into this, but only the public available videos have this. If you want to watch the full episodes (the ones that need an account), you need at least Flash Player 11.2 😞

MichaelDeBoey · 2016-08-15T23:02:24Z

@yan12125, @dstftw & @remitamine: I think I provided enough info to make a new extractor or do I need to provide some extra info?

yan12125 · 2016-08-16T13:03:03Z

That should be enough. Need someone with an account to implement it.

MichaelDeBoey · 2016-08-16T14:20:36Z

@yan12125 you can easily make a new Medialaan account for yourself if you want. :-)
The site's in Dutch, but I think you understand the form-fields?

MichaelDeBoey · 2016-08-16T18:14:05Z

Or I can make you a new one as well @yan12125?

yan12125 · 2016-08-17T08:42:25Z

Sorry, but currently I don't have much time on implementing this. Maybe others can help.

MichaelDeBoey · 2016-08-17T13:59:57Z

No problem @yan12125 :-)
@TomGijselinck is going to give it a try when he has the time, but that's not for the near future unfortunately :/

yan12125 added the request label Jul 1, 2016

yan12125 mentioned this issue Jul 1, 2016

New site request: vtm.be #4318

Closed

yan12125 changed the title ~~Can't figure out how to download this video (full episode) from VTM.be~~ Support member-only videos on VTM.be Jul 1, 2016

yan12125 added the account-needed label Jul 1, 2016

mrBliss mentioned this issue Jan 31, 2017

[vtm] Add extractor for member-only videos on VTM.be #11912

Closed

4 of 8 tasks complete

dstftw closed this in e7a51a4 Mar 17, 2017

ytdl-org / youtube-dl

Support member-only videos on VTM.be #9974

Support member-only videos on VTM.be #9974

MichaelDeBoey commented Jul 1, 2016 •

edited

himanshuraja commented Jul 1, 2016

MichaelDeBoey commented Jul 1, 2016

TRox1972 commented Jul 1, 2016

MichaelDeBoey commented Jul 1, 2016 •

edited

TRox1972 commented Jul 1, 2016 •

edited

yan12125 commented Jul 1, 2016

MichaelDeBoey commented Jul 1, 2016 •

edited

yan12125 commented Jul 1, 2016

himanshuraja commented Jul 1, 2016

MichaelDeBoey commented Jul 1, 2016

yan12125 commented Jul 1, 2016

MichaelDeBoey commented Jul 1, 2016 •

edited

yan12125 commented Jul 1, 2016

MichaelDeBoey commented Jul 1, 2016 •

edited

yan12125 commented Jul 1, 2016

MichaelDeBoey commented Jul 1, 2016 •

edited

MichaelDeBoey commented Jul 2, 2016 •

edited

MichaelDeBoey commented Jul 2, 2016

MichaelDeBoey commented Aug 15, 2016 •

edited

yan12125 commented Aug 16, 2016

MichaelDeBoey commented Aug 16, 2016 •

edited

MichaelDeBoey commented Aug 16, 2016

yan12125 commented Aug 17, 2016

MichaelDeBoey commented Aug 17, 2016

ytdl-org / youtube-dl

Join GitHub today

GitHub is where the world builds software

Support member-only videos on VTM.be #9974

Support member-only videos on VTM.be #9974

Comments

MichaelDeBoey commented Jul 1, 2016 • edited

What is the purpose of your issue?

Description of your issue, suggested solution and other information

himanshuraja commented Jul 1, 2016

MichaelDeBoey commented Jul 1, 2016

TRox1972 commented Jul 1, 2016

MichaelDeBoey commented Jul 1, 2016 • edited

TRox1972 commented Jul 1, 2016 • edited

yan12125 commented Jul 1, 2016

MichaelDeBoey commented Jul 1, 2016 • edited

yan12125 commented Jul 1, 2016

himanshuraja commented Jul 1, 2016

MichaelDeBoey commented Jul 1, 2016

yan12125 commented Jul 1, 2016

MichaelDeBoey commented Jul 1, 2016 • edited

yan12125 commented Jul 1, 2016

MichaelDeBoey commented Jul 1, 2016 • edited

yan12125 commented Jul 1, 2016

MichaelDeBoey commented Jul 1, 2016 • edited

MichaelDeBoey commented Jul 2, 2016 • edited

MichaelDeBoey commented Jul 2, 2016

MichaelDeBoey commented Aug 15, 2016 • edited

yan12125 commented Aug 16, 2016

MichaelDeBoey commented Aug 16, 2016 • edited

MichaelDeBoey commented Aug 16, 2016

yan12125 commented Aug 17, 2016

MichaelDeBoey commented Aug 17, 2016

Essential cookies

Always active

Analytics cookies

MichaelDeBoey commented Jul 1, 2016 •

edited

MichaelDeBoey commented Jul 1, 2016 •

edited

TRox1972 commented Jul 1, 2016 •

edited

MichaelDeBoey commented Jul 1, 2016 •

edited

MichaelDeBoey commented Jul 1, 2016 •

edited

MichaelDeBoey commented Jul 1, 2016 •

edited

MichaelDeBoey commented Jul 1, 2016 •

edited

MichaelDeBoey commented Jul 2, 2016 •

edited

MichaelDeBoey commented Aug 15, 2016 •

edited

MichaelDeBoey commented Aug 16, 2016 •

edited