Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RTLXL.nl stopped working #4115

Closed
MatrixView opened this issue Nov 5, 2014 · 12 comments
Closed

RTLXL.nl stopped working #4115

MatrixView opened this issue Nov 5, 2014 · 12 comments

Comments

@MatrixView
Copy link

@MatrixView MatrixView commented Nov 5, 2014

youtube-dl http://www.rtlxl.nl/#!/rtl-late-night-301978/14291f72-b3c2-33f3-8fa5-01cf48b31400

[rtlxl.nl] 14291f72-b3c2-33f3-8fa5-01cf48b31400: Downloading JSON metadata
[rtlxl.nl] 14291f72-b3c2-33f3-8fa5-01cf48b31400: Downloading f4m manifest
ERROR: Unable to download f4m manifest: HTTP Error 404: File or directory not found. (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Be sure
to call youtube-dl with the --verbose flag and include its complete output. Make sure you are using the latest version; type youtube-dl -U to update.

I'm running the latest YT-DL version on Win7. Something probably changed on the RTLxl.nl site?
Thanx in advance for looking into this issue!

@iggyvolz
Copy link

@iggyvolz iggyvolz commented Nov 5, 2014

Be sure to call youtube-dl with the --verbose flag and include its complete output.

@MatrixView
Copy link
Author

@MatrixView MatrixView commented Nov 5, 2014

youtube-dl --verbose http://www.rtlxl.nl/#!/rtl-late-night-301978/14291f72-b3c2-33f3-8fa5-01cf48b31400

[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--verbose', 'http://www.rtlxl.nl/#!/rtl-late-night-301978/14291f72-b3c2-33f3-8fa5-01cf48b31400']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2014.11.04
[debug] Python version 2.7.8 - Windows-7-6.1.7601-SP1
[debug] exe versions: ffmpeg N-67388-, ffprobe N-67388-, rtmpdump 2.4-47-gc26c541
[debug] Proxy map: {}
[rtlxl.nl] 14291f72-b3c2-33f3-8fa5-01cf48b31400: Downloading JSON metadata
[rtlxl.nl] 14291f72-b3c2-33f3-8fa5-01cf48b31400: Downloading f4m manifest
ERROR: Unable to download f4m manifest: HTTP Error 404: File or directory not found. (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Be sure
to call youtube-dl with the --verbose flag and include its complete output. Make sure you are using the latest version; type youtube-dl -U to update.
File "youtube_dl\extractor\common.pyo", line 232, in _request_webpage
File "youtube_dl\YoutubeDL.pyo", line 1289, in urlopen
File "urllib2.pyo", line 410, in open
File "urllib2.pyo", line 523, in http_response
File "urllib2.pyo", line 448, in error
File "urllib2.pyo", line 382, in _call_chain
File "urllib2.pyo", line 531, in http_error_default

@iggyvolz
Copy link

@iggyvolz iggyvolz commented Nov 5, 2014

Strangely enough, that command wouldn't work at all on my mac's Bash shell - I believe it was picking up ! as an event of some sort. Running export E="!" then replacing the ! with $E in the command gave me the same error:

$ export E="!"
$ youtube-dl --verbose "http://www.rtlxl.nl/#$E/rtl-late-night-301978/14291f72-b3c2-33f3-8fa5-01cf48b31400"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--verbose', 'http://www.rtlxl.nl/#!/rtl-late-night-301978/14291f72-b3c2-33f3-8fa5-01cf48b31400']
[debug] Encodings: locale UTF-8, fs utf-8, out None, pref UTF-8
[debug] youtube-dl version 2014.11.02.1
[debug] Python version 2.7.8 - Darwin-13.4.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 2.4.2, ffprobe 2.4.2
[debug] Proxy map: {}
[rtlxl.nl] 14291f72-b3c2-33f3-8fa5-01cf48b31400: Downloading JSON metadata
[rtlxl.nl] 14291f72-b3c2-33f3-8fa5-01cf48b31400: Downloading f4m manifest
ERROR: Unable to download f4m manifest: HTTP Error 404: File or directory not found. (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Be sure to call youtube-dl with the --verbose flag and include its complete output. Make sure you are using the latest version; type  youtube-dl -U  to update.
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 231, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1286, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

I'll look more into this later, but it seems to be that the extractor API has changed.

@naglis
Copy link
Collaborator

@naglis naglis commented Nov 18, 2014

I have changed the rtlxl.nl extractor to use m3u8 streams instead of f4m, so this issue should be fixed in the next release of youtube-dl.

@iggyvolz, wow that's quite a trick you've pulled with the variable substitution! However, you could've just used single quotes instead of double quotes, as single quotes inhibit all interpretation of the string inside the quotes.

@naglis naglis closed this Nov 18, 2014
@Reino17
Copy link

@Reino17 Reino17 commented Nov 19, 2014

I see you've chosen to go for the 'adaptive-json' (DRM variant). Why? (d=pc/ in line 33 of rtlnl.py isnt necessary btw)
I hardly know any python, but wouldn't the following work? (with the emphasis on replace())

info = self._download_json(
    'http://www.rtl.nl/system/s4m/vfd/version=2/uuid=%s/fmt=flash/' % uuid,
uuid)
...
videopath = material['videopath'].replace('f4m','m3u8')
m3u8_url = 'http://manifest.us.rtl.nl' + videopath

formats = self._extract_m3u8_formats(m3u8_url, uuid, ext='mp4')

video_urlpart = videopath.split('/flash/')[1][:-4]

Simply replace 'f4m' with 'm3u8' in "videopath" of the 'flash-json', and you have the 'non-DRM-manifest-playlist'.

@naglis
Copy link
Collaborator

@naglis naglis commented Nov 20, 2014

My motivation was to make it more robust, so I have tried to mimic the behavior of the flash player in the browser. If you can confirm that the method of simply replacing the f4m with m3u8 works ubiquitously (as I don't use rtlxl.nl personally) and/or you have issues with the encrypted HLS streams - I don't have anything against changing it so that it would work better for you and others 😉

@Reino17
Copy link

@Reino17 Reino17 commented Nov 20, 2014

I have just tested numerous videos and I can confirm this method works.

@Reino17
Copy link

@Reino17 Reino17 commented Nov 22, 2014

Hi naglis, are you still going to make that change? If not, then at least change line 47 accordingly:

video_urlpart = videopath.split('/adaptive/')[1][:-5]
naglis pushed a commit that referenced this issue Nov 22, 2014
@naglis
Copy link
Collaborator

@naglis naglis commented Nov 22, 2014

Hi @corone17,
I've made the change, it should be included in the next release of youtube-dl. Thanks again for your help.

BTW, this got a little mixed with the original issue, feel free to open new issues for any problems/suggestions you might have in the future ;)

@Reino17
Copy link

@Reino17 Reino17 commented Nov 23, 2014

Sorry to yet again comment here, but this still concerns this issue. Like I pointed out above, line 47 is still incorrect.

From the JSON:

"videopath":"/rtlxl/v166/network/pc/flash/components/actueel/rtlnieuws/307089/2014w45/e9ee31
e4-e18b-3bca-bca4-dd9aa0da6976.ssm/e9ee31e4-e18b-3bca-bca4-dd9aa0da6976.f4m",

Then because videopath = material['videopath'].replace('.f4m', '.m3u8') videopath becomes:

/rtlxl/v166/network/pc/flash/components/actueel/rtlnieuws/307089/2014w45/e9ee31e4-e18b-3bca-
bca4-dd9aa0da6976.ssm/e9ee31e4-e18b-3bca-bca4-dd9aa0da6976.m3u8

The extension is now 4 characters instead of 3 and therefore after

 video_urlpart = videopath.split('/flash/')[1][:-4]
PG_URL_TEMPLATE = 'http://pg.us.rtl.nl/rtlxl/network/%s/progressive/%s.mp4'
formats.extend([
{
'url': PG_URL_TEMPLATE % ('a2m', video_urlpart),
'format_id': 'pg-sd',
},
{
'url': PG_URL_TEMPLATE % ('a3m', video_urlpart),
'format_id': 'pg-hd',
'quality': 0,
}
])

format pg-hd is

http://pg.us.rtl.nl/rtlxl/network/a3m/progressive/components/actueel/rtlnieuws/307089/2014w4
5/e9ee31e4-e18b-3bca-bca4-dd9aa0da6976.ssm/e9ee31e4-e18b-3bca-bca4-dd9aa0da6976..mp4

and has a period too many. Line 47 therefore has to be:

video_urlpart = videopath.split('/adaptive/')[1][:-5]
naglis pushed a commit that referenced this issue Nov 23, 2014
@naglis
Copy link
Collaborator

@naglis naglis commented Nov 23, 2014

That was pretty sloppy of me. Thanks! Should be fixed in the next version of youtube-dl.

@Reino17
Copy link

@Reino17 Reino17 commented Nov 25, 2014

Working fine now. Thanks, naglis.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.