Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot handle if website use javascript to generate names on the fly #13172

Closed
5 of 8 tasks
yoonghm opened this issue May 20, 2017 · 0 comments
Closed
5 of 8 tasks

Cannot handle if website use javascript to generate names on the fly #13172

yoonghm opened this issue May 20, 2017 · 0 comments

Comments

@yoonghm
Copy link

yoonghm commented May 20, 2017

Please follow the guide below

  • You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
  • Put an x into all the boxes [ ] relevant to your issue (like that [x])
  • Use Preview tab to see how your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2017.05.18.1. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

  • I've verified and I assure that I'm running youtube-dl 2017.05.18.1

Before submitting an issue make sure you have:

  • At least skimmed through README and most notably FAQ and BUGS sections
  • Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

  • Bug report (encountered problems with youtube-dl)
  • Site support request (request for adding support for a new site)
  • Feature request (request for a new functionality)
  • Question
  • Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue


If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add -v flag to your command line you run youtube-dl with, copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

$ youtube-dl -v https://video.toggle.sg/en/tv-show/news/may-2017-cna-singapore-tonight/fri-19-may-2017/512456
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://video.toggle.sg/en/tv-show/news/may-2017-cna-singapore-tonight/fri-19-may-2017/512456']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.05.18.1
[debug] Python version 2.7.12 - Linux-4.4.0-75-generic-x86_64-with-Ubuntu-16.04-xenial
[debug] exe versions: ffmpeg 2.8.11-0ubuntu0.16.04.1, ffprobe 2.8.11-0ubuntu0.16.04.1
[debug] Proxy map: {}
[generic] 512456: Requesting header
WARNING: Falling back on generic information extractor.
[generic] 512456: Downloading webpage
[generic] 512456: Extracting information
ERROR: Unsupported URL: https://video.toggle.sg/en/tv-show/news/may-2017-cna-singapore-tonight/fri-19-may-2017/512456
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/generic.py", line 1970, in _real_extract
    doc = compat_etree_fromstring(webpage.encode('utf-8'))
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/compat.py", line 2526, in compat_etree_fromstring
    doc = _XML(text, parser=etree.XMLParser(target=_TreeBuilder(element_factory=_element_factory)))
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/compat.py", line 2515, in _XML
    parser.feed(text)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1653, in feed
    self._raiseerror(v)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1517, in _raiseerror
    raise err
ParseError: not well-formed (invalid token): line 11, column 50
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 760, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 433, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/generic.py", line 2795, in _real_extract
    raise UnsupportedError(url)
UnsupportedError: Unsupported URL: https://video.toggle.sg/en/tv-show/news/may-2017-cna-singapore-tonight/fri-19-may-2017/512456
...
<end of log>

If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):

https://video.toggle.sg/en/tv-show/news/may-2017-news-5/fri-19-may-2017/512465

Note that youtube-dl does not support sites dedicated to copyright infringement. In order for site support request to be accepted all provided example URLs should not violate any copyrights.


Description of your issue, suggested solution and other information

I guess the website use javascript to generate final links (video and audio) for streaming.

https://k-toggle.akamaized.net/dash/p/2082311/sp/208231100/serveFlavor/entryId/0_h7ttac2w/v/,12/pv/1/flavorId/0_98vlffar,12/pv/1/flavorId/0_11hqfxh5,12/pv/1/flavorId/0_sthtlgvo,2/pv/1/flavorId/0_uys1uif6,12/pv/1/flavorId/0_oeirwtcp,/forceproxy/true/name/a.mp4.urlset/fragment-109-f2-a1-x3.m4s
https://k-toggle.akamaized.net/dash/p/2082311/sp/208231100/serveFlavor/entryId/0_h7ttac2w/v/,12/pv/1/flavorId/0_98vlffar,12/pv/1/flavorId/0_11hqfxh5,12/pv/1/flavorId/0_sthtlgvo,2/pv/1/flavorId/0_uys1uif6,12/pv/1/flavorId/0_oeirwtcp,/forceproxy/true/name/a.mp4.urlset/fragment-109-f5-v1-x3.m4s

Is there a mechanism for youtube_dl to call javascript and see the generated URLs? Must find away HOW the streamed videos and audios are passed to web browser.

@dstftw dstftw closed this as completed in 35a2d22 May 20, 2017
khavishbhundoo referenced this issue in khavishbhundoo/youtube-dl Jun 14, 2017
* [cbsinteractive] fix extractor

* [cbsinteractive] update test cases

* [cbsinteractive] extract formats with `CBSIE`

* [extractor/common] Fix rtmp and rtsp formats' URLs in _extract_wowza_formats

* [vier] Extract more info

Extract the `episode_number` and `upload_date`. Also extract the real
`description`.

* [vier] Relax regexes and extract more metadata (closes #12539)

* [jsinterp] Add support for quoted names and indexers (closes #13123, closes #13130)

* [ChangeLog] Actualize

* release 2017.05.18

* [ChangeLog] Fix typo

* [jsinterp] Fix typo and cleanup regexes (closes #13134)

* [ChangeLog] Actualize

* release 2017.05.18.1

* [mitele] Update app key regex

* [hitbox] Add support for smashcast.tv (closes #13154)

* [njpwworld] Fix extraction (closes #13162)

* [toypics] Fix extraction

* [toypics] Improve and modernize

* [adobepass] Add support for Brighthouse MSO

* [toggle] Relax _VALID_URL (closes #13172)

* [youtube] Fix DASH manifest signature decryption (closes #8944)

* [youtube] Modernize

* [streamcz] Add support for subtitles

* [downloader/external] Pass -loglevel to ffmpeg downloader (closes #13183)

* Credit @zurfyx for atresplayer improvements (#12548)

* Credit @mphe for streamango (#12643)

* Credit @fredbourni for noovo (#12792)

* [ChangeLog] Actualize

* release 2017.05.23

* Credit @timendum for rai (#11790) and mediaset (#12964)

* Credit @gritstub for vevo fix (#12879)

* [cbsnews] fix extraction for 60 Minutes videos

* [vimeo] Fix formats' sorting (closes #13189)

* [postprocessor/ffmpeg] Fix metadata filename handling on Python 2

Fixes #13182

* [udemy] Fix extraction for outputs' format entries without URL (closes #13192)

* [youku] Fix extraction (closes #13191)

* [utils] Recognize more patterns in strip_jsonp()

Used in Youku Show pages

* [youku:show] Fix extraction

* [tudou] Merge into youku extractor (fixes #12214)

Also, there are no tudou playlists anymore. All playlist URLs points to youku
playlists.

* [bbc] Add support for authentication

* Revert "[youtube] Don't use the DASH manifest from 'get_video_info' if 'use_cipher_signature' is True (#5118)"

This reverts commit 87dc451.

* [ChangeLog] Update after the fix for #11381

* [ChangeLog] Actualize

* release 2017.05.26

* [cbsnews] Fix extraction (closes #13205)

* [youku] Extract more metadata (closes #10433)

* [adn] fix formats extraction

* [utils] Drop an compatibility wrapper for Python < 2.6

addinfourl.getcode is added since Python 2.6a1. As youtube-dl now
requires 2.6+, this is no longer necessary.

See python/cpython@9b0d46d

* [cbsinteractive] Relax _VALID_URL (closes #13213)

* [beam:vod] Add extractor

* [beam] Improve and add support for mixer.com (closes #13032)

* [dvtv] Parse adaptive formats as well

The old code hit an error when it attempted to parse the string
"adaptive" for video height. Actually parsing the returned playlists is
a good idea because it adds more output formats, including some
audio-only-ones.

* [dvtv] Improve and fix playlists support (closes #13063)

* [medialaan] Fix videos with missing videoUrl

A rough trick to get around the two different json styles medialaan seems to be using.
Fix for these example videos:
https://vtmkzoom.be/video?aid=45724
https://vtmkzoom.be/video?aid=45425

* [medialaan] PEP 8 (closes #12774)

* [gaskrank] Fix extraction

* [gaskrank] Improve (closes #12493)

* [abcnews] Add support for embed URLs

* [abcnews] Improve and remove duplicate test (closes #12851)

* [xhamster] Extract categories (closes #11728)

* [xhamster] Fix author and like/dislike count extraction

* [xhamster] Simplify (closes #13216)

* [youtube] Parse player_url if format URLs are encrypted or DASH MPDs are requested

Fixes #13211

* [ChangeLog] Actualize

* release 2017.05.29

* [README.md] Add an example for how to use .netrc on Windows

That's a Python bug: http://bugs.python.org/issue28334
Most likely it will be fixed in Python 3.7: python/cpython#123

* [README.md] Mention http_dash_segments protocol

* [packtpub] Fix authentication(closes #13240)

* [drbonanza] Fix extraction (closes #13231)

* [francetv] Relax _VALID_URL

* [1tv] Lower preference for http formats (closes #13246)

* [youtube] Improve chapters extraction (closes #13247)

* [safari] Fix typo (closes #13252)

* [YoutubeDL] Don't emit ANSI escape codes on Windows

* [godtv] Remove extractor (closes #13175)

* [pornhub:playlist] Fix extraction (closes #13281)

* [pornhub:uservideos] Add missing raise

* [bandcamp:weekly] Add extractor

* [bandcamp:weekly] Improve and extract more metadata (closes #12758)

* Credit @adamvoss for bandcamp:weekly (#12758)

* Credit @mikf for beam:vod (#13032)

* Credit @jktjkt for dvtv formats (#13063)

* [ChangeLog] Actualize

* release 2017.06.05

* [tvplayer] Fix extraction (closes #13291)

* [rtlnl] Improve _VALID_URL (closes #13295)

* [streamango] Make title optional

* [streamango] Skip download for test (closes #13292)

* [README.md] Clarify output template references (closes #13316)

* [README.md] Improve man page formatting

* [YoutubeDL] Sanitize more fields (#13313)

* [liveleak] Ensure height is int (closes #13313)

* [safari] Improve authentication detection (closes #13319)

* [sohu] Fix numeric fields

* [flickr] Ensure format id is string

* [foxgay] Ensure height is int

* [gfycat] Ensure filesize is int

* [golem] Ensure format id is string

* [jove] Ensure comment count is int

* [sexu] Ensure height is int

* [turbo] Ensure format id is string

* [extractor/common] Return unicode string from _match_id

* [extractor/generic] Ensure format id is unicode string

* [msn] Fix formats extraction

* [newgrounds] Improve formats and uploader extraction (closes #13346)

* [newgrounds:playlist] Add extractor (closes #10611)

* [utils] Improve unified_timestamp

* [newgrounds] Extract more metadata (closes #13232)

* [rutv] Add support for testplayer.vgtrk.com (closes #13347)

* [xfileshare] Modernize and pass referrer

* [xfileshare] Add support for rapidvideo (closes #13348)

* [compat] Introduce compat_HTMLParseError

* [utils] Handle HTMLParseError in extract_attributes (closes #13349)

* [xfileshare] PEP 8

* [ChangeLog] Actualize

* release 2017.06.12

* [compat] Add compat_HTMLParseError to __all__

* [corus] Add support for history.ca (closes #13359)

* [corus] Add support for showcase.ca
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant