gvsearch and ybsearch -for downloading subtitles only- giving UnsupportedURL error and URLError

[aysenurbilgin]

Hi there,

I am trying to run gvsearch with the following arguments:

youtube-dl gvsearch10:'artificial intelligence history documentary' --skip-download --write-sub --sub-lang='en' --write-info-json -o './videoCorpus/sub/%(id)s.%(ext)s'

But I receive the following error:

[video.google:search] gvsearch:artificial intelligence history documentary: Downloading result page 1
[video.google:search] gvsearch:artificial intelligence history documentary: Downloading result page 2
[download] Downloading playlist: artificial intelligence history documentary
[video.google:search] playlist artificial intelligence history documentary: Collected 10 video ids (downloading 10 of them)
[download] Downloading video 1 of 10
[generic] 2215: Requesting header
WARNING: Falling back on generic information extractor.
[generic] 2215: Downloading webpage
[generic] 2215: Extracting information
ERROR: Unsupported URL: <url here>

I also tried the following command (with --default-search auto):

youtube-dl gvsearch10:'artificial intelligence history documentary' --default-search auto --skip-download --write-sub --sub-lang='en' --write-info-json -o './videoCorpus/sub/%(id)s.%(ext)s'

Then changed the keywords to 'artificial intelligence' and was going well until:

......
[download] Downloading video 5 of 10
[youtube] _19pRsZRiz4: Downloading webpage
[youtube] _19pRsZRiz4: Downloading video info webpage
[youtube] _19pRsZRiz4: Extracting video information
WARNING: video doesn't have subtitles
[youtube] _19pRsZRiz4: Downloading MPD manifest
[info] Writing video description metadata as JSON to: ./videoCorpus/sub/_19pRsZRiz4.info.json
[download] Downloading video 6 of 10
[generic] mark-zuckerberg-rejects-fears-of-rogue-artificial-intelligence: Requesting header
WARNING: Falling back on generic information extractor.
[generic] mark-zuckerberg-rejects-fears-of-rogue-artificial-intelligence: Downloading webpage
[generic] mark-zuckerberg-rejects-fears-of-rogue-artificial-intelligence: Extracting information
ERROR: Unsupported URL: <url here>

Also tried ybsearch:

youtube-dl ybsearch10:'artificial intelligence' --default-search auto --skip-download --write-sub --sub-lang='en' --write-info-json -o './videoCorpus/sub/%(id)s.%(ext)s'

And received the following:

[generic] ybsearch10:artificial intelligence: Requesting header
WARNING: Could not send HEAD request to ybsearch10:artificial intelligence: <urlopen error unknown url type: ybsearch10>
[generic] ybsearch10:artificial intelligence: Downloading webpage
ERROR: Unable to download webpage: <urlopen error unknown url type: ybsearch10> (caused by URLError(u'unknown url type: ybsearch10',))

I was wondering whether there is a way to force continue and ignore errors? I intend to do this search for larger numbers and with other keywords and it would be great if I could leave it running. Any suggestions appreciated.

Thanks for looking into it!

[dstftw]

Post the full output of youtube-dl when run with -v, i.e. add -v flag to your command line, copy the whole output and post it in the issue body wrapped in ``` for better formatting. It should look similar to this:

$ youtube-dl -v http://www.youtube.com/watch?v=BaW_jenozKcj
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2015.12.06
[debug] Git HEAD: 135392e
[debug] Python version 2.6.6 - Windows-2003Server-5.2.3790-SP2
[debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
[debug] Proxy map: {}
...

Do not post screenshots of verbose log only plain text is acceptable.

The output (including the first lines) contains important debugging information. Issues without the full output are often not reproducible and therefore do not get solved in short order, if ever.

[dstftw]

i.e. add -v flag to your command line

[aysenurbilgin]

[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'ybsearch10:artificial intelligence', u'--default-search', u'auto', u'--skip-download', u'--write-sub', u'--sub-lang=en', u'--write-info-json', u'-o', u'./videoCorpus/sub/%(id)s.%(ext)s']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.02.13
[debug] Python version 2.7.10 - Darwin-15.3.0-x86_64-i386-64bit
[debug] exe versions: none
[debug] Proxy map: {}
[generic] ybsearch10:artificial intelligence: Requesting header
WARNING: Could not send HEAD request to ybsearch10:artificial intelligence: <urlopen error unknown url type: ybsearch10>
[generic] ybsearch10:artificial intelligence: Downloading webpage
ERROR: Unable to download webpage: <urlopen error unknown url type: ybsearch10> (caused by URLError(u'unknown url type: ybsearch10',))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/extractor/common.py", line 354, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 1905, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 454, in _open
    'unknown_open', req)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1265, in unknown_open
    raise URLError('unknown url type: %s' % type)

[aysenurbilgin]

youtube-dl -v gvsearch10:'artificial intelligence' --default-search auto --skip-download --write-sub --sub-lang='en' --write-info-json -o './videoCorpus/sub/%(id)s.%(ext)s' 
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'gvsearch10:artificial intelligence', u'--default-search', u'auto', u'--skip-download', u'--write-sub', u'--sub-lang=en', u'--write-info-json', u'-o', u'./videoCorpus/sub/%(id)s.%(ext)s']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.02.13
[debug] Python version 2.7.10 - Darwin-15.3.0-x86_64-i386-64bit
[debug] exe versions: none
[debug] Proxy map: {}
[video.google:search] gvsearch:artificial intelligence: Downloading result page 1
[download] Downloading playlist: artificial intelligence
[video.google:search] playlist artificial intelligence: Collected 10 video ids (downloading 10 of them)
...
[download] Downloading video 6 of 10
[generic] mark-zuckerberg-rejects-fears-of-rogue-artificial-intelligence: Requesting header
WARNING: Falling back on generic information extractor.
[generic] mark-zuckerberg-rejects-fears-of-rogue-artificial-intelligence: Downloading webpage
[generic] mark-zuckerberg-rejects-fears-of-rogue-artificial-intelligence: Extracting information
ERROR: Unsupported URL: http://www.cnbc.com/2016/01/04/mark-zuckerberg-rejects-fears-of-rogue-artificial-intelligence.html
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/extractor/generic.py", line 1308, in _real_extract
    doc = compat_etree_fromstring(webpage.encode('utf-8'))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/compat.py", line 248, in compat_etree_fromstring
    doc = _XML(text, parser=etree.XMLParser(target=etree.TreeBuilder(element_factory=_element_factory)))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/compat.py", line 237, in _XML
    parser.feed(text)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1642, in feed
    self._raiseerror(v)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1506, in _raiseerror
    raise err
ParseError: not well-formed (invalid token): line 4, column 47
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 666, in extract_info
    ie_result = ie.extract(url)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/extractor/common.py", line 315, in extract
    return self._real_extract(url)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/extractor/generic.py", line 1945, in _real_extract
    raise UnsupportedError(url)
UnsupportedError: Unsupported URL: http://www.cnbc.com/2016/01/04/mark-zuckerberg-rejects-fears-of-rogue-artificial-intelligence.html

[dstftw]

1. There is no such search shortcut ybsearch.
2. cnbc.com is not supported.

[aysenurbilgin]

1.Oh, I saw it in #274

Is it possible to bypass these errors and continue searching?

[dstftw]

Use -i.

[aysenurbilgin]

Thanks! It worked.

[aysenurbilgin]

And of course, it's in the documentation but I am sorry I must have
magically missed it.
Thanks for looking into it.

On Fri, Feb 19, 2016 at 5:37 PM, Sergey M. notifications@github.com wrote:

Use -i.

—
Reply to this email directly or view it on GitHub
#8601 (comment).