Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TypeError: __init__() got an unexpected keyword argument 'check_hostname' [while running 'Download the YouTube video'] #17974

Closed
hanmeng31 opened this issue Oct 25, 2018 · 5 comments

Comments

@hanmeng31
Copy link

@hanmeng31 hanmeng31 commented Oct 25, 2018

Please follow the guide below

  • You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
  • Put an x into all the boxes [ ] relevant to your issue (like this: [x])
  • Use the Preview tab to see what your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.10.05. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

  • I've verified and I assure that I'm running youtube-dl 2018.10.05

Before submitting an issue make sure you have:

  • At least skimmed through the README, most notably the FAQ and BUGS sections
  • Searched the bugtracker for similar issues including closed ones
  • Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

  • Bug report (encountered problems with youtube-dl)
  • Site support request (request for adding support for a new site)
  • Feature request (request for a new functionality)
  • Question
  • Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue


If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

[debug] youtube-dl version 2018.10.05
[debug] Python version 2.7.15 - MacOS 10.13.6
[debug] Apache Beam 2.7.0

The issue was quite straight forward. I could run the following code in a single independent python file without any problem.

import youtube_dl

web_url = ANY VALID YOUTUBE URL

download_location = web_url.split('/')[-1] + '.wav'

options = {
	'format': 'bestaudio/best',
	'extractaudio': True,
	'audioformat': "wav",
	'outtmpl': download_location,
	'noplaylist': True,
	}

with youtube_dl.YoutubeDL(options) as ydl:
	print("The url is", web_url)
	ydl.download([web_url])

But when I run the same code in an Apache Beam program with the same virtual environment, it produces the following error:

File "/anaconda3/envs/py27/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 2001, in download url, force_generic_extractor=self.params.get('force_generic_extractor', False)) File "/anaconda3/envs/py27/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 792, in extract_info ie_result = ie.extract(url) File "/anaconda3/envs/py27/lib/python2.7/site-packages/youtube_dl/extractor/common.py", line 507, in extract ie_result = self._real_extract(url) File "/anaconda3/envs/py27/lib/python2.7/site-packages/youtube_dl/extractor/youtube.py", line 1519, in _real_extract video_webpage = self._download_webpage(url, video_id) File "/anaconda3/envs/py27/lib/python2.7/site-packages/youtube_dl/extractor/common.py", line 766, in _download_webpage expected_status=expected_status) File "/anaconda3/envs/py27/lib/python2.7/site-packages/youtube_dl/extractor/youtube.py", line 266, in _download_webpage_handle *args, **compat_kwargs(kwargs)) File "/anaconda3/envs/py27/lib/python2.7/site-packages/youtube_dl/extractor/common.py", line 632, in _download_webpage_handle urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status) File "/anaconda3/envs/py27/lib/python2.7/site-packages/youtube_dl/extractor/common.py", line 604, in _request_webpage return self._downloader.urlopen(url_or_request) File "/anaconda3/envs/py27/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 2211, in urlopen return self._opener.open(req, timeout=self._socket_timeout) File "/anaconda3/envs/py27/lib/python2.7/site-packages/future/backports/urllib/request.py", line 494, in open response = self._open(req, data) File "/anaconda3/envs/py27/lib/python2.7/site-packages/future/backports/urllib/request.py", line 512, in _open '_open', req) File "/anaconda3/envs/py27/lib/python2.7/site-packages/future/backports/urllib/request.py", line 466, in _call_chain result = func(*args) File "/anaconda3/envs/py27/lib/python2.7/site-packages/youtube_dl/utils.py", line 1139, in https_open req, **kwargs) File "/anaconda3/envs/py27/lib/python2.7/site-packages/future/backports/urllib/request.py", line 1255, in do_open h = http_class(host, timeout=req.timeout, **http_conn_args) File "/anaconda3/envs/py27/lib/python2.7/site-packages/youtube_dl/utils.py", line 882, in _create_http_connection hc = http_class(*args, **compat_kwargs(kwargs)) TypeError: __init__() got an unexpected keyword argument 'check_hostname' [while running 'Download the YouTube video']

@dstftw
Copy link
Collaborator

@dstftw dstftw commented Oct 28, 2018

Provide complete log with verbose set to True.

@dstftw dstftw closed this Oct 28, 2018
@dstftw dstftw added the incomplete label Oct 28, 2018
@hanmeng31
Copy link
Author

@hanmeng31 hanmeng31 commented Oct 29, 2018

I added the verbose option to be True in Python.

    options = {
    'format': 'bestaudio/best',
    'extractaudio': True,  # only keep the audio
    'audioformat': "wav",  # convert to wav
    'outtmpl': unicode(whole_path),  # name the file the ID of the video
    'noplaylist': True,
    'verbose': True
}

I got the following log

INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.client:Refreshing access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.client:Refreshing access_token
INFO:root:==================== <function annotate_downstream_side_inputs at 0x1122901b8> ====================
INFO:root:==================== <function fix_side_input_pcoll_coders at 0x112290488> ====================
INFO:root:==================== <function lift_combiners at 0x110f4af50> ====================
INFO:root:==================== <function expand_gbk at 0x112290410> ====================
INFO:root:==================== <function sink_flattens at 0x112290398> ====================
INFO:root:==================== <function greedily_fuse at 0x112290320> ====================
INFO:root:==================== <function sort_stages at 0x1122900c8> ====================
INFO:root:Running (ref_AppliedPTransform_Read the meta file./Read_3)+((ref_AppliedPTransform_Process each line of video data_4)+(ref_AppliedPTransform_Download the YouTube video_5))
INFO:root:start <DoOperation Download the YouTube video output_tags=['out']>
INFO:root:start <DoOperation Process each line of video data output_tags=['out']>
INFO:root:start <ReadOperation Read the meta file./Read source=SourceBundle(weight=1.0, source=<apache_beam.io.textio._TextSource object at 0x112603d50>, start_position=None, stop_position=None)>
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.client:Refreshing access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.client:Refreshing access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.client:Refreshing access_token
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.10.05
[debug] Python version 2.7.15 (CPython) - Darwin-17.7.0-x86_64-i386-64bit
[debug] exe versions: avconv 12.3, avprobe 12.3, ffmpeg 4.0.1, ffprobe 4.0.1
[debug] Proxy map: {}
('url is:', 'https://www.youtube.com/watch?v=--PJHxphWEs')
[youtube] --PJHxphWEs: Downloading webpage
/anaconda3/envs/py27/lib/python2.7/site-packages/future/backports/urllib/request.py:353: DeprecationWarning: Request.is_unverifiable method is deprecated.
  warnings.warn(msg, DeprecationWarning, stacklevel=1)
Traceback (most recent call last):
  File "/Users/***/Desktop/project/Pipeline/get_audioset_pipeline_code/pipeline_main.py", line 125, in <module>
    run()
  File "/Users/***/Desktop/project/Pipeline/get_audioset_pipeline_code/pipeline_main.py", line 119, in run
    result = p.run()
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/apache_beam/pipeline.py", line 394, in run
    self.to_runner_api(), self.runner, self._options).run(False)
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/apache_beam/pipeline.py", line 407, in run
    return self.runner.run_pipeline(self)
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py", line 135, in run_pipeline
    return runner.run_pipeline(pipeline)
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 223, in run_pipeline
    return self.run_via_runner_api(pipeline.to_runner_api())
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 226, in run_via_runner_api
    return self.run_stages(*self.create_stages(pipeline_proto))
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 864, in run_stages
    pcoll_buffers, safe_coders).process_bundle.metrics
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 977, in run_stage
    self._progress_frequency).process_bundle(data_input, data_output)
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 1181, in process_bundle
    result_future = self._controller.control_handler.push(process_bundle)
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 1061, in push
    response = self.worker.do_instruction(request)
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 215, in do_instruction
    request.instruction_id)
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 237, in process_bundle
    processor.process_bundle(instruction_id)
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 292, in process_bundle
    op.start()
  File "apache_beam/runners/worker/operations.py", line 235, in apache_beam.runners.worker.operations.ReadOperation.start
  File "apache_beam/runners/worker/operations.py", line 236, in apache_beam.runners.worker.operations.ReadOperation.start
  File "apache_beam/runners/worker/operations.py", line 245, in apache_beam.runners.worker.operations.ReadOperation.start
  File "apache_beam/runners/worker/operations.py", line 167, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 87, in apache_beam.runners.worker.operations.ConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 387, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 388, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 589, in apache_beam.runners.common.DoFnRunner.receive
  File "apache_beam/runners/common.py", line 595, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 612, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 593, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 363, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "apache_beam/runners/common.py", line 698, in apache_beam.runners.common._OutputProcessor.process_outputs
  File "apache_beam/runners/worker/operations.py", line 87, in apache_beam.runners.worker.operations.ConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 387, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 388, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 589, in apache_beam.runners.common.DoFnRunner.receive
  File "apache_beam/runners/common.py", line 595, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 628, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 593, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 472, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam/runners/common.py", line 523, in apache_beam.runners.common.PerWindowInvoker._invoke_per_window
  File "/Users/***/Desktop/project/Pipeline/get_audioset_pipeline_code/process_data/download_files.py", line 70, in process
    ydl.download([web_url])  # download the audio file from YouTube to a file named random_file_name
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 2001, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 792, in extract_info
    ie_result = ie.extract(url)
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/youtube_dl/extractor/common.py", line 507, in extract
    ie_result = self._real_extract(url)
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/youtube_dl/extractor/youtube.py", line 1519, in _real_extract
    video_webpage = self._download_webpage(url, video_id)
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/youtube_dl/extractor/common.py", line 766, in _download_webpage
    expected_status=expected_status)
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/youtube_dl/extractor/youtube.py", line 266, in _download_webpage_handle
    *args, **compat_kwargs(kwargs))
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/youtube_dl/extractor/common.py", line 632, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/youtube_dl/extractor/common.py", line 604, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 2211, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/future/backports/urllib/request.py", line 494, in open
    response = self._open(req, data)
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/future/backports/urllib/request.py", line 512, in _open
    '_open', req)
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/future/backports/urllib/request.py", line 466, in _call_chain
    result = func(*args)
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/youtube_dl/utils.py", line 1139, in https_open
    req, **kwargs)
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/future/backports/urllib/request.py", line 1255, in do_open
    h = http_class(host, timeout=req.timeout, **http_conn_args)
  File "/anaconda3/envs/py27/lib/python2.7/site-packages/youtube_dl/utils.py", line 882, in _create_http_connection
    hc = http_class(*args, **compat_kwargs(kwargs))
TypeError: __init__() got an unexpected keyword argument 'check_hostname' [while running 'Download the YouTube video']
@dstftw dstftw added cant-reproduce and removed incomplete labels Oct 29, 2018
@ptphuy
Copy link

@ptphuy ptphuy commented Apr 25, 2019

I also had this issue and I am trying to find solution for this issue. Please help!

@bjester
Copy link

@bjester bjester commented Jun 11, 2019

I've encountered this same issue running Python 2.7.15rc1 on Ubuntu 18.04, along with a coworker running 2.7.10 on Mac, as well as on Travis here: https://travis-ci.com/learningequality/pressurecooker/jobs/207150978

The error seemed like it related to SSL since verifying the host name matches a certificate is expected with secure requests. Doing some research, I found this which led me to believe it had to do with urllib along with future/backports/urllib/request.py in the stack trace. Removing the try/except and ensuring this line always happens fixes the issue for me.

Importing urllib.request does indeed throw a ImportError but something must be going wrong as this isn't working as expected because forcing urllib2 instead makes everything happy in Python 2 at least.

@bjester
Copy link

@bjester bjester commented Jun 11, 2019

So digging in some more, it would appear this is because urllib.request is being shimmed because of another package in our project using future. Therefore, the try/except doesn't play nice with that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.