Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upGitHub is where the world builds software
Millions of developers and companies build, ship, and maintain their software on GitHub — the largest and most advanced development platform in the world.
C-SPAN videos not working #18806
C-SPAN videos not working #18806
Comments
|
I have noticed that as well. There is an extremely easy way to get the m3u8 for all C-Span videos that you can use to play or record the video in VLC. Go to the html source code (ctrl-U in Chrome and Firefox) for the C-Span webpage that contains the video and search "m3u8." The playable and recordable m3u8 is right there. You can also download the m3u8 URL in a browser. Then, open it as a text file. You'll see several resolutions and bittrates for the video. |
|
Using Youtube-dl for C-Span was working for me a few days ago with following command: c:>youtube-dl -f bestvideo[ext=mp Today it won't work... instead I get following when executing same command... Not sure what has changed in Youtube-dl or on C-Span site but clearly something has changed.... and I am using current version Youtube dl 2019.02.08 What follows is the command and response from website when I try today! Please help.... V:\Dots02a\War-Criminals\Elliot-Abrams\New folder>youtube-dl -f bestvideo[ext=mp |
|
This issue is still present in the March 1, 2019 update. |
|
Issue still present with the April 17, 2019 update |
|
Error still there but thanks @bonacker for the tip :) |
|
The URL used for finding MP4 links, Working link 1 in the issue description resolves to https://www.c-span.org/assets/player/ajax-player.php?os=android&html5=program&id=518366 (still works as of 2020-03-13) Failing link 1 in the issue description resolves to https://www.c-span.org/assets/player/ajax-player.php?os=android&html5=program&id=518828 (no good) The affected links seem to always return:
This diff is outdated, please see bottom of commentI'm not very familiar with Python or youtube-dl. Here is a WIP diff that retrieves some updated information from the page, including an m3u8 URL: diff --git a/youtube_dl/extractor/cspan.py b/youtube_dl/extractor/cspan.py
index 68ba05844..0acd2a316 100644
--- a/youtube_dl/extractor/cspan.py
+++ b/youtube_dl/extractor/cspan.py
@@ -12,6 +12,7 @@ from ..utils import (
int_or_none,
smuggle_url,
unescapeHTML,
+ js_to_json,
)
from .senateisvp import SenateISVPIE
from .ustream import UstreamIE
@@ -142,7 +143,17 @@ class CSpanIE(InfoExtractor):
'http://www.c-span.org/assets/player/ajax-player.php?os=android&html5=%s&id=%s' % (video_type, video_id),
video_id)['video']
if data['@status'] != 'Success':
- raise ExtractorError('%s said: %s' % (self.IE_NAME, get_text_attr(data, 'error')), expected=True)
+ jw_setup_js = self._search_regex(
+ r'jwsetup\s*=\s*(\{[^;]+\});',
+ webpage, 'jwsetup config')
+ jw_setup = self._parse_json(
+ jw_setup_js, url, transform_source=js_to_json)
+ if jw_setup:
+ m3u8_url = jw_setup['playlist'][0]['sources'][0]['file']
+ # not sure where to go from here, _extract_m3u8_formats?
+ raise ExtractorError('%s retrieved m3u8 URL but not sure how to handle it: %s' % (self.IE_NAME, m3u8_url), expected=True)
+ else:
+ raise ExtractorError('%s said: %s, and we could not parse jw_setup' % (self.IE_NAME, get_text_attr(data, 'error')), expected=True)
doc = self._download_xml(
'http://www.c-span.org/common/services/flashXml.php?%sid=%s' % (video_type, video_id),and here is a testcase I am running with diff --git a/youtube_dl/extractor/cspan.py b/youtube_dl/extractor/cspan.py
index 67d6df4b0..68ba05844 100644
--- a/youtube_dl/extractor/cspan.py
+++ b/youtube_dl/extractor/cspan.py
@@ -73,6 +73,14 @@ class CSpanIE(InfoExtractor):
# Audio Only
'url': 'https://www.c-span.org/video/?437336-1/judiciary-antitrust-competition-policy-consumer-rights',
'only_matching': True,
+ }, {
+ 'url': 'https://www.c-span.org/video/?470351-1/president-trump-declares-national-emergency-response-coronavirus',
+ 'info_dict': {
+ 'id': '518828',
+ 'ext': 'mp4',
+ 'title': 'President Trump News Conference on Coronavirus Response',
+ 'description': 'President Trump held a news conference at the White House on the coronavirus.',
+ },
}]
BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/%s/%s_%s/index.html?videoId=%s'Would love if anyone could continue with this or update for the newer links This is the diff I'm using with diff --git a/youtube_dl/extractor/cspan.py b/youtube_dl/extractor/cspan.py
index 67d6df4b0..505892a15 100644
--- a/youtube_dl/extractor/cspan.py
+++ b/youtube_dl/extractor/cspan.py
@@ -12,6 +12,7 @@ from ..utils import (
int_or_none,
smuggle_url,
unescapeHTML,
+ js_to_json,
)
from .senateisvp import SenateISVPIE
from .ustream import UstreamIE
@@ -73,6 +74,17 @@ class CSpanIE(InfoExtractor):
# Audio Only
'url': 'https://www.c-span.org/video/?437336-1/judiciary-antitrust-competition-policy-consumer-rights',
'only_matching': True,
+ }, {
+ 'url': 'https://www.c-span.org/video/?470351-1/president-trump-declares-national-emergency-response-coronavirus',
+ 'info_dict': {
+ 'id': '543252',
+ 'ext': 'mp4',
+ 'title': 'President Trump News Conference on Coronavirus Response',
+ 'description': 'Speaking in the Rose Garden, President Trump declares the coronavirus pandemic a national emergency.',
+ },
+ 'params': {
+ 'skip_download': True, # m3u8 downloads
+ },
}]
BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/%s/%s_%s/index.html?videoId=%s'
@@ -130,12 +142,6 @@ class CSpanIE(InfoExtractor):
def get_text_attr(d, attr):
return d.get(attr, {}).get('#text')
- data = self._download_json(
- 'http://www.c-span.org/assets/player/ajax-player.php?os=android&html5=%s&id=%s' % (video_type, video_id),
- video_id)['video']
- if data['@status'] != 'Success':
- raise ExtractorError('%s said: %s' % (self.IE_NAME, get_text_attr(data, 'error')), expected=True)
-
doc = self._download_xml(
'http://www.c-span.org/common/services/flashXml.php?%sid=%s' % (video_type, video_id),
video_id)
@@ -145,6 +151,35 @@ class CSpanIE(InfoExtractor):
title = find_xpath_attr(doc, './/string', 'name', 'title').text
thumbnail = find_xpath_attr(doc, './/string', 'name', 'poster').text
+ data = self._download_json(
+ 'http://www.c-span.org/assets/player/ajax-player.php?os=android&html5=%s&id=%s' % (video_type, video_id),
+ video_id)['video']
+ if data['@status'] != 'Success':
+ jw_setup_js = self._search_regex(
+ r'jwsetup\s*=\s*(\{[^;]+\});',
+ webpage, 'jwsetup config')
+ jw_setup = self._parse_json(
+ jw_setup_js, video_id, transform_source=js_to_json)
+ if jw_setup:
+ m3u8_url = jw_setup['playlist'][0]['sources'][0]['file']
+ formats = self._extract_m3u8_formats(
+ m3u8_url, video_id, 'mp4',
+ entry_protocol='m3u8_native', m3u8_id='hls', fatal=False
+ )
+ if formats:
+ for format in formats:
+ format['http_headers'] = {'Referer': 'https://c-span.org/'}
+ return {
+ 'id': video_id,
+ 'title': title,
+ 'description': description,
+ 'formats': formats,
+ }
+ # not sure where to go from here, _extract_m3u8_formats?
+ raise ExtractorError('%s retrieved m3u8 URL but not sure how to handle it: %s' % (self.IE_NAME, m3u8_url), expected=True)
+ else:
+ raise ExtractorError('%s said: %s, and we could not parse jw_setup' % (self.IE_NAME, get_text_attr(data, 'error')), expected=True)
+
files = data['files']
capfile = get_text_attr(data, 'capfile')This diff successfully downloads links that were failing for me. |
|
@AlbinoDrought I can confirm that your last patch works for me, with youtube-dl version 2020.03.08 on a C-SPAN user clip URL. |
|
@AlbinoDrought, I am using $ youtube-dl https://www.c-span.org/video/?471024-1/coronavirus-curve-bending-california-number-cases-slows
[CSpan] 471024: Downloading webpage
[CSpan] 544454: Downloading JSON metadata
ERROR: CSpan retrieved m3u8 URL but not sure how to handle it: https://m3u8-1.c-spanvideo.org/program/program.544454.m3u8 |
|
Hey @gurjeet , I kept an old non-working diff in my previous comment which might've been misleading. I've hidden it now. This is the working version of the diff - is it the one you applied? diff --git a/youtube_dl/extractor/cspan.py b/youtube_dl/extractor/cspan.py
index 68ba05844..0acd2a316 100644
--- a/youtube_dl/extractor/cspan.py
+++ b/youtube_dl/extractor/cspan.py
@@ -12,6 +12,7 @@ from ..utils import (
int_or_none,
smuggle_url,
unescapeHTML,
+ js_to_json,
)
from .senateisvp import SenateISVPIE
from .ustream import UstreamIE
@@ -142,7 +143,17 @@ class CSpanIE(InfoExtractor):
'http://www.c-span.org/assets/player/ajax-player.php?os=android&html5=%s&id=%s' % (video_type, video_id),
video_id)['video']
if data['@status'] != 'Success':
- raise ExtractorError('%s said: %s' % (self.IE_NAME, get_text_attr(data, 'error')), expected=True)
+ jw_setup_js = self._search_regex(
+ r'jwsetup\s*=\s*(\{[^;]+\});',
+ webpage, 'jwsetup config')
+ jw_setup = self._parse_json(
+ jw_setup_js, url, transform_source=js_to_json)
+ if jw_setup:
+ m3u8_url = jw_setup['playlist'][0]['sources'][0]['file']
+ # not sure where to go from here, _extract_m3u8_formats?
+ raise ExtractorError('%s retrieved m3u8 URL but not sure how to handle it: %s' % (self.IE_NAME, m3u8_url), expected=True)
+ else:
+ raise ExtractorError('%s said: %s, and we could not parse jw_setup' % (self.IE_NAME, get_text_attr(data, 'error')), expected=True)
doc = self._download_xml(
'http://www.c-span.org/common/services/flashXml.php?%sid=%s' % (video_type, video_id),and here is a testcase I am running with diff --git a/youtube_dl/extractor/cspan.py b/youtube_dl/extractor/cspan.py
index 67d6df4b0..68ba05844 100644
--- a/youtube_dl/extractor/cspan.py
+++ b/youtube_dl/extractor/cspan.py
@@ -73,6 +73,14 @@ class CSpanIE(InfoExtractor):
# Audio Only
'url': 'https://www.c-span.org/video/?437336-1/judiciary-antitrust-competition-policy-consumer-rights',
'only_matching': True,
+ }, {
+ 'url': 'https://www.c-span.org/video/?470351-1/president-trump-declares-national-emergency-response-coronavirus',
+ 'info_dict': {
+ 'id': '518828',
+ 'ext': 'mp4',
+ 'title': 'President Trump News Conference on Coronavirus Response',
+ 'description': 'President Trump held a news conference at the White House on the coronavirus.',
+ },
}]
BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/%s/%s_%s/index.html?videoId=%s'Would love if anyone could continue with this or update for the newer links This is the diff I'm using with diff --git a/youtube_dl/extractor/cspan.py b/youtube_dl/extractor/cspan.py
index 67d6df4b0..505892a15 100644
--- a/youtube_dl/extractor/cspan.py
+++ b/youtube_dl/extractor/cspan.py
@@ -12,6 +12,7 @@ from ..utils import (
int_or_none,
smuggle_url,
unescapeHTML,
+ js_to_json,
)
from .senateisvp import SenateISVPIE
from .ustream import UstreamIE
@@ -73,6 +74,17 @@ class CSpanIE(InfoExtractor):
# Audio Only
'url': 'https://www.c-span.org/video/?437336-1/judiciary-antitrust-competition-policy-consumer-rights',
'only_matching': True,
+ }, {
+ 'url': 'https://www.c-span.org/video/?470351-1/president-trump-declares-national-emergency-response-coronavirus',
+ 'info_dict': {
+ 'id': '543252',
+ 'ext': 'mp4',
+ 'title': 'President Trump News Conference on Coronavirus Response',
+ 'description': 'Speaking in the Rose Garden, President Trump declares the coronavirus pandemic a national emergency.',
+ },
+ 'params': {
+ 'skip_download': True, # m3u8 downloads
+ },
}]
BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/%s/%s_%s/index.html?videoId=%s'
@@ -130,12 +142,6 @@ class CSpanIE(InfoExtractor):
def get_text_attr(d, attr):
return d.get(attr, {}).get('#text')
- data = self._download_json(
- 'http://www.c-span.org/assets/player/ajax-player.php?os=android&html5=%s&id=%s' % (video_type, video_id),
- video_id)['video']
- if data['@status'] != 'Success':
- raise ExtractorError('%s said: %s' % (self.IE_NAME, get_text_attr(data, 'error')), expected=True)
-
doc = self._download_xml(
'http://www.c-span.org/common/services/flashXml.php?%sid=%s' % (video_type, video_id),
video_id)
@@ -145,6 +151,35 @@ class CSpanIE(InfoExtractor):
title = find_xpath_attr(doc, './/string', 'name', 'title').text
thumbnail = find_xpath_attr(doc, './/string', 'name', 'poster').text
+ data = self._download_json(
+ 'http://www.c-span.org/assets/player/ajax-player.php?os=android&html5=%s&id=%s' % (video_type, video_id),
+ video_id)['video']
+ if data['@status'] != 'Success':
+ jw_setup_js = self._search_regex(
+ r'jwsetup\s*=\s*(\{[^;]+\});',
+ webpage, 'jwsetup config')
+ jw_setup = self._parse_json(
+ jw_setup_js, video_id, transform_source=js_to_json)
+ if jw_setup:
+ m3u8_url = jw_setup['playlist'][0]['sources'][0]['file']
+ formats = self._extract_m3u8_formats(
+ m3u8_url, video_id, 'mp4',
+ entry_protocol='m3u8_native', m3u8_id='hls', fatal=False
+ )
+ if formats:
+ for format in formats:
+ format['http_headers'] = {'Referer': 'https://c-span.org/'}
+ return {
+ 'id': video_id,
+ 'title': title,
+ 'description': description,
+ 'formats': formats,
+ }
+ # not sure where to go from here, _extract_m3u8_formats?
+ raise ExtractorError('%s retrieved m3u8 URL but not sure how to handle it: %s' % (self.IE_NAME, m3u8_url), expected=True)
+ else:
+ raise ExtractorError('%s said: %s, and we could not parse jw_setup' % (self.IE_NAME, get_text_attr(data, 'error')), expected=True)
+
files = data['files']
capfile = get_text_attr(data, 'capfile')For what it's worth, that link successfully downloads in my patched version |
|
@AlbinoDrought, please disregard my complaint. As is apparent to me now, I was using a very old version of youtube-dl; in my defense, though, this is the latest version provided by Nixpkgs on macOS. I now installed youtube-dl via Homebrew which provided version 2020.03.24. This version worked in the example I provided above, without any patches. Thanks for following up! My spam filter caught the Github notification of your response, so I saw your response very late. |
|
youtube-dl 2020.06.06 |
|
Same here, anybody have any ideas? Confirmed using youtube-dl --version 2020.06.16.1
|
|
Currently experiencing this with I'm able to successfully grab about 1/4 of the videos I want. 1/4 give 403's, the rest give the usual What's weird is it seems both the working and non-working ones initialize in the same way, at least when viewed through Network tab in DevTools - GET to Raw curl of the ajax endpoint gives Wild guess, some videos have a more accessible endpoint exposed through the ajax endpoint? Not sure how much can be further obtained through that endpoint, may be worthwhile to look into emulating the xhr chunk GETs. |
|
I fixed this on my machine by setting the |
Make sure you are using the latest version: run
youtube-dl --versionand ensure your version is 2019.01.10. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.Before submitting an issue make sure you have:
What is the purpose of your issue?
If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:
Description of your issue, suggested solution and other information
Youtube-dl can no longer download some C-SPAN videos. I'm not sure if this is an issue with youtube-dl, or with C-SPAN itself, but videos do continue to work in the browser. I've also noticed that this seems to only be effecting newer videos (posted within the last few weeks). I almost wonder if it's related to the government shutdown? In any case, thanks for taking a look!
Some videos that work:
Some that don't work: