New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TheScene] Fixed Extraction #12245

Merged
merged 1 commit into from Feb 24, 2017

Conversation

Projects
None yet
2 participants
@ThomasChr
Copy link
Contributor

ThomasChr commented Feb 24, 2017

Please follow the guide below

  • You will be asked some questions, please read them carefully and answer honestly
  • Put an x into all the boxes [ ] relevant to your pull request (like that [x])
  • Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

  • I am the original author of this code and I am willing to release it under Unlicense
  • I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

  • Bug fix
  • Improvement
  • New extractor
  • New feature

Description of your pull request and other information

Explanation of your pull request in arbitrary form goes here. Please make sure the description explains the purpose and effect of your pull request and is worded well enough to be understood. Provide as much context and examples as possible.

Fix for Issue #12235

@ThomasChr ThomasChr referenced this pull request Feb 24, 2017

Closed

[TheScene] ERROR: Unable to extract info json #12235

4 of 8 tasks complete

@dstftw dstftw merged commit 51ed496 into rg3:master Feb 24, 2017

1 check failed

continuous-integration/travis-ci/pr The Travis CI build failed
Details

dstftw added a commit that referenced this pull request Mar 4, 2017

Dedsec1 added a commit to Dedsec1/youtube-dl that referenced this pull request Mar 18, 2017

Update from upstream repo rg3/youtube-dl (#1)
* [pornhub] Extract video URL from tv platform site (rg3#12007, rg3#12129)

* [vod.pl] Add new extractor

* [vodpl] Make more robust and add another test (closes rg3#12122)

* [onetmvp] Add shortcut extractor

* [onetpl] Add support for onet.pl (closes rg3#10507)

* [onetpl] Add support for businessinsider.com.pl and plejada.pl

* [pinkbike] Fix uploader extraction (closes rg3#12054)

* [amcnetworks] fix extraction(closes rg3#12127)

* [utils] Add support for quoted string literals in --match-filter (closes rg3#8050, closes rg3#12142, closes rg3#12144)

* [test_YoutubeDL] Fix invalid escape sequences

* [devscripts/run_tests.sh] Change permission for script to 755

* [ceskatelevize] Prefix format ids

* [ceskatelevize] Lower priority for audio description sources (rg3#12119)

* [ChangeLog] Actualize

* release 2017.02.16

* [elpais] Fix typo and improve extraction (closes rg3#12139)

* [generic] Try parsing JWPlayer embedded videos (closes rg3#12030)

* [einthusan] Relax _VALID_URL (closes rg3#12141, closes rg3#12159)

* [theplatform] Recognize URLs with whitespaces (closes rg3#12044)

* [openload] Semifix extraction (closes rg3#10408)

just updated the code. i don't do much python still i tried to convert my code. lemme know if there is any prob with it

* [openload] Fix extraction (closes rg3#12002)

* [ellentv] Improve

* [heise] Support videos embedded in any article.

* [heise] Improve (closes rg3#9725)

* [options] Add missing spaces

* [ChangeLog] Actualize

* release 2017.02.17

* [viceland] improve info extraction and update test

* [metacafe] Bypass family filter

If you don't send this user=ffilter: false cookie, it will 301 redirect you to a page asking about it, and then the title check will fail.

* [metacafe] Improve (closes rg3#10371)

* [thisav] Add support for html5 media (closes rg3#11771)

* [tvn24] Add extractor

* [tvn24] Improve extraction (closes rg3#11679)

* [options] Mention quoted string literals for --match-filter

* [brightcove:legacy] restrict videoPlayer value(closes rg3#12040)

* [utils] Introduce YoutubeDLError base class for all youtube-dl exceptions

* Add experimental geo restriction bypass mechanism
Based on faking X-Forwarded-For HTTP header

* [dramafever] Improve geo restriction detection and use geo bypass mechanism

* [go] Improve geo restriction detection and use geo bypass mechanism

* [itv] Improve geo restriction detection and use geo bypass mechanism

* [nrk] Improve geo restriction detection and use geo bypass mechanism

* [ondemandkorea] Improve geo restriction detection and use geo bypass mechanism

* [pbs] Improve geo restriction detection and use geo bypass mechanism

* [svt] Improve geo restriction detection and use geo bypass mechanism

* [vbox7] Improve geo restriction detection and use geo bypass mechanism

* [srgssr] Improve geo restriction detection

* [vgtv] Improve geo restriction detection

* [viki] Improve geo restriction detection

* [utils] Make random_ipv4 return unicode string

* Add faked X-Forwarded-For to formats' HTTP headers

* Rename bypass geo restriction options

* Improve geo bypass mechanism
* Rename options to preffixly match with --geo-verification-proxy
* Introduce _GEO_COUNTRIES for extractors
* Implement faking IP right away for sites with known geo restriction

* [options] Move geo restriction related options to separate section

* [downloader/dash] Honor HTTP headers when downloading fragments
For example, https://www.oppetarkiv.se/video/1196142/natten-ar-dagens-mor

* [downloader/ism] Honor HTTP headers when downloading fragments

* [tv4] Switch to hls3 protocol (closes rg3#12177)

* [tv4] Bypass geo restriction and improve detection

* [travis] Run tests in parallel

[test_download] Print test names in case of network errors

[test_download] Add comments for nose parameters

[test_download] Modify outtmpl to prevent info JSON filename conflicts

Thanks @jaimeMF for the idea.

[travis] Only download tests should be run in parallel

* [nrk] Update _API_HOST and relax _VALID_URL

* [prosiebensat1] Throw ExtractionError on unsupported page type (closes rg3#12180)

* [nrk] PEP 8

* [svt] PEP 8

* [commonmistakes] Disable UnicodeBOM extractor test for python 3.2

* [iprima] Modernize

* [iprima] Improve geo restriction detection and disable geo bypass

* [extractor/common] Emphasize geo bypass APIs are experimental

* [spankbang] Make uploader optional (closes rg3#12193)

* [adobepass] add support for Time Warner Cable(closes rg3#12191)

* [ninecninemedia] use geo bypass mechanism

* [extractor/common] Allow calling _initialize_geo_bypass from extractors (rg3#11970)

* [extractor/common] Fix typo

* [extractor/common] Do not quit _initialize_geo_bypass on empty countries

* [extractor/common] Print origin country for fake IP

* [ChangeLog] Actualize

* release 2017.02.21

* [limelight] extract PlaylistService errors

* [telequebec] use geo bypass mechanism

* [cbc] use geo bypass mechanism

* [tfo] Improve geo restriction detection and use geo bypass mechanism

* Revert "[cbc] use geo bypass mechanism"

This reverts commit 86466a8.

* [viewster] use geo verifcation headers

* [crackle] use geo bypass mechanism

* [setup] Add python 3.6 classifier

* [setup] Actualize maintainer info

* [24video] Add support for 24video.tube (closes rg3#12217)

* [vidzi] Add support for vidzi.cc

* [vidzi] Add test for rg3#12213

* [dailymotion] Make comment count optional (closes rg3#12209)
Not served anymore

* [crunchyroll] Fix descriptions with double quotes (closes rg3#12124)

* [ChangeLog] Actualize

* release 2017.02.22

* [leeco] raise GeoRestrictedError and use geo bypass mechanism

* [sohu] raise GeoRestrictedError

* [mgtv] fix extraction

* [crunchyroll] extract playlist entries ids

* [instagram] Add support for multi video posts (closes rg3#12226)

* [skylinewebcams] Add extractor (closes rg3#12221)

* [devscripts/run_tests] Exclude youtube lists tests from core build

* [compat] Add compat_numeric_types

* [YoutubeDL] Add support for string formatting operations in output template

* [YoutubeDL] Rewrite outtmpl for playlist_index and autonumber for backward compatibility

* [YoutubeDL] Add more numeric fields for NA substitution in outtmpl

* [options] Deprecate --autonumber-size

* [README.md] Clarify sequence types in output template and document numeric string formatting operations

* [go] sign all uplynk urls and use geo bypass only for free videos(closes rg3#12087)(closes rg3#12210)

* [devscripts/make_lazy_extractors] Fix making lazy extractors on python 3 under Windows

* [postprocessor/ffmpeg] Add mising space (closes rg3#12232)

* [lynda:course] Add webpage extraction fallback (closes rg3#12238)

* Suppress help for all deprecated options and print warning when used

* [ChangeLog] Actualize

* release 2017.02.24

* [ivi] Raise GeoRestrictedError

* [openload] fix extraction(closes rg3#10408)

* [tubitv] use geo bypass mechanism

* [thescene] Fix extraction (closes rg3#12235)

* [thescene] Extract more metadata

* [noco] Swtich login URL to https (closes rg3#12246)

* [noco] Modernize

* [ChangeLog] Actualize

* release 2017.02.24.1

* [amcnetworks] Relax _VALID_URL (rg3#12127)

* [extractor/common] Add 'preference' to _parse_html5_media_entries

Some websites, like NJPWorld, put different qualities on different
player pages.

* [njpwworld] Add new extractor (closes rg3#11561)

* [compat] Use try except for compat_numeric_types

* [etonline] Add extractor (closes rg3#12236)

* [xhamster] Capture and output videoClosed error (rg3#12263)

* [downloader/common] Limit displaying 2 digits after decimal point in sleep interval message

* [freshlive] Add extractor

* [freshlive] Fix issues and improve (closes rg3#12175)

* [vevo] Fix videos with the new streams/streamsV3 format (closes rg3#11719)

* [tvigle] Raise GeoRestrictedError

* [MDR] Relax _VALID_URL and playerURL matching and update _TESTS

Ref: rg3#12169

* [dailymotion] Raise GeoRestrictedError

* [youtube] Raise GeoRestrictedError

* [crunchyroll] Extract uploader name that's not a link

Provide the Crunchyroll extractor with the ability to extract uploader
names that aren't links. Add a test for this new functionality.
This fixes rg3#12267.

* [cda] Decode URL (fixes rg3#12255)

* [scivee] Remove extractor (rg3#9315)

The Wikipedia page is changed from active to down:
https://en.wikipedia.org/w/index.php?title=SciVee&diff=prev&oldid=723161154

Some other interesting bits:

$ nslookup www.scivee.tv
Server:         8.8.8.8
Address:        8.8.8.8#53

Non-authoritative answer:
www.scivee.tv   canonical name = scivee.rcsb.org.
Name:   scivee.rcsb.org
Address: 132.249.231.211

$ nslookup rcsb.org
Server:         8.8.8.8
Address:        8.8.8.8#53

Non-authoritative answer:
Name:   rcsb.org
Address: 132.249.231.77

Both IPs are from UCSD. I guess it's maintained by a lab and they don't
maintain it anymore.

* [youtube:channel] Remove duplicate test

* [npo] Add support for hetklokhuis.nl (closes rg3#12293)

* [npo] Add support for zapp.nl

* [hetklokhuis] Fix IE_NAME

* [npo] Relax _VALID_URL for zapp.nl

* [ChangeLog] Actualize

* release 2017.02.27

* [douyu] Fix extraction and update _TESTS

They've switched from flv to hls

Closes rg3#12301

* [aes] Add aes_cbc_encrypt

Used in daisuki.net (rg3#4738)

* [utils] Add pkcs1pad

Used in daisuki.net (rg3#4738)

* [utils] Add bytes_to_long() and long_to_bytes()

Used in daisuki.net (rg3#4738)

Both are adapted from public domain PyCrypto:
https://github.com/dlitz/pycrypto/blob/master/lib/Crypto/Util/number.py

* [daisuki] Add new extractor (closes rg3#4738)

* [daisuki] Add subtitles (rg3#4738)

* [youtube:playlist] Recognize another playlist pattern (closes rg3#11928, closes rg3#12286)

* [azmedien:showplaylist] Add support for all episodes playlists

* [azmedien:showplaylist] Improve (closes rg3#12160)

* [ChangeLog] Actualize

* release 2017.02.28

* [npo] Adapt to app.php API (closes rg3#12311)

* [npo] Improve extraction and update tests

* [npo] Add support for audio

* [YoutubeDL] Don't sanitize identifiers (closes rg3#12317)

* [utils] Carry long doc string

* [adobepass] Add Charter Spectrum (rg3#11465)

Thanks @tv21 for the fix!

* [youtube] Mark errors about rental videos as expected

Closes rg3#12324

* [facebook] Fix extraction (closes rg3#12323)

Almost all videos now use the pagelet type 'permalink_video_pagelet'

* [ChangeLog] Actualize

* release 2017.03.02

* [ruutu] Disable DASH formats (closes rg3#12322)
Due to causing out of sync issue

* [24video] Use original host for requests (closes rg3#12339)

* [go] Relax _VALID_URL (closes rg3#12341)

* [brightcove:new] Raise GeoRestrictedError

* [vrak] Add extractor

* [brightcove:new] Add ability to smuggle geo_countries into URL

* [vrak] Improve and update test (closes rg3#11452)

* [rutube] Relax _VALID_URL

* [generic] Add support for rutube embeds

* [douyutv] Switch to the PC API to escape the 5-min limitation

Thanks @spacemeowx2 for the algo.

Ref: https://gist.github.com/spacemeowx2/629b1d131bd7e240a7d28742048e80fc

Closes rg3#12316

* [redbulltv] Add extractor

* [redbulltv] Improve extraction (closes rg3#11948, closes rg3#3919)

* [vier] Add support for vijf.be

vier.be and vijf.be run on the same CMS and are property of the same company,
so the same extractor can be used for both of them.

* Credit @ThomasChr for rg3#12015 and rg3#12245

* Credit @p2004a for vodpl (rg3#12122)

* Credit @TobiX for rg3#9725

* Credit @obilodeau for vrak (rg3#11452)

* Credit @vierbergenlars for vijf.be (rg3#12304)

* [twitch] Add basic support for two-factor authentication

* [twitch] Use better naming and simplify (closes rg3#11974)

* [ChangeLog] Actualize

* release 2017.03.05

* [external:ffmpeg] In test harness, limit to 10k download size

Otherwise, if you screw up a playlist test by including a playlist
dictionary key, you'll be there for eons while it downloads all the
files before erroring out.

* [downloader/external] Add missing import and PEP8

* [extractor/common] Move jwplayer formats extraction in separate method

* [extractor/common] Improve height extraction and extract bitrate

* [extractor/common] Pass arguments to _parse_jwplayer_formats and PEP8

* [tunepk] Add extractor (closes rg3#12197, closes rg3#12243)

* [addanime] Skip an invalid test

* [drtv:live] Bypass geo restriction

* [drtv] Add geo countries to GeoRestrictedError

* [utils] Process bytestrings in urljoin (closes rg3#12369)

* [ChangeLog] Actualize

* release 2017.03.06

* Openload fix extraction (rg3#12357)

* Fix extraction

* [ChangeLog] Update after rg3#12357

* [__init__] Metadata should be added after conversion

Fixes rg3#5594

* [soundcloud] Update client id (closes rg3#12376)

* [ChangeLog] Actualize

* release 2017.03.07

* Credit @benages for redbull.tv (rg3#11948)

* Credit @puxlit for twitch 2fa (rg3#11974)

* [twitch] PEP8

* [brightcove:legacy] Relax videoPlayer validation check (closes rg3#12381)

* [extractor/generic] Add test for brigthcove UUID-like videoPlayer

* [openload] Fix extraction

* [telequebec] Fix description extraction and update test (closes rg3#12399)

* [miomio] Fix extraction

Closes rg3#12291
Closes rg3#12388
Closes rg3#12402

* [dplayit] Separate and rewrite extractor and bypass geo restriction (closes rg3#12393)

* [dplayit] Relax playback info URL extraction

* [prosiebensat1] Improve title extraction (closes rg3#12318)

* [prosiebensat1] Improve title extraction and add test

* [wdr:maus] Fix extraction and update tests

* [wdr] Actualize comment

* [extractor/generic] Make title optional for jwplayer embeds (closes rg3#12410)

* [ChangeLog] Actualize

* release 2017.03.10

* [__init__] Fix missing subtitles if --add-metadata is used (rg3#12423)

The previous fix for rg3#5594 is incorrect

* [discoverygo:playlist] Add extractor (closes rg3#12424)

* [discoverygo] Bypass geo restriction

* [discoverygo] Actualize test

* [crunchyroll] Extract season name

* [streamable] Update API URL

* [openload] Fix extraction

Just a minor fix for openload

* [mitele] Add support for ooyala videos (closes rg3#12430)

* [telecinco] Add test for rg3#12430

* [facebook] Make title optional (closes rg3#12443)

* [Openload] Fixed Extraction

They did changed it again.

* [ChangeLog] Actualize

* release 2017.03.15

* [redbull] improve extraction

- extract 1080p quality
- correct ttml subtitle ext
- catch api errors
- reduce request size

* [extractor/generic] Add forgotten return for jwplayer formats

* [extractor/common] Extract SMIL formats from jwplayer

* [postprocessor/ffmpeg] Add support for flac

Requested at http://stackoverflow.com/q/42828041/35070

* [options] Mention flac support and sort alphabetically among the audio formats

* [ChangeLog] Actualize

* release 2017.03.16

* [test_InfoExtractor] Add some realworld tests for _extract_jwplayer_data

* [openload] Fix extraction (closes rg3#10408)

Thanks to @makgun02

Ref: http://pastebin.com/raw/JX9gHFUz

* [discoverynetworks] add support for more domains and bypass geo restiction

* [vtm] Add extractor (closes rg3#9974)

Implementation of the approach described in rg3#9974.

* [medialaan] Fix and improve extraction (closes rg3#11912)

* [openload] Fix extraction

* [adobepass] Detect and output error on authz token extraction (rg3#12472)

* [medialaan] Remove unrelated test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment