Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Check for oreilly login with new url #31524

Open
wants to merge 16 commits into
base: master
Choose a base branch
from
Open

Check for oreilly login with new url #31524

wants to merge 16 commits into from

Conversation

epsilonSpider
Copy link

Resolves #30884

Before submitting a pull request make sure you have:

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

  • I am the original author of this code and I am willing to release it under Unlicense
  • I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

  • Bug fix
  • Improvement
  • New extractor
  • New feature

Description of your pull request and other information

This pull request resolves the issue seen in #30884. With this change, the following types of URLs should function again:

This was tested with oreilly account credentials passed via --username and --password.

dirkf and others added 14 commits February 9, 2023 11:01
@dirkf
[YouTube] Bypass age-gating for certain restricted videos
b14b33a 
* Use TVHTML5_SIMPLY_EMBEDDED_PLAYER client

* Also add and fix tests

* Introduce and use new utility function `update_url()`
@dirkf
[YouTube] Add signatureTimestamp for age-gate bypass
2be0cd2
@dirkf
[test] Fix TestAgeRestriction
ce81ae3 
* age restriction may cause DownloadError
* update obsolete test URLs
[skip ci]
@dirkf
[compat] Systematise compat_ naming
9ca224b 
[skip ci]
@dirkf
[utils] Add parse_qs, update_url
a6f7d10 
[skip ci]
@dirkf
[compat] Update test_compat
b337af9 
[skip ci]
@dirkf
[YouTube] Refresh compat/utils usage
f640916 
* import parse_qs()
* import parse_qs in lazy_extractors (clears old TODO)
* clean up old compiled lazy_extractors for Py2
* use update_url()
@dirkf
[YouTube] Fix tests
bcf597e
@Valentin-Metz @dirkf
[rbgtum] Add new extractor (ytdl-org#31305)
80c0228 
* [rbgtum] Add new extractor

* Small update, force CI

---------

Co-authored-by: dirkf <fieldhouse@gmx.net>
@teddy171 @dirkf
[feat]: Add support to external downloader aria2p (ytdl-org#31500)
9256b02 
* feat: add class Aria2pFD

* feat: create call_downloader function

* feat: a colorful download interface to aria2pFD

* feat: change value name

* Apply suggestions from code review

Co-authored-by: dirkf <fieldhouse@gmx.net>

* Typo in suggestion

* fix: remove unused value

* fix: add not function to return value(0 is normal); add total_seconds to download.eta(timedelta object); add waiting status when hook progress

* fix: remove unuse method ..utils.format_bytes

* fix: be up to flake8

* fix: be up to flake8

* Apply suggestions from code review

* [feat] test external downloader aria2p

* [feat] test external downloader aria2p

* [fix] test_external_downloader.py

* Apply suggestions from code review

Co-authored-by: dirkf <fieldhouse@gmx.net>

* Apply suggestions from code review

Co-authored-by: dirkf <fieldhouse@gmx.net>

* Update test/test_external_downloader.py

Co-authored-by: dirkf <fieldhouse@gmx.net>

* Update test/test_external_downloader.py

Co-authored-by: dirkf <fieldhouse@gmx.net>

* Update youtube_dl/downloader/external.py

Co-authored-by: dirkf <fieldhouse@gmx.net>

* refactoring code and fix bugs

* Apply suggestions from code review

* Rename test_external_downloader.py to test_downloader_external.py

---------

Co-authored-by: dirkf <fieldhouse@gmx.net>
@fonkap @dirkf
[FileMoonIE] Add extractor for filemoon.sx (ytdl-org#31515)
cf5fb7e 
---------

Co-authored-by: dirkf <fieldhouse@gmx.net>
@fonkap @dirkf
[KommunetvIE] Add extractor for kommunetv.no (ytdl-org#31516)
99c3cf6 
* Add extractor for kommunetv.no
* Using utils.update_url instead of regex

---------

Co-authored-by: dirkf <fieldhouse@gmx.net>
@fonkap @dirkf
[StreamsbIE] Add extractor for streamsb.com (viewsb.com) (ytdl-org#31517
70a4a8b 
)

* Add extractor for streamsb.com (viewsb.com)

* make data url using app.js version

---------

Co-authored-by: dirkf <fieldhouse@gmx.net>
@epsilonSpider
Check for oreilly login with new url
d524ac1 
Resolves ytdl-org#30884
dirkf
dirkf requested changes Feb 13, 2023
View reviewed changes
Copy link
Contributor

@dirkf dirkf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this looks like the answer to a long-standing issue.

Please post a verbose log showing this in action, since it can't be tested in the CI, or by devs without a login.

youtube_dl/extractor/safari.py Outdated
Comment on lines 41 to 42
return ('https://learning.oreilly.com/member/login/' == urlh.geturl()
or 'learning.oreilly.com/home/' in urlh.geturl())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be more in keeping with the previous code:

Suggested change
return ('https://learning.oreilly.com/member/login/' == urlh.geturl()
or 'learning.oreilly.com/home/' in urlh.geturl())
return any('learning.oreilly.com/%s/' % x in urlh.geturl() for x in ('member/login', 'home'))

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Depending on how oreilly is accessed, the URL can end up being "https://learning.oreilly.com/member/login/?next=[snip]" for cases where the user is not yet logged in. So we don't want to catch that case here with the in instead of ==.

Given that, is there a more conventional way to format the check being done here?

Copy link
Contributor

@dirkf dirkf Mar 31, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe

            return bool(re.search(r'learning.oreilly.com/(?:%s)' % '|'.join(('member/login/$', 'home/')), urlh.geturl()))

or

            parsed_url = compat_urlparse.urlparse(url)
            return parsed_url.hostname.endswith('learning.oreilly.com') and (
                parsed_url.path.startswith('/home/')
                or (parsed_url.path == '/member/login' and not parsed_url.query))

Pick either of these or your original, check your test, and then I think it's good to go.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Committed your second suggestion.

Here are two logs showing successful downloads. The first is for a course with multiple videos. This is the case that was broken before this pull request. The second is for a specific video to ensure that still works.

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', '-u', 'PRIVATE', '-p', 'PRIVATE', 'https://learning.oreilly.com/videos/design-patterns-in/9781491935828/']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: 899af2ef6
[debug] Python version 3.10.10 (CPython) - macOS-13.2.1-arm64-64bit
[debug] exe versions: none
[debug] Proxy map: {}
[safari:course] Downloading login page
[safari:course] Logging in
[safari:course] Completing login
[safari:course] 9781491935828: Downloading course JSON
[download] Downloading playlist: Design Patterns in the Real World, an Analysis-Based Approach
[safari:course] playlist Design Patterns in the Real World, an Analysis-Based Approach: Collected 22 video ids (downloading 22 of them)
[download] Downloading video 1 of 22
[safari:api] Downloading login page
[safari:api] 9781491935828/video226613: Downloading part JSON
[safari] Downloading login page
[safari] 9781491935828-video226613: Downloading webpage
[safari] 9781491935828-video226613: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226613: Downloading webpage
[Kaltura] 0_g3w6syuh: Downloading video info JSON
[Kaltura] 0_g3w6syuh: Checking mp4-5170 URL
[Kaltura] 0_g3w6syuh: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Introduction-0_g3w6syuh.mp4 has already been downloaded

[download] 100% of 274.18MiB
[download] Downloading video 2 of 22
[safari:api] 9781491935828/video226617: Downloading part JSON
[safari] 9781491935828-video226617: Downloading webpage
[safari] 9781491935828-video226617: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226617: Downloading webpage
[Kaltura] 0_2r8msk12: Downloading video info JSON
[Kaltura] 0_2r8msk12: Checking mp4-4407 URL
[Kaltura] 0_2r8msk12: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] What Is A Design Pattern-0_2r8msk12.mp4 has already been downloaded

[download] 100% of 914.23MiB
[download] Downloading video 3 of 22
[safari:api] 9781491935828/video226618: Downloading part JSON
[safari] 9781491935828-video226618: Downloading webpage
[safari] 9781491935828-video226618: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226618: Downloading webpage
[Kaltura] 0_4iuhnj0j: Downloading video info JSON
[Kaltura] 0_4iuhnj0j: Checking mp4-5105 URL
[Kaltura] 0_4iuhnj0j: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Template Method -0_4iuhnj0j.mp4 has already been downloaded

[download] 100% of 463.59MiB
[download] Downloading video 4 of 22
[safari:api] 9781491935828/video226619: Downloading part JSON
[safari] 9781491935828-video226619: Downloading webpage
[safari] 9781491935828-video226619: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226619: Downloading webpage
[Kaltura] 0_frguhxo1: Downloading video info JSON
[Kaltura] 0_frguhxo1: Checking mp4-3556 URL
[Kaltura] 0_frguhxo1: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Builder - Separating the UI from the Domain Object-0_frguhxo1.mp4 has already been downloaded

[download] 100% of 590.87MiB
[download] Downloading video 5 of 22
[safari:api] 9781491935828/video226620: Downloading part JSON
[safari] 9781491935828-video226620: Downloading webpage
[safari] 9781491935828-video226620: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226620: Downloading webpage
[Kaltura] 0_0stusjis: Downloading video info JSON
[Kaltura] 0_0stusjis: Checking mp4-5019 URL
[Kaltura] 0_0stusjis: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Singleton and Utility-0_0stusjis.mp4 has already been downloaded

[download] 100% of 1.79GiB
[download] Downloading video 6 of 22
[safari:api] 9781491935828/video226621: Downloading part JSON
[safari] 9781491935828-video226621: Downloading webpage
[safari] 9781491935828-video226621: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226621: Downloading webpage
[Kaltura] 0_5raty95u: Downloading video info JSON
[Kaltura] 0_5raty95u: Checking mp4-4877 URL
[Kaltura] 0_5raty95u: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Abstract Factory-0_5raty95u.mp4 has already been downloaded

[download] 100% of 500.54MiB
[download] Downloading video 7 of 22
[safari:api] 9781491935828/video226647: Downloading part JSON
[safari] 9781491935828-video226647: Downloading webpage
[safari] 9781491935828-video226647: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226647: Downloading webpage
[Kaltura] 0_3at32aa3: Downloading video info JSON
[Kaltura] 0_3at32aa3: Checking mp4-4586 URL
[Kaltura] 0_3at32aa3: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Class and Abstract Adapters-0_3at32aa3.mp4 has already been downloaded

[download] 100% of 422.74MiB
[download] Downloading video 8 of 22
[safari:api] 9781491935828/video226623: Downloading part JSON
[safari] 9781491935828-video226623: Downloading webpage
[safari] 9781491935828-video226623: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226623: Downloading webpage
[Kaltura] 0_ct40r8x4: Downloading video info JSON
[Kaltura] 0_ct40r8x4: Checking mp4-4591 URL
[Kaltura] 0_ct40r8x4: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Decorator-0_ct40r8x4.mp4 has already been downloaded

[download] 100% of 505.65MiB
[download] Downloading video 9 of 22
[safari:api] 9781491935828/video226624: Downloading part JSON
[safari] 9781491935828-video226624: Downloading webpage
[safari] 9781491935828-video226624: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226624: Downloading webpage
[Kaltura] 0_5d9gg0t5: Downloading video info JSON
[Kaltura] 0_5d9gg0t5: Checking mp4-4875 URL
[Kaltura] 0_5d9gg0t5: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Bridge-0_5d9gg0t5.mp4 has already been downloaded

[download] 100% of 286.61MiB
[download] Downloading video 10 of 22
[safari:api] 9781491935828/video226603: Downloading part JSON
[safari] 9781491935828-video226603: Downloading webpage
[safari] 9781491935828-video226603: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226603: Downloading webpage
[Kaltura] 0_69jm0swk: Downloading video info JSON
[Kaltura] 0_69jm0swk: Checking mp4-5247 URL
[Kaltura] 0_69jm0swk: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Facade-0_69jm0swk.mp4 has already been downloaded

[download] 100% of 258.80MiB
[download] Downloading video 11 of 22
[safari:api] 9781491935828/video226604: Downloading part JSON
[safari] 9781491935828-video226604: Downloading webpage
[safari] 9781491935828-video226604: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226604: Downloading webpage
[Kaltura] 0_cygkp3d6: Downloading video info JSON
[Kaltura] 0_cygkp3d6: Checking mp4-5115 URL
[Kaltura] 0_cygkp3d6: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Prototype and Composite-0_cygkp3d6.mp4 has already been downloaded

[download] 100% of 674.01MiB
[download] Downloading video 12 of 22
[safari:api] 9781491935828/video227094: Downloading part JSON
[safari] 9781491935828-video227094: Downloading webpage
[safari] 9781491935828-video227094: Downloading kaltura session JSON
[Kaltura] 9781491935828-video227094: Downloading webpage
[Kaltura] 0_tz0yk85g: Downloading video info JSON
[Kaltura] 0_tz0yk85g: Checking mp4-4628 URL
[Kaltura] 0_tz0yk85g: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Flyweight and Flyweight Pool-0_tz0yk85g.mp4 has already been downloaded

[download] 100% of 611.50MiB
[download] Downloading video 13 of 22
[safari:api] 9781491935828/video226606: Downloading part JSON
[safari] 9781491935828-video226606: Downloading webpage
[safari] 9781491935828-video226606: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226606: Downloading webpage
[Kaltura] 0_kzkuhyzz: Downloading video info JSON
[Kaltura] 0_kzkuhyzz: Checking mp4-5069 URL
[Kaltura] 0_kzkuhyzz: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Proxy-0_kzkuhyzz.mp4 has already been downloaded

[download] 100% of 793.54MiB
[download] Downloading video 14 of 22
[safari:api] 9781491935828/video227136: Downloading part JSON
[safari] 9781491935828-video227136: Downloading webpage
[safari] 9781491935828-video227136: Downloading kaltura session JSON
[Kaltura] 9781491935828-video227136: Downloading webpage
[Kaltura] 0_haawrt9w: Downloading video info JSON
[Kaltura] 0_haawrt9w: Checking mp4-5121 URL
[Kaltura] 0_haawrt9w: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Command -0_haawrt9w.mp4 has already been downloaded

[download] 100% of 866.30MiB
[download] Downloading video 15 of 22
[safari:api] 9781491935828/video226608: Downloading part JSON
[safari] 9781491935828-video226608: Downloading webpage
[safari] 9781491935828-video226608: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226608: Downloading webpage
[Kaltura] 0_tgp5h7g9: Downloading video info JSON
[Kaltura] 0_tgp5h7g9: Checking mp4-4897 URL
[Kaltura] 0_tgp5h7g9: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Strategy-0_tgp5h7g9.mp4 has already been downloaded

[download] 100% of 488.89MiB
[download] Downloading video 16 of 22
[safari:api] 9781491935828/video227096: Downloading part JSON
[safari] 9781491935828-video227096: Downloading webpage
[safari] 9781491935828-video227096: Downloading kaltura session JSON
[Kaltura] 9781491935828-video227096: Downloading webpage
[Kaltura] 0_tu2f1fy6: Downloading video info JSON
[Kaltura] 0_tu2f1fy6: Checking mp4-4877 URL
[Kaltura] 0_tu2f1fy6: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] The Chain of Responsibility-0_tu2f1fy6.mp4 has already been downloaded

[download] 100% of 1.06GiB
[download] Downloading video 17 of 22
[safari:api] 9781491935828/video226610: Downloading part JSON
[safari] 9781491935828-video226610: Downloading webpage
[safari] 9781491935828-video226610: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226610: Downloading webpage
[Kaltura] 0_eoczuk3i: Downloading video info JSON
[Kaltura] 0_eoczuk3i: Checking mp4-3958 URL
[Kaltura] 0_eoczuk3i: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Iterators (Internal and External)-0_eoczuk3i.mp4 has already been downloaded

[download] 100% of 580.50MiB
[download] Downloading video 18 of 22
[safari:api] 9781491935828/video226611: Downloading part JSON
[safari] 9781491935828-video226611: Downloading webpage
[safari] 9781491935828-video226611: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226611: Downloading webpage
[Kaltura] 0_wmf5wiha: Downloading video info JSON
[Kaltura] 0_wmf5wiha: Checking mp4-5116 URL
[Kaltura] 0_wmf5wiha: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Observer-0_wmf5wiha.mp4 has already been downloaded

[download] 100% of 1.32GiB
[download] Downloading video 19 of 22
[safari:api] 9781491935828/video226612: Downloading part JSON
[safari] 9781491935828-video226612: Downloading webpage
[safari] 9781491935828-video226612: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226612: Downloading webpage
[Kaltura] 0_2gwh0mzi: Downloading video info JSON
[Kaltura] 0_2gwh0mzi: Checking mp4-5100 URL
[Kaltura] 0_2gwh0mzi: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Mediator-0_2gwh0mzi.mp4 has already been downloaded

[download] 100% of 580.33MiB
[download] Downloading video 20 of 22
[safari:api] 9781491935828/video226614: Downloading part JSON
[safari] 9781491935828-video226614: Downloading webpage
[safari] 9781491935828-video226614: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226614: Downloading webpage
[Kaltura] 0_vjs3vbtx: Downloading video info JSON
[Kaltura] 0_vjs3vbtx: Checking mp4-2900 URL
[Kaltura] 0_vjs3vbtx: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Memento-0_vjs3vbtx.mp4 has already been downloaded

[download] 100% of 354.84MiB
[download] Downloading video 21 of 22
[safari:api] 9781491935828/video226615: Downloading part JSON
[safari] 9781491935828-video226615: Downloading webpage
[safari] 9781491935828-video226615: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226615: Downloading webpage
[Kaltura] 0_nomyq1zy: Downloading video info JSON
[Kaltura] 0_nomyq1zy: Checking mp4-4176 URL
[Kaltura] 0_nomyq1zy: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] State-0_nomyq1zy.mp4 has already been downloaded

[download] 100% of 402.77MiB
[download] Downloading video 22 of 22
[safari:api] 9781491935828/video226616: Downloading part JSON
[safari] 9781491935828-video226616: Downloading webpage
[safari] 9781491935828-video226616: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226616: Downloading webpage
[Kaltura] 0_n6g3eyfo: Downloading video info JSON
[Kaltura] 0_n6g3eyfo: Checking mp4-4432 URL
[Kaltura] 0_n6g3eyfo: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Active Object-0_n6g3eyfo.mp4 has already been downloaded

[download] 100% of 548.35MiB
[download] Finished downloading playlist: Design Patterns in the Real World, an Analysis-Based Approach

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', '-u', 'PRIVATE', '-p', 'PRIVATE', 'https://learning.oreilly.com/api/v1/book/9781491935828/chapter/video226616.html']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: 899af2ef6
[debug] Python version 3.10.10 (CPython) - macOS-13.2.1-arm64-64bit
[debug] exe versions: none
[debug] Proxy map: {}
[safari:api] Downloading login page
[safari:api] Logging in
[safari:api] Completing login
[safari:api] 9781491935828/video226616: Downloading part JSON
[safari] Downloading login page
[safari] 9781491935828-video226616: Downloading webpage
[safari] 9781491935828-video226616: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226616: Downloading webpage
[Kaltura] 0_n6g3eyfo: Downloading video info JSON
[Kaltura] 0_n6g3eyfo: Checking mp4-4432 URL
[Kaltura] 0_n6g3eyfo: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Active Object-0_n6g3eyfo.mp4 has already been downloaded

[download] 100% of 548.35MiB

@epsilonSpider
Copy link
Author

@dirkf

Please post a verbose log showing this in action, since it can't be tested in the CI, or by devs without a login.

I obscured the actual cdn URLs with [snip] but the rest of the log is untouched:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', '-u', 'PRIVATE', '-p', 'PRIVATE', 'https://learning.oreilly.com/videos/design-patterns-in/9781491935828/']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: d524ac189
[debug] Python version 3.10.10 (CPython) - macOS-13.2-arm64-64bit
[debug] exe versions: none
[debug] Proxy map: {}
[safari:course] Downloading login page
[safari:course] Logging in
[safari:course] Completing login
[safari:course] 9781491935828: Downloading course JSON
[download] Downloading playlist: Design Patterns in the Real World, an Analysis-Based Approach
[safari:course] playlist Design Patterns in the Real World, an Analysis-Based Approach: Collected 22 video ids (downloading 22 of them)
[download] Downloading video 1 of 22
[safari:api] Downloading login page
[safari:api] 9781491935828/video226613: Downloading part JSON
[safari] Downloading login page
[safari] 9781491935828-video226613: Downloading webpage
[safari] 9781491935828-video226613: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226613: Downloading webpage
[Kaltura] 0_g3w6syuh: Downloading video info JSON
[Kaltura] 0_g3w6syuh: Checking mp4-5170 URL
[Kaltura] 0_g3w6syuh: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Introduction-0_g3w6syuh.mp4 has already been downloaded
[download] 100% of 274.18MiB
[download] Downloading video 2 of 22
[safari:api] 9781491935828/video226617: Downloading part JSON
[safari] 9781491935828-video226617: Downloading webpage
[safari] 9781491935828-video226617: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226617: Downloading webpage
[Kaltura] 0_2r8msk12: Downloading video info JSON
[Kaltura] 0_2r8msk12: Checking mp4-4407 URL
[Kaltura] 0_2r8msk12: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] What Is A Design Pattern-0_2r8msk12.mp4 has already been downloaded
[download] 100% of 914.23MiB
[download] Downloading video 3 of 22
[safari:api] 9781491935828/video226618: Downloading part JSON
[safari] 9781491935828-video226618: Downloading webpage
[safari] 9781491935828-video226618: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226618: Downloading webpage
[Kaltura] 0_4iuhnj0j: Downloading video info JSON
[Kaltura] 0_4iuhnj0j: Checking mp4-5105 URL
[Kaltura] 0_4iuhnj0j: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Template Method -0_4iuhnj0j.mp4 has already been downloaded
[download] 100% of 463.59MiB
[download] Downloading video 4 of 22
[safari:api] 9781491935828/video226619: Downloading part JSON
[safari] 9781491935828-video226619: Downloading webpage
[safari] 9781491935828-video226619: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226619: Downloading webpage
[Kaltura] 0_frguhxo1: Downloading video info JSON
[Kaltura] 0_frguhxo1: Checking mp4-3556 URL
[Kaltura] 0_frguhxo1: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Builder - Separating the UI from the Domain Object-0_frguhxo1.mp4 has already been downloaded
[download] 100% of 590.87MiB
[download] Downloading video 5 of 22
[safari:api] 9781491935828/video226620: Downloading part JSON
[safari] 9781491935828-video226620: Downloading webpage
[safari] 9781491935828-video226620: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226620: Downloading webpage
[Kaltura] 0_0stusjis: Downloading video info JSON
[Kaltura] 0_0stusjis: Checking mp4-5019 URL
[Kaltura] 0_0stusjis: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Singleton and Utility-0_0stusjis.mp4 has already been downloaded
[download] 100% of 1.79GiB
[download] Downloading video 6 of 22
[safari:api] 9781491935828/video226621: Downloading part JSON
[safari] 9781491935828-video226621: Downloading webpage
[safari] 9781491935828-video226621: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226621: Downloading webpage
[Kaltura] 0_5raty95u: Downloading video info JSON
[Kaltura] 0_5raty95u: Checking mp4-4877 URL
[Kaltura] 0_5raty95u: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Abstract Factory-0_5raty95u.mp4 has already been downloaded
[download] 100% of 500.54MiB
[download] Downloading video 7 of 22
[safari:api] 9781491935828/video226647: Downloading part JSON
[safari] 9781491935828-video226647: Downloading webpage
[safari] 9781491935828-video226647: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226647: Downloading webpage
[Kaltura] 0_3at32aa3: Downloading video info JSON
[Kaltura] 0_3at32aa3: Checking mp4-4586 URL
[Kaltura] 0_3at32aa3: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Class and Abstract Adapters-0_3at32aa3.mp4 has already been downloaded
[download] 100% of 422.74MiB
[download] Downloading video 8 of 22
[safari:api] 9781491935828/video226623: Downloading part JSON
[safari] 9781491935828-video226623: Downloading webpage
[safari] 9781491935828-video226623: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226623: Downloading webpage
[Kaltura] 0_ct40r8x4: Downloading video info JSON
[Kaltura] 0_ct40r8x4: Checking mp4-4591 URL
[Kaltura] 0_ct40r8x4: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Decorator-0_ct40r8x4.mp4 has already been downloaded
[download] 100% of 505.65MiB
[download] Downloading video 9 of 22
[safari:api] 9781491935828/video226624: Downloading part JSON
[safari] 9781491935828-video226624: Downloading webpage
[safari] 9781491935828-video226624: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226624: Downloading webpage
[Kaltura] 0_5d9gg0t5: Downloading video info JSON
[Kaltura] 0_5d9gg0t5: Checking mp4-4875 URL
[Kaltura] 0_5d9gg0t5: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Bridge-0_5d9gg0t5.mp4 has already been downloaded
[download] 100% of 286.61MiB
[download] Downloading video 10 of 22
[safari:api] 9781491935828/video226603: Downloading part JSON
[safari] 9781491935828-video226603: Downloading webpage
[safari] 9781491935828-video226603: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226603: Downloading webpage
[Kaltura] 0_69jm0swk: Downloading video info JSON
[Kaltura] 0_69jm0swk: Checking mp4-5247 URL
[Kaltura] 0_69jm0swk: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Facade-0_69jm0swk.mp4 has already been downloaded
[download] 100% of 258.80MiB
[download] Downloading video 11 of 22
[safari:api] 9781491935828/video226604: Downloading part JSON
[safari] 9781491935828-video226604: Downloading webpage
[safari] 9781491935828-video226604: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226604: Downloading webpage
[Kaltura] 0_cygkp3d6: Downloading video info JSON
[Kaltura] 0_cygkp3d6: Checking mp4-5115 URL
[Kaltura] 0_cygkp3d6: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Prototype and Composite-0_cygkp3d6.mp4 has already been downloaded
[download] 100% of 674.01MiB
[download] Downloading video 12 of 22
[safari:api] 9781491935828/video227094: Downloading part JSON
[safari] 9781491935828-video227094: Downloading webpage
[safari] 9781491935828-video227094: Downloading kaltura session JSON
[Kaltura] 9781491935828-video227094: Downloading webpage
[Kaltura] 0_tz0yk85g: Downloading video info JSON
[Kaltura] 0_tz0yk85g: Checking mp4-4628 URL
[Kaltura] 0_tz0yk85g: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Flyweight and Flyweight Pool-0_tz0yk85g.mp4 has already been downloaded
[download] 100% of 611.50MiB
[download] Downloading video 13 of 22
[safari:api] 9781491935828/video226606: Downloading part JSON
[safari] 9781491935828-video226606: Downloading webpage
[safari] 9781491935828-video226606: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226606: Downloading webpage
[Kaltura] 0_kzkuhyzz: Downloading video info JSON
[Kaltura] 0_kzkuhyzz: Checking mp4-5069 URL
[Kaltura] 0_kzkuhyzz: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Proxy-0_kzkuhyzz.mp4 has already been downloaded
[download] 100% of 793.54MiB
[download] Downloading video 14 of 22
[safari:api] 9781491935828/video227136: Downloading part JSON
[safari] 9781491935828-video227136: Downloading webpage
[safari] 9781491935828-video227136: Downloading kaltura session JSON
[Kaltura] 9781491935828-video227136: Downloading webpage
[Kaltura] 0_haawrt9w: Downloading video info JSON
[Kaltura] 0_haawrt9w: Checking mp4-5121 URL
[Kaltura] 0_haawrt9w: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Command -0_haawrt9w.mp4 has already been downloaded
[download] 100% of 866.30MiB
[download] Downloading video 15 of 22
[safari:api] 9781491935828/video226608: Downloading part JSON
[safari] 9781491935828-video226608: Downloading webpage
[safari] 9781491935828-video226608: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226608: Downloading webpage
[Kaltura] 0_tgp5h7g9: Downloading video info JSON
[Kaltura] 0_tgp5h7g9: Checking mp4-4897 URL
[Kaltura] 0_tgp5h7g9: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Strategy-0_tgp5h7g9.mp4 has already been downloaded
[download] 100% of 488.89MiB
[download] Downloading video 16 of 22
[safari:api] 9781491935828/video227096: Downloading part JSON
[safari] 9781491935828-video227096: Downloading webpage
[safari] 9781491935828-video227096: Downloading kaltura session JSON
[Kaltura] 9781491935828-video227096: Downloading webpage
[Kaltura] 0_tu2f1fy6: Downloading video info JSON
[Kaltura] 0_tu2f1fy6: Checking mp4-4877 URL
[Kaltura] 0_tu2f1fy6: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] The Chain of Responsibility-0_tu2f1fy6.mp4 has already been downloaded
[download] 100% of 1.06GiB
[download] Downloading video 17 of 22
[safari:api] 9781491935828/video226610: Downloading part JSON
[safari] 9781491935828-video226610: Downloading webpage
[safari] 9781491935828-video226610: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226610: Downloading webpage
[Kaltura] 0_eoczuk3i: Downloading video info JSON
[Kaltura] 0_eoczuk3i: Checking mp4-3958 URL
[Kaltura] 0_eoczuk3i: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Iterators (Internal and External)-0_eoczuk3i.mp4 has already been downloaded
[download] 100% of 580.50MiB
[download] Downloading video 18 of 22
[safari:api] 9781491935828/video226611: Downloading part JSON
[safari] 9781491935828-video226611: Downloading webpage
[safari] 9781491935828-video226611: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226611: Downloading webpage
[Kaltura] 0_wmf5wiha: Downloading video info JSON
[Kaltura] 0_wmf5wiha: Checking mp4-5116 URL
[Kaltura] 0_wmf5wiha: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Observer-0_wmf5wiha.mp4 has already been downloaded
[download] 100% of 1.32GiB
[download] Downloading video 19 of 22
[safari:api] 9781491935828/video226612: Downloading part JSON
[safari] 9781491935828-video226612: Downloading webpage
[safari] 9781491935828-video226612: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226612: Downloading webpage
[Kaltura] 0_2gwh0mzi: Downloading video info JSON
[Kaltura] 0_2gwh0mzi: Checking mp4-5100 URL
[Kaltura] 0_2gwh0mzi: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Mediator-0_2gwh0mzi.mp4 has already been downloaded
[download] 100% of 580.33MiB
[download] Downloading video 20 of 22
[safari:api] 9781491935828/video226614: Downloading part JSON
[safari] 9781491935828-video226614: Downloading webpage
[safari] 9781491935828-video226614: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226614: Downloading webpage
[Kaltura] 0_vjs3vbtx: Downloading video info JSON
[Kaltura] 0_vjs3vbtx: Checking mp4-2900 URL
[Kaltura] 0_vjs3vbtx: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Memento-0_vjs3vbtx.mp4 has already been downloaded
[download] 100% of 354.84MiB
[download] Downloading video 21 of 22
[safari:api] 9781491935828/video226615: Downloading part JSON
[safari] 9781491935828-video226615: Downloading webpage
[safari] 9781491935828-video226615: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226615: Downloading webpage
[Kaltura] 0_nomyq1zy: Downloading video info JSON
[Kaltura] 0_nomyq1zy: Checking mp4-4176 URL
[Kaltura] 0_nomyq1zy: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] State-0_nomyq1zy.mp4 has already been downloaded
[download] 100% of 402.77MiB
[download] Downloading video 22 of 22
[safari:api] 9781491935828/video226616: Downloading part JSON
[safari] 9781491935828-video226616: Downloading webpage
[safari] 9781491935828-video226616: Downloading kaltura session JSON
[Kaltura] 9781491935828-video226616: Downloading webpage
[Kaltura] 0_n6g3eyfo: Downloading video info JSON
[Kaltura] 0_n6g3eyfo: Checking mp4-4432 URL
[Kaltura] 0_n6g3eyfo: Downloading m3u8 information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/[snip]'
[download] Active Object-0_n6g3eyfo.mp4 has already been downloaded
[download] 100% of 548.35MiB
[download] Finished downloading playlist: Design Patterns in the Real World, an Analysis-Based Approach

@epsilonSpider
Copy link
Author

@dirkf hey, just wanted to check in. Can i do anything to help get this merged?

I see there are conflicts now. I'm happy to work through resolving those if desired. But can you advise me on what else I can do to help this get merged? Thank you.

@dirkf dirkf mentioned this pull request Apr 10, 2023
Closed
3 tasks
@prdsharma
Copy link

Trying to test the fix #31524 locally, Videos is cutting off at 1:00 min when using --cookies and getting ExtractorError for username and password.
For more details:
#29823 (comment)

@Kerruba Kerruba mentioned this pull request Jul 17, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dirkf dirkf Awaiting requested review from dirkf

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

oreilly login page error
8 participants
@epsilonSpider @prdsharma @dirkf @pukkandan @Vangelis66 @Valentin-Metz @teddy171 @fonkap