[spreaker] Add new extractor #13877

Tatsh · 2017-08-10T12:17:38Z

Please follow the guide below

You will be asked some questions, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your pull request (like that [x])
Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
Searched the bugtracker for similar pull requests

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

I am the original author of this code and I am willing to release it under Unlicense
I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

Bug fix
Improvement
New extractor
New feature

Description of your pull request and other information

Extractor for Spreaker.com podcast site. Fixes issue #13480

dstftw · 2017-08-10T15:18:01Z

youtube_dl/extractor/spreaker.py

+    _VALID_URL = r"""(?x)^
+        https?://
+        (?:www.|api.)?
+        spreaker.com/


Dots must be escaped.

dstftw · 2017-08-10T15:18:54Z

youtube_dl/extractor/spreaker.py

+        (?:
+            show/[a-z0-9_-]+|
+            user/[a-z0-9_-]+/[a-z0-9_-]|
+            episode/(?P<id>[0-9]+)


Each should be a separate extractor.

dstftw · 2017-08-10T15:19:03Z

youtube_dl/extractor/spreaker.py

+        },
+        {
+            'url': ('https://www.spreaker.com/user/9780658/swm-ep15-how-to-'
+                    'market-your-music-part-2'),


Don't carry URLs.

dstftw · 2017-08-10T15:19:32Z

youtube_dl/extractor/spreaker.py

+    ]
+
+    def _spreaker_episode_data_to_info(self, data):
+        upload_date = data['published_at'][0:10].replace('-', '')


Should not break if published_at is missing.

dstftw · 2017-08-10T15:20:05Z

youtube_dl/extractor/spreaker.py

+        if not author:
+            author = {}
+        stats = data.get('stats')
+        view_count = like_count = comment_count = 0


Lack of information is denoted by None.

dstftw · 2017-08-10T15:20:51Z

youtube_dl/extractor/spreaker.py

+        view_count = like_count = comment_count = 0
+        show = data.get('show')
+        if not show:
+            show = {}


show = data.get('show') or {}

dstftw · 2017-08-10T15:24:51Z

youtube_dl/extractor/spreaker.py

+            show = {}
+        else:
+            show_image = show.get('image')
+        if not show_image:


Reference before assignment for show_image if not show.

Escape . in regexes Make separate extractors for episode page, playlist (show), API Support API's direct links to MP3 files Make counts set to None in case they are not found Handle when published_at is not present Other fixes

Tatsh · 2017-09-01T17:39:08Z

@dstftw This is ready to go on my end.

youtube_dl/extractor/spreaker.py

peterk · 2018-07-01T20:12:03Z

@Tatsh I am also looking for this. Do you plan on finishing it? Was thinking about starting a new PR.

Tatsh · 2018-07-01T21:29:06Z

@peterk I pushed new code up with the requested changes from @dstftw. This should be good to go now.

I suggest maintaining a local branch of your own, merging master and merging the pull requests you like. This way you can use the newest features and stay up-to-date with master. Just don't use this branch for pull requests (always start with master).

rudolphos · 2018-09-17T21:39:58Z

Is there an update on this?

Tatsh · 2019-01-26T00:53:59Z

@dstftw Please review again since the last commit.

Tatsh · 2019-03-20T01:20:19Z

Bump

rudolphos · 2019-09-07T00:09:26Z

bump, still doesn't work

#13480 (comment)

Tatsh · 2019-09-08T20:33:29Z

@rudolphos This version works fine.

rudolphos · 2019-09-14T23:40:17Z

I'm using the latest version and it doesn't work

Input URL: https://www.spreaker.com/user/hjmadigan/51-lost-in-a-good-game --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-ci', '--write-thumbnail', '--no-check-certificate', '--sub-format', 'ass/srt/best', '--write-auto-sub', '--convert-subs', 'srt', '--write-description', '-f', 'bestvideo+bestaudio/best', '--download-archive', 'downloaded.txt', '--output', '/Videos/%(uploader)s - %(title)s (%(id)s).%(ext)s', 'https://www.spreaker.com/user/hjmadigan/51-lost-in-a-good-game', '--verbose']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2019.09.12.1
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.18362
[debug] exe versions: ffmpeg N-93825-gc967128952, ffprobe N-93825-gc967128952, rtmpdump 2.4
[debug] Proxy map: {}
[generic] 51-lost-in-a-good-game: Requesting header
WARNING: Falling back on generic information extractor.
[generic] 51-lost-in-a-good-game: Downloading webpage
[generic] 51-lost-in-a-good-game: Extracting information
[generic] 8913843fbb3c94807f9351c73d45ca5e: Requesting header
WARNING: Falling back on generic information extractor.
[generic] 8913843fbb3c94807f9351c73d45ca5e: Downloading webpage
[generic] 8913843fbb3c94807f9351c73d45ca5e: Extracting information
ERROR: Unsupported URL: https://widget.spreaker.com/player?episode_id=19032297&autoplay=false&playlist=show&cover_image_url=https://d3wo5wojvuv7l.cloudfront.net/images.spreaker.com/original/8913843fbb3c94807f9351c73d45ca5e.jpg
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmphopugs4t\build\youtube_dl\YoutubeDL.py", line 796, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmphopugs4t\build\youtube_dl\extractor\common.py", line 530, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmphopugs4t\build\youtube_dl\extractor\generic.py", line 3355, in _real_extract
youtube_dl.utils.UnsupportedError: Unsupported URL: https://widget.spreaker.com/player?episode_id=19032297&autoplay=false&playlist=show&cover_image_url=https://d3wo5wojvuv7l.cloudfront.net/images.spreaker.com/original/8913843fbb3c94807f9351c73d45ca5e.jpg

Input URL: https://www.spreaker.com/show/psychology-of-video-games-podcast_1 --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-ci', '--write-thumbnail', '--no-check-certificate', '--sub-format', 'ass/srt/best', '--write-auto-sub', '--convert-subs', 'srt', '--write-description', '-f', 'bestvideo+bestaudio/best', '--download-archive', 'downloaded.txt', '--output', '/Videos/%(uploader)s - %(title)s (%(id)s).%(ext)s', 'https://www.spreaker.com/show/psychology-of-video-games-podcast_1', '--verbose']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2019.09.12.1
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.18362
[debug] exe versions: ffmpeg N-93825-gc967128952, ffprobe N-93825-gc967128952, rtmpdump 2.4
[debug] Proxy map: {}
[generic] psychology-of-video-games-podcast_1: Requesting header
WARNING: Falling back on generic information extractor.
[generic] psychology-of-video-games-podcast_1: Downloading webpage
[generic] psychology-of-video-games-podcast_1: Extracting information
[generic] 6773d4af57b93eac576d2c1ab827bf40: Requesting header
WARNING: Falling back on generic information extractor.
[generic] 6773d4af57b93eac576d2c1ab827bf40: Downloading webpage
[generic] 6773d4af57b93eac576d2c1ab827bf40: Extracting information
ERROR: Unsupported URL: https://widget.spreaker.com/player?show_id=3329893&autoplay=false&cover_image_url=https://d1bm3dmew779uf.cloudfront.net/cover/6773d4af57b93eac576d2c1ab827bf40.jpg
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmphopugs4t\build\youtube_dl\YoutubeDL.py", line 796, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmphopugs4t\build\youtube_dl\extractor\common.py", line 530, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmphopugs4t\build\youtube_dl\extractor\generic.py", line 3355, in _real_extract
youtube_dl.utils.UnsupportedError: Unsupported URL: https://widget.spreaker.com/player?show_id=3329893&autoplay=false&cover_image_url=https://d1bm3dmew779uf.cloudfront.net/cover/6773d4af57b93eac576d2c1ab827bf40.jpg

Tatsh · 2020-11-17T19:17:19Z

@dstftw If you want me to make changes, I'll need to re-make this PR. I asked GitHub and they cannot fix branch references.

dstftw · 2020-11-25T20:00:39Z

Picked some code in 686e898. Thanks.

[spreaker] Add new extractor

91e64c6

dstftw requested changes Aug 10, 2017

View reviewed changes

dstftw added the pending-fixes label Aug 10, 2017

Tatsh added 4 commits August 10, 2017 16:38

[spreaker] Fixes requested

2eb228d

Escape . in regexes Make separate extractors for episode page, playlist (show), API Support API's direct links to MP3 files Make counts set to None in case they are not found Handle when published_at is not present Other fixes

[spreaker] Make IE_NAME values unique

a894bc1

[spreaker] Set extractor name to spreaker to override IE_NAME

1c9e16b

[spreaker] Handle when playlist JSON has multiple pages

b653c19

dstftw mentioned this pull request Sep 16, 2017

[spreaker] Add new extractor #14222

Closed

8 tasks

dstftw force-pushed the master branch from 37318e1 to 65220c3 Compare January 27, 2018 22:49

dstftw requested changes Feb 25, 2018

View reviewed changes

Merge remote-tracking branch 'upstream/master' into spreaker

9017351

Tatsh added 2 commits July 1, 2018 17:06

Merge remote-tracking branch 'upstream/master' into spreaker

b3f6814

[spreaker] Fixes requested

5ec9047

Tatsh requested a review from dstftw November 28, 2019 04:18

dstftw closed this in 686e898 Nov 25, 2020

ThirumalaiK pushed a commit to ThirumalaiK/youtube-dl that referenced this pull request Jan 28, 2021

[spreaker] Add extractor (closes ytdl-org#13480, closes ytdl-org#13877)

0a3fcf0

cypheron mentioned this pull request Feb 3, 2021

Evaluation / overview of new proposed extractors / sites #28054

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[spreaker] Add new extractor #13877

[spreaker] Add new extractor #13877

Tatsh commented Aug 10, 2017

dstftw Aug 10, 2017

dstftw Aug 10, 2017

dstftw Aug 10, 2017

dstftw Aug 10, 2017

dstftw Aug 10, 2017

dstftw Aug 10, 2017

dstftw Aug 10, 2017

Tatsh commented Sep 1, 2017

peterk commented Jul 1, 2018

Tatsh commented Jul 1, 2018

rudolphos commented Sep 17, 2018

Tatsh commented Jan 26, 2019

Tatsh commented Mar 20, 2019

rudolphos commented Sep 7, 2019

Tatsh commented Sep 8, 2019

rudolphos commented Sep 14, 2019

Tatsh commented Nov 17, 2020

dstftw commented Nov 25, 2020

[spreaker] Add new extractor #13877

[spreaker] Add new extractor #13877

Conversation

Tatsh commented Aug 10, 2017

Please follow the guide below

Before submitting a pull request make sure you have:

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

What is the purpose of your pull request?

Description of your pull request and other information

dstftw Aug 10, 2017

Choose a reason for hiding this comment

dstftw Aug 10, 2017

Choose a reason for hiding this comment

dstftw Aug 10, 2017

Choose a reason for hiding this comment

dstftw Aug 10, 2017

Choose a reason for hiding this comment

dstftw Aug 10, 2017

Choose a reason for hiding this comment

dstftw Aug 10, 2017

Choose a reason for hiding this comment

dstftw Aug 10, 2017

Choose a reason for hiding this comment

Tatsh commented Sep 1, 2017

peterk commented Jul 1, 2018

Tatsh commented Jul 1, 2018

rudolphos commented Sep 17, 2018

Tatsh commented Jan 26, 2019

Tatsh commented Mar 20, 2019

rudolphos commented Sep 7, 2019

Tatsh commented Sep 8, 2019

rudolphos commented Sep 14, 2019

Tatsh commented Nov 17, 2020

dstftw commented Nov 25, 2020