[Panopto] Add Panopto extractors #13449

kmark · 2017-06-21T07:56:58Z

Please follow the guide below

You will be asked some questions, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your pull request (like that [x])
Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
Searched the bugtracker for similar pull requests

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

I am the original author of this code and I am willing to release it under Unlicense
I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

Bug fix
Improvement
New extractor
New feature

Description of your pull request and other information

No test cases are included as I am not aware of any publicly
available Panopto recordings that this extractor will work with.

Supports downloading individual recordings or entire folders
recursively. Folders are seperated with a ' -- ' in the playlist
title.

Cookies are likely required to use this extractor specifically their
.ASPXAUTH cookie which can be obtained from your browser after
logging in.

--write-all-thumbnails can be used to download PowerPoint slides if
they are not included as a video stream.

Suggested output format is 'out/%(playlist)s/%(title)s.%(ext)s'

No test cases are included as I am not aware of any publicly available Panopto recordings that this extractor will work with. Supports downloading individual recordings or entire folders recursively. Folders are seperated with a ' -- ' in the playlist title. Cookies are likely required to use this extractor specifically their .ASPXAUTH cookie which can be obtained from your browser after logging in. --write-all-thumbnails can be used to download PowerPoint slides if they are not included as a video stream. Suggested output format is 'out/%(playlist)s/%(title)s.%(ext)s'

dstftw · 2017-06-21T17:51:49Z

Read

youtube-dl coding conventions

and fix all issues.
2. You have to provide account credentials and concrete example URL for testing.

Testing may be impossible for the Folder extractor, or I'm just doing it wrong. With the current test we enter a catch-22 where it claims we need an 'ext' entry to continue testing but upon adding that it claims it expected None.

kmark · 2017-07-07T05:02:07Z

I've fixed all the issues noted by QuantifiedCode. I've also added tests for the Panopto extractor that both work on my machine but fail the Travis test. I have no idea why this is. Please let me know if there are any further places where conventions aren't being followed.

As noted in one of my commits:

Testing may be impossible for the Folder extractor, or I'm just
doing it wrong. With the current test we enter a catch-22 where it
claims we need an 'ext' entry to continue testing but upon adding
that it claims it expected None.

I've still included the faulty PanoptoFolder test for review and reproduction of this issue.

Thanks!

dstftw · 2017-07-07T15:00:12Z

youtube_dl/extractor/panopto.py

+class PanoptoIE(PanoptoBaseIE):
+    """Extracts a single Panopto video including all available streams."""
+
+    _VALID_URL = r'^https?:\/\/(?P<org>[a-z0-9]+)\.hosted\.panopto.com\/Panopto\/Pages\/Viewer\.aspx\?id=(?P<id>[a-f0-9-]+)'


No need to escape \.

dstftw · 2017-07-07T15:00:27Z

youtube_dl/extractor/panopto.py

+        for c in contribs:
+            display_name = c.get('DisplayName')
+            if display_name is not None:
+                s += '{}, '.format(display_name)


{} does not work in python 2.6.

dstftw · 2017-07-07T15:03:05Z

youtube_dl/extractor/panopto.py

+            result['entries'] = streams
+
+        # We already know Delivery exists since we need it for stream extraction
+        contributors = delivery_info['Delivery'].get('Contributors')


Read coding conventions and fix all optional fields.

I'm not sure how this is at odds with the coding conventions. Are you referring to the use of ['Delivery']? The Delivery key must exist assuming that code is ever executed because the key is needed to retrieve non-optional fields earlier. It would fail at line 120:

for this_stream in delivery_info['Delivery']['Streams']:

Because we need ['Delivery'] (and more specifically ['Delivery']['Streams']) to even find the non-optional information.

@dstftw Is this still an issue?

https://stackoverflow.com/a/19668429/238374

kmark · 2017-07-08T20:21:26Z

The Travis errors appear to be from ffmpeg not being available:

DownloadError: ERROR: m3u8 download detected but ffmpeg or avconv could not be found. Please install one.

Once ffmpeg is installed it fixed the above error but the tests still failed with MD5 mismatches. It looks like the --test option isn't working properly on Linux:

$ python2.7 -m youtube_dl --test https://demo.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=26b3ae9e-4a48-4dcc-96ba-0befba08a0fb
$ wc -c Panopto\ for\ Business-26b3ae9e-4a48-4dcc-96ba-0befba08a0fb.mp4
13663 Panopto for Business-26b3ae9e-4a48-4dcc-96ba-0befba08a0fb.mp4
$ md5sum Panopto\ for\ Business-26b3ae9e-4a48-4dcc-96ba-0befba08a0fb.mp4
76d6bc8e500b1e47b53541514e3d1ea6  Panopto for Business-26b3ae9e-4a48-4dcc-96ba-0befba08a0fb.mp4

or macOS:

$ python2.7 -m youtube_dl --test https://demo.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=26b3ae9e-4a48-4dcc-96ba-0befba08a0fb
$ gwc -c Panopto\ for\ Business-26b3ae9e-4a48-4dcc-96ba-0befba08a0fb.mp4
13679 Panopto for Business-26b3ae9e-4a48-4dcc-96ba-0befba08a0fb.mp4
$ gmd5sum Panopto\ for\ Business-26b3ae9e-4a48-4dcc-96ba-0befba08a0fb.mp4
06fb292a3510aa5bc5f0c950fe58c9f7  Panopto for Business-26b3ae9e-4a48-4dcc-96ba-0befba08a0fb.mp4

Different lengths means both platforms will generate different MD5 sums during testing. If I truncate both to 10241 bytes I get the same MD5 but neither platform will test correctly against that MD5.

It's worth adding that if I select -f 0 to avoid M3U8/ffmpeg stuff then both platforms properly truncate to 10241 bytes.

kmark · 2017-07-08T22:10:40Z

Looks like the issue is with youtube-dl assuming that the -fs option of FFmpeg will limit the file size to 10241 bytes in a deterministic fashion. According to the documentation it doesn't guarantee a final file size and even claims the download will be larger by some undefined degree:

Set the file size limit, expressed in bytes. No further chunk of bytes is written after the limit is exceeded. The size of the output file is slightly more than the requested file size.

So we have two different versions of FFmpeg on two different platforms with FFmpeg's -fs option producing two different final file sizes. When I bring the FFmpeg versions closer together on both platforms (latest or near-latest from master) the file sizes are equal again. Unless I'm misunderstanding something about youtube-dl, I would put this in the bug category.

Edit: Even when I brought the FFmpeg versions closer together (roughly 19 days apart) the MD5s were still different. I brought both up to N-86744-gfe92422 and only then was I able to get identical MD5s.

This should prevent issues like that in #13449 where the FFmpeg downloader was not reliably trimming to the correct length.

manuel-uberti · 2020-08-21T13:49:39Z

Hello, any update on this? Thank you for working on it.

kmark · 2020-08-22T01:10:35Z

Hey Manuel, I haven't touched this or tried to use it since 2017. I imagine that it may need to be updated if Panopto has changed their (semi-private) API. This was waiting on another round of reviews, I believe I had addressed all of the outstanding comments.

As I mentioned earlier there seemed to be a bug in the way youtube-dl was doing its tests against FFmpeg, relying on the -fs option to behave deterministically across versions, which it does not. I opened up a PR to try and address this but did not receive any feedback on the second iteration of the patch. I am not sure if this is still an outstanding issue.

If there are users willing to test this patch and perhaps even work on it I would certainly appreciate it, and I'm sure the maintainers would too.

rgreenblatt · 2020-09-11T21:10:41Z

youtube_dl/extractor/panopto.py

+    _TESTS = [
+        {
+            'url': 'https://demo.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=26b3ae9e-4a48-4dcc-96ba-0befba08a0fb',
+            'md5': 'e8e6ef6b0572dd5985f5f8c3e096f717',


Suggested change

'md5': 'e8e6ef6b0572dd5985f5f8c3e096f717',

'md5': '048dab5c2f8ab97f2bd75ab4cf3f463a',

Maybe this video was changed, this now seems to be the correct hash.

rgreenblatt · 2020-09-11T21:12:22Z

youtube_dl/extractor/panopto.py

+                'title': this_stream['Tag'],
+                'formats': [],
+            }
+            if 'StreamHttpUrl' in this_stream:


Suggested change

if 'StreamHttpUrl' in this_stream:

if 'StreamHttpUrl' in this_stream and this_stream['StreamHttpUrl'] is not None:

It looks like this now can be null/none

rgreenblatt · 2020-09-11T21:14:05Z

youtube_dl/extractor/panopto.py

+                new_stream['formats'].append({
+                    'url': this_stream['StreamHttpUrl'],
+                })
+            if 'StreamUrl' in this_stream:


Suggested change

if 'StreamUrl' in this_stream:

if 'StreamUrl' in this_stream and this_stream['StreamUrl'] is not None:

Maybe the same change as for StreamHttpUrl should also be done for StreamUrl? I haven't seen any cases where this is None/null, but it seems like a good idea.

rgreenblatt · 2020-09-11T21:17:06Z

I tested this PR and I was able to get things working (just test/test_download.py and an example private video) after a few changes (see review).

rgreenblatt · 2020-09-14T18:18:31Z

youtube_dl/extractor/panopto.py

+                m3u8_formats = self._extract_m3u8_formats(this_stream['StreamUrl'], video_id, 'mp4')
+                self._sort_formats(m3u8_formats)
+                new_stream['formats'].extend(m3u8_formats)


Suggested change

m3u8_formats = self._extract_m3u8_formats(this_stream['StreamUrl'], video_id, 'mp4')

self._sort_formats(m3u8_formats)

new_stream['formats'].extend(m3u8_formats)

get_stream_url_matcher = lambda mid : re.compile(r'^https?://.*?/.*?/.*?' + mid + r'\?InvocationID.*')

if '_M3U8_MATCHER' not in self.__dict__:

self._M3U8_MATCHER = get_stream_url_matcher(r'\.hls/master\.m3u8')

if '_MP4_MATCHER' not in self.__dict__:

self._MP4_MATCHER = get_stream_url_matcher(r'\.mp4')

if self._M3U8_MATCHER.match(this_stream['StreamUrl']):

m3u8_formats = self._extract_m3u8_formats(this_stream['StreamUrl'], video_id, 'mp4')

self._sort_formats(m3u8_formats)

new_stream['formats'].extend(m3u8_formats)

elif self._MP4_MATCHER.match(this_stream['StreamUrl']):

new_stream['formats'].append({'url': this_stream['StreamUrl']})

else:

raise ExtractorError('Unexpected StreamUrl format')

In some cases, it looks like StreamUrl is a single mp4 file instead of m3u8 (when there is only a single video I assume?). I don't know how fragile this regex is, but it passes all of my tests.

jxu · 2020-09-30T00:08:34Z

https://github.com/jstrieb/panopto-download I just found about this that might help. I looked briefly through the PR and saw the links looked like they were hardcoded. They are available through the RSS feed of each folder currently.

rgreenblatt · 2020-10-17T21:04:51Z

https://github.com/jstrieb/panopto-download I just found about this that might help. I looked briefly through the PR and saw the links looked like they were hardcoded. They are available through the RSS feed of each folder currently.

Interestingly, the links in the RSS feed don't require any authentication, so it should be possible to convert from the folder URL to the RSS feed url and download all the videos without requiring cookies. Unfortunately, I don't think its possible to get the RSS feed url directly from a viewer video url, so cookies are needed when downloading a single video.

For my personal use cases, using the RSS feed directly is perfect. I set up a daemon which periodically runs youtube-dl on the RSS feed and sends notifications when a new video is downloaded. See this script from my dotfiles. This doesn't require this PR at all.

For the use case of downloading a single video, I think that using the RSS feed would increase complexity.

For the folder use case, I think that using the RSS should be considerably cleaner, but I don't know how that would work for recursive folder downloads. Also, I haven't tested recursive folder downloads, so it is possible that they are broken right now. I probably won't write this change for a few reasons, but I think this would be reasonably easy to do (aside from subfolders perhaps).

coletdjnz · 2022-02-28T08:21:33Z

Reviving this in yt-dlp: yt-dlp/yt-dlp#2908

Based on ytdl-org/youtube-dl#13449 Closes #1946 Authored by: coletdjnz, kmark

lebdron · 2022-07-19T12:14:29Z

@dirkf Hi! Any chance to get this merged? Thanks!

jxu · 2022-07-19T17:14:53Z

it's merged in yt-dlp btw

dirkf · 2022-07-19T19:12:29Z

We should back-port the yt-dlp extractor unless it does something weird wrt yt-dl.

coletdjnz · 2022-07-19T23:20:05Z

iirc there were some bugs in core that had to be fixed for panopto to work completely, so those will need to be backported too

jxu · 2022-07-31T21:08:36Z

at the risk of sounding like a broken record, if you compare youtube-dl commits and bufixes/new features to yt-dlp, youtube-dl as (mostly) controlled by @dirkf will always be behind in a project that has to keep up with constantly changing unofficial APIs

lebdron · 2022-07-31T21:23:57Z

@jxu no worries, I am working on a backport #31097 so that there will be at least some implementation

jxu · 2022-07-31T21:35:43Z

@lebdron I digress (rant): from what I can tell, unnecessary energy is spent backporting, but that is the reality of having a better maintained fork compete with the formerly undisputed project and fracturing the community. Especially when the project insists on supporting python2 and the 0.1% of people who choose run youtube-dl on an ancient embedded device that somehow doesn't have python3 but those people put the burden on contributors.

dirkf · 2022-07-31T23:14:12Z

@jxu can feel free not to spend any effort back-porting code to yt-dl, and should probably ignore yt-dl altogether.

Meanwhile thx @lebdron: please include "closes # 13449" in your PR text (but without the space after the #).

dstftw added the pending-fixes label Jun 21, 2017

dstftw force-pushed the master branch from fa77986 to 0c7a631 Compare June 24, 2017 22:04

kmark added 7 commits July 6, 2017 20:11

[Panopto] Document classes wrt PEP 257

cbb1753

[Panopto] Preferring static over instance methods

bd7d814

[Panopto] Replace % with format() wrt PEP 3101

0807e39

[Panopto] Document methods wrt PEP 257

a29e634

[Panopto] Bring the code in line with conventions

fed0a6a

[Panopto] Prefer the highest quality stream

dec449e

[Panopto] Add tests

b313bc6

Testing may be impossible for the Folder extractor, or I'm just doing it wrong. With the current test we enter a catch-22 where it claims we need an 'ext' entry to continue testing but upon adding that it claims it expected None.

dstftw requested changes Jul 7, 2017

View reviewed changes

kmark added 2 commits July 8, 2017 15:05

[Panopto] No need to escape /

7063d53

[Panopto] {} does not work in Python 2.6

d592aed

https://stackoverflow.com/a/19668429/238374

kmark referenced this pull request in kmark/youtube-dl Jul 8, 2017

Always truncate files to 10241 bytes with --test

c53c676

This should prevent issues like that in #13449 where the FFmpeg downloader was not reliably trimming to the correct length.

kmark mentioned this pull request Jul 8, 2017

Always truncate files to 10241 bytes with --test #13609

Closed

8 tasks

dstftw force-pushed the master branch from 4991699 to 1141e91 Compare August 5, 2017 00:42

kmark mentioned this pull request Dec 20, 2017

Allow entire file to match or first 10241 bytes #15036

Closed

9 tasks

kmark added 2 commits December 20, 2017 06:09

[Panopto] Fix all tests

56a8efd

[Panopto] More resilient folder regex

935b195

dstftw force-pushed the master branch from 37318e1 to 65220c3 Compare January 27, 2018 22:49

dstftw force-pushed the master branch from d99bab0 to e118a87 Compare January 23, 2019 18:40

jxu mentioned this pull request Nov 13, 2019

site support for panopto.com #22688

Open

5 tasks

rgreenblatt reviewed Sep 11, 2020

View reviewed changes

dstftw force-pushed the master branch from 5e26784 to da2069f Compare September 13, 2020 13:52

rgreenblatt reviewed Sep 14, 2020

View reviewed changes

pukkandan mentioned this pull request Dec 9, 2021

Site support request: Panopto yt-dlp/yt-dlp#1946

Closed

7 tasks

coletdjnz mentioned this pull request Feb 28, 2022

[panopto] Add Extractors yt-dlp/yt-dlp#2908

Merged

10 tasks

pukkandan pushed a commit to yt-dlp/yt-dlp that referenced this pull request Mar 8, 2022

[panopto] Add extractors (#2908)

e248be3

Based on ytdl-org/youtube-dl#13449 Closes #1946 Authored by: coletdjnz, kmark

lebdron mentioned this pull request Jul 19, 2022

[Panopto] Backport from yt-dlp #31097

Open

11 tasks

dirkf force-pushed the master branch from 01bf89e to 4c6fba3 Compare August 26, 2022 07:51

dirkf closed this Aug 1, 2023

dirkf added the defunct PR source branch is not accessible label Oct 2, 2023

	'md5': 'e8e6ef6b0572dd5985f5f8c3e096f717',
	'md5': '048dab5c2f8ab97f2bd75ab4cf3f463a',

	if 'StreamHttpUrl' in this_stream:
	if 'StreamHttpUrl' in this_stream and this_stream['StreamHttpUrl'] is not None:

	if 'StreamUrl' in this_stream:
	if 'StreamUrl' in this_stream and this_stream['StreamUrl'] is not None:

-                m3u8_formats = self._extract_m3u8_formats(this_stream['StreamUrl'], video_id, 'mp4')
-                self._sort_formats(m3u8_formats)
-                new_stream['formats'].extend(m3u8_formats)
+                get_stream_url_matcher = lambda mid : re.compile(r'^https?://.*?/.*?/.*?' + mid + r'\?InvocationID.*')
+                if '_M3U8_MATCHER' not in self.__dict__:
+                    self._M3U8_MATCHER = get_stream_url_matcher(r'\.hls/master\.m3u8')
+                if '_MP4_MATCHER' not in self.__dict__:
+                    self._MP4_MATCHER = get_stream_url_matcher(r'\.mp4')
+                if self._M3U8_MATCHER.match(this_stream['StreamUrl']):
+                    m3u8_formats = self._extract_m3u8_formats(this_stream['StreamUrl'], video_id, 'mp4')
+                    self._sort_formats(m3u8_formats)
+                    new_stream['formats'].extend(m3u8_formats)
+                elif self._MP4_MATCHER.match(this_stream['StreamUrl']):
+                    new_stream['formats'].append({'url': this_stream['StreamUrl']})
+                else:
+                    raise ExtractorError('Unexpected StreamUrl format')

[Panopto] Add Panopto extractors #13449

[Panopto] Add Panopto extractors #13449

Uh oh!

Conversation

kmark commented Jun 21, 2017

Please follow the guide below

Before submitting a pull request make sure you have:

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

What is the purpose of your pull request?

Description of your pull request and other information

Uh oh!

dstftw commented Jun 21, 2017

Uh oh!

kmark commented Jul 7, 2017

Uh oh!

dstftw Jul 7, 2017

Choose a reason for hiding this comment

Uh oh!

dstftw Jul 7, 2017

Choose a reason for hiding this comment

Uh oh!

dstftw Jul 7, 2017

Choose a reason for hiding this comment

Uh oh!

kmark Jul 8, 2017

Choose a reason for hiding this comment

Uh oh!

kmark Dec 20, 2017

Choose a reason for hiding this comment

Uh oh!

kmark commented Jul 8, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kmark commented Jul 8, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

manuel-uberti commented Aug 21, 2020

Uh oh!

kmark commented Aug 22, 2020

Uh oh!

rgreenblatt Sep 11, 2020

Choose a reason for hiding this comment

Uh oh!

rgreenblatt Sep 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rgreenblatt Sep 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rgreenblatt commented Sep 11, 2020

Uh oh!

rgreenblatt Sep 14, 2020

Choose a reason for hiding this comment

Uh oh!

jxu commented Sep 30, 2020

Uh oh!

rgreenblatt commented Oct 17, 2020

Uh oh!

coletdjnz commented Feb 28, 2022

Uh oh!

lebdron commented Jul 19, 2022

Uh oh!

jxu commented Jul 19, 2022

Uh oh!

dirkf commented Jul 19, 2022

Uh oh!

coletdjnz commented Jul 19, 2022

Uh oh!

jxu commented Jul 31, 2022

Uh oh!

lebdron commented Jul 31, 2022

Uh oh!

jxu commented Jul 31, 2022

Uh oh!

dirkf commented Jul 31, 2022

Uh oh!

Reviewers

Assignees

kmark commented Jul 8, 2017 •

edited

Loading

kmark commented Jul 8, 2017 •

edited

Loading

rgreenblatt Sep 11, 2020 •

edited

Loading

rgreenblatt Sep 11, 2020 •

edited

Loading