Mediasite catalogue fails with HTTP Error 500

[qqii]

$ youtube-dl --cookies cookies.txt https://mediasite.bris.ac.uk/Mediasite/Catalog/Full/48736718de2148bea4cab381176fc64c21 -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--cookies', 'cookies.txt', 'https://mediasite.bris.ac.uk/Mediasite/Catalog/Full/48736718de2148bea4cab381176fc64c21', '-v']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2020.03.01
[debug] Lazy loading extractors enabled
[debug] Python version 3.7.6 (CPython) - Linux-5.4.17-x86_64-with-glibc2.2.5
[debug] exe versions: ffmpeg 4.2.2, ffprobe 4.2.2, rtmpdump 2.4
[debug] Proxy map: {}
[MediasiteCatalog] 48736718de2148bea4cab381176fc64c21: Downloading webpage
[MediasiteCatalog] 48736718de2148bea4cab381176fc64c21: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 500: Internal Server Error (caused by <HTTPError 500: 'Internal Server Error'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/nix/store/vkc0cybrica4mpyq8n6m3r8i5s5d4bxa-python3.7-youtube-dl-2020.03.01/lib/python3.7/site-packages/youtube_dl/extractor/common.py", line 627, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/nix/store/vkc0cybrica4mpyq8n6m3r8i5s5d4bxa-python3.7-youtube-dl-2020.03.01/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 2238, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/nix/store/fara2bdicvvi0p8bmrgcz6s8d6yicijc-python3-3.7.6/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/nix/store/fara2bdicvvi0p8bmrgcz6s8d6yicijc-python3-3.7.6/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/nix/store/fara2bdicvvi0p8bmrgcz6s8d6yicijc-python3-3.7.6/lib/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/nix/store/fara2bdicvvi0p8bmrgcz6s8d6yicijc-python3-3.7.6/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/nix/store/fara2bdicvvi0p8bmrgcz6s8d6yicijc-python3-3.7.6/lib/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

A quick gander of the code leads me to assume it was implemented for #20507 at https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/extractor/mediasite.py#L236

Currently as a workaround, I'm scraping the links using:

copy(Array.from(document.getElementsByClassName("titleLink")).reduce((acc, a) => acc + " " + a.href))

Which seems to work well enough on the site linked in #20507, but only copies form the current page.

[dstftw]

You must provide account credentials for any work on this.

[qqii]

What's the best way to do that? I'm also happy to try to give it a go myself.

[qqii]

Comparing the request with Chrome's network log the issue is the AuthTicket. Will investigate further as to where this comes from.

Would you be willing to reopen the issue?

[qqii]

I've found where the AuthTicket is defined, the (AntiForgeryToken)[https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/extractor/mediasite.py#L289] is defined in a similar way.

// Taken from http://events7.mediasite.com/Mediasite/Catalog/Full/631f9e48530d454381549f955d08c75e21
//<![CDATA[
     var MediasiteCatalogRequest =
     {
         CatalogId: '631f9e48-530d-4543-8154-9f955d08c75e',
         FolderId: '',
         LinkUrl: 'http://www.sonicfoundry.com/?siteid=85212d78-c652-495e-ab93-be846e2c0901?isa=false',
         RootDynamicFolderId: '',
         Preview: '',
         AuthTicket: '', // <----- HERE
         AntiForgeryToken: 'KWgl3BL-iJgmQMZQiXZdHF9CNiORCtUuVMzFHjJRHSZwEw_954dP037bXGNc0Vd58-T548MxpuQ5WTrK-vy7r0VXO581:NizATt_l8WeJsH-hxZL-3mc_LSMLK3AQzDJZ_qqPC2N5Du0DJIxyNVPMnd0jcQ3x_2WI6LSeurOgcg3deI1EE4tYfBs1',
         AntiForgeryHeaderName: 'X-SOFO-AntiForgeryHeader',
         SearchTerm: '',
         BaseHref: '/Mediasite/Catalog/',
         BuildNumber: '3945',
         Culture: 'en-GB',
         PrivacyPolicyUrl: ''
     };

[qqii]

I've made that change in a forked repo, but it looks like downloading the videos then fail at youtube_dl/extractor/mediasite.py#L155:

Traceback (most recent call last):
  File "/nix/store/fara2bdicvvi0p8bmrgcz6s8d6yicijc-python3-3.7.6/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/nix/store/fara2bdicvvi0p8bmrgcz6s8d6yicijc-python3-3.7.6/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/qqii/Documents/youtube-dl/youtube_dl/__main__.py", line 19, in <module>
    youtube_dl.main()
  File "/home/qqii/Documents/youtube-dl/youtube_dl/__init__.py", line 474, in main
    _real_main(argv)
  File "/home/qqii/Documents/youtube-dl/youtube_dl/__init__.py", line 464, in _real_main
    retcode = ydl.download(all_urls)
  File "/home/qqii/Documents/youtube-dl/youtube_dl/YoutubeDL.py", line 2019, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "/home/qqii/Documents/youtube-dl/youtube_dl/YoutubeDL.py", line 808, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/home/qqii/Documents/youtube-dl/youtube_dl/YoutubeDL.py", line 1008, in process_ie_result
    extra_info=extra)
  File "/home/qqii/Documents/youtube-dl/youtube_dl/YoutubeDL.py", line 870, in process_ie_result
    extra_info=extra_info)
  File "/home/qqii/Documents/youtube-dl/youtube_dl/YoutubeDL.py", line 797, in extract_info
    ie_result = ie.extract(url)
  File "/home/qqii/Documents/youtube-dl/youtube_dl/extractor/common.py", line 530, in extract
    ie_result = self._real_extract(url)
  File "/home/qqii/Documents/youtube-dl/youtube_dl/extractor/mediasite.py", line 155, in _real_extract
    title = presentation['Title']
TypeError: 'NoneType' object is not subscriptable

(Pdb) player_options
{'__type': 'PlayerOptions:#SonicFoundry.Mediasite.Model.Serialization', 'CoverageEvents': None, 'GlobalOptions': None, 'LoginUrl': 'https://mediasite.bris.ac.uk/Mediasite/Login', 'PlaybackTicket': None, 'PlayerLayoutOptions': None, 'PlayerPresentationStatus': 6, 'PlayerPresentationStatusMessage': "You are not authorized to view the requested content. Username: ''. Resource: 'beffad5e-04a3-4643-bea0-b27dc522d669'. Type: 'Presentation'.", 'Presentation': None, 'PresentationBookmark': None, 'PresentationId': None}

Will have to investigate further.