[bbc.co.uk] geolocation error for some episodes that play in the browser #23933

OmarWKH · 2020-02-01T20:05:00Z

Checklist

I'm reporting a broken site support
I've verified that I'm running youtube-dl version 2020.01.24
I've checked that all provided URLs are alive and playable in a browser
I've checked that all URLs and arguments with special characters are properly quoted or escaped
I've searched the bugtracker for similar issues including closed ones

Verbose log

> youtube-dl --verbose -s -i https://www.bbc.co.uk/sounds/play/m0009r5v https://www.bbc.co.uk/sounds/play/m0009rj7 https://www.bbc.co.uk/sounds/play/m0009tn5
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', '-s', '-i', 'https://www.bbc.co.uk/sounds/play/m0009r5v', 'https://www.bbc.co.uk/sounds/play/m0009rj7', 'https://www.bbc.co.uk/sounds/play/m0009tn5']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2020.01.24
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.18362
[debug] exe versions: ffmpeg 4.2.1, phantomjs 2.1.1, rtmpdump 2.3
[debug] Proxy map: {}
[bbc.co.uk] m0009r5v: Downloading video page
[bbc.co.uk] m0009r5v: Downloading playlist JSON
[bbc.co.uk] m0009r5t: Downloading media selection XML
[bbc.co.uk] m0009r5t: Downloading media selection XML
ERROR: bbc.co.uk returned error: geolocation
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpw0w300z1\build\youtube_dl\YoutubeDL.py", line 796, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpw0w300z1\build\youtube_dl\extractor\common.py", line 530, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpw0w300z1\build\youtube_dl\extractor\bbc.py", line 590, in _real_extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpw0w300z1\build\youtube_dl\extractor\bbc.py", line 486, in _download_playlist
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpw0w300z1\build\youtube_dl\extractor\bbc.py", line 362, in _download_media_selector
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpw0w300z1\build\youtube_dl\extractor\bbc.py", line 349, in _raise_extractor_error
youtube_dl.utils.ExtractorError: bbc.co.uk returned error: geolocation

[bbc.co.uk] m0009rj7: Downloading video page
[bbc.co.uk] m0009rj7: Downloading playlist JSON
[bbc.co.uk] m0009rj6: Downloading media selection XML
[bbc.co.uk] m0009rj6: Downloading media selection XML
ERROR: bbc.co.uk returned error: geolocation
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpw0w300z1\build\youtube_dl\YoutubeDL.py", line 796, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpw0w300z1\build\youtube_dl\extractor\common.py", line 530, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpw0w300z1\build\youtube_dl\extractor\bbc.py", line 590, in _real_extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpw0w300z1\build\youtube_dl\extractor\bbc.py", line 486, in _download_playlist
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpw0w300z1\build\youtube_dl\extractor\bbc.py", line 362, in _download_media_selector
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpw0w300z1\build\youtube_dl\extractor\bbc.py", line 349, in _raise_extractor_error
youtube_dl.utils.ExtractorError: bbc.co.uk returned error: geolocation

[bbc.co.uk] m0009tn5: Downloading video page
[bbc.co.uk] m0009tn5: Downloading playlist JSON
[bbc.co.uk] m0009tn4: Downloading media selection XML
[bbc.co.uk] m0009tn4: Downloading media selection XML
ERROR: bbc.co.uk returned error: geolocation
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpw0w300z1\build\youtube_dl\YoutubeDL.py", line 796, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpw0w300z1\build\youtube_dl\extractor\common.py", line 530, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpw0w300z1\build\youtube_dl\extractor\bbc.py", line 590, in _real_extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpw0w300z1\build\youtube_dl\extractor\bbc.py", line 486, in _download_playlist
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpw0w300z1\build\youtube_dl\extractor\bbc.py", line 362, in _download_media_selector
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpw0w300z1\build\youtube_dl\extractor\bbc.py", line 349, in _raise_extractor_error
youtube_dl.utils.ExtractorError: bbc.co.uk returned error: geolocation

Description

The episodes play in the browser but give a geolocation error in youtube-dl. I think this is because the page uses a URL with mediaselector/6 and youtube-dl uses mediaselector/5 (I'm not sure why they'd make one geo-restricted but not the other).

This problem happens with episodes of this particular series: https://www.bbc.co.uk/sounds/brand/m0009ql8

Direct links to the episodes:

These mediaselector links (for episode 1) also show a geolocation error in the same browser:

The page in the browser seems to be using a different mediaselector link.
It made two mediaselector requests:

https://open.live.bbc.co.uk/mediaselector/6/select/version/2.0/mediaset/pc/vpid/m0009r5t/format/json/atk/<some hex string>/asn/1/jsfunc/JS_callbacks0
https://open.live.bbc.co.uk/mediaselector/6/select/version/2.0/mediaset/pc/vpid/m0009rj6/format/json/atk/<some hex string>/asn/1/jsfunc/JS_callbacks0

And this is the response of https://open.live.bbc.co.uk/mediaselector/6/select/version/2.0/mediaset/pc/vpid/m0009r5t in the browser (I shortened some of the links):

{
  "media": [
    {
      "bitrate": "320",
      "service": "[deprecated]",
      "kind": "audio",
      "type": "audio/mp4",
      "encoding": "aac",
      "expires": "2024-10-26T15:00:00Z",
      "connection": [
        {
          "priority": "20",
          "protocol": "http",
          "authExpiresOffset": 21600,
          "supplier": "mf_akamai_nonbidi",
          "authExpires": "2020-02-02T01:12:33Z",
          "href": "http://aod-dash-ww-live.akamaized.net/usp/auth/vod/piff_abr_full_audio/9eea88-m0009r5t...",
          "transferFormat": "dash",
          "dpw": "50"
        },
        {
          "priority": "20",
          "protocol": "https",
          "authExpiresOffset": 21600,
          "supplier": "mf_akamai_nonbidi",
          "authExpires": "2020-02-02T01:12:33Z",
          "href": "https://aod-dash-ww-live.akamaized.net/usp/auth/vod/piff_abr_full_audio/9eea88-m0009r5t...",
          "transferFormat": "dash",
          "dpw": "50"
        },
        {
          "priority": "30",
          "protocol": "http",
          "authExpiresOffset": 21600,
          "supplier": "mf_limelight_nonbidi",
          "authExpires": "2020-02-02T01:12:33Z",
          "href": "http://aod-dash-ww-live.bbcfmt.hs.llnwd.net/usp/auth/vod/piff_abr_full_audio/9eea88-m0009r5t...",
          "transferFormat": "dash",
          "dpw": "50"
        },
        {
          "priority": "30",
          "protocol": "https",
          "authExpiresOffset": 21600,
          "supplier": "mf_limelight_nonbidi",
          "authExpires": "2020-02-02T01:12:33Z",
          "href": "https://aod-dash-ww-live.bbcfmt.hs.llnwd.net/usp/auth/vod/piff_abr_full_audio/9eea88-m0009r5t...",
          "transferFormat": "dash",
          "dpw": "50"
        },
        {
          "priority": "20",
          "protocol": "http",
          "authExpiresOffset": 21600,
          "supplier": "mf_akamai_nonbidi",
          "authExpires": "2020-02-02T01:12:33Z",
          "href": "http://aod-hds-ww-live.akamaized.net/usp/auth/vod/piff_abr_full_audio/9eea88-m0009r5t...",
          "transferFormat": "hds",
          "dpw": "50"
        },
        {
          "priority": "20",
          "protocol": "https",
          "authExpiresOffset": 21600,
          "supplier": "mf_akamai_nonbidi",
          "authExpires": "2020-02-02T01:12:33Z",
          "href": "https://aod-hds-ww-live.akamaized.net/usp/auth/vod/piff_abr_full_audio/9eea88-m0009r5t...",
          "transferFormat": "hds",
          "dpw": "50"
        },
        {
          "priority": "20",
          "protocol": "http",
          "authExpiresOffset": 21600,
          "supplier": "mf_akamai_nonbidi",
          "authExpires": "2020-02-02T01:12:33Z",
          "href": "http://aod-hls-ww-live.akamaized.net/usp/auth/vod/piff_abr_full_audio/9eea88-m0009r5t...",
          "transferFormat": "hls",
          "dpw": "50"
        },
        {
          "priority": "20",
          "protocol": "https",
          "authExpiresOffset": 21600,
          "supplier": "mf_akamai_nonbidi",
          "authExpires": "2020-02-02T01:12:33Z",
          "href": "https://aod-hls-ww-live.akamaized.net/usp/auth/vod/piff_abr_full_audio/9eea88-m0009r5t...",
          "transferFormat": "hls",
          "dpw": "50"
        },
        {
          "priority": "30",
          "protocol": "http",
          "authExpiresOffset": 21600,
          "supplier": "mf_limelight_nonbidi",
          "authExpires": "2020-02-02T01:12:33Z",
          "href": "http://aod-hds-ww-live.bbcfmt.hs.llnwd.net/usp/auth/vod/piff_abr_full_audio/9eea88-m0009r5t...",
          "transferFormat": "hds",
          "dpw": "50"
        },
        {
          "priority": "30",
          "protocol": "https",
          "authExpiresOffset": 21600,
          "supplier": "mf_limelight_nonbidi",
          "authExpires": "2020-02-02T01:12:33Z",
          "href": "https://aod-hds-ww-live.bbcfmt.hs.llnwd.net/usp/auth/vod/piff_abr_full_audio/9eea88-m0009r5t...",
          "transferFormat": "hds",
          "dpw": "50"
        },
        {
          "priority": "30",
          "protocol": "http",
          "authExpiresOffset": 21600,
          "supplier": "mf_limelight_nonbidi",
          "authExpires": "2020-02-02T01:12:33Z",
          "href": "http://aod-hls-ww-live.bbcfmt.hs.llnwd.net/usp/auth/vod/piff_abr_full_audio/9eea88-m0009r5t...",
          "transferFormat": "hls",
          "dpw": "50"
        },
        {
          "priority": "30",
          "protocol": "https",
          "authExpiresOffset": 21600,
          "supplier": "mf_limelight_nonbidi",
          "authExpires": "2020-02-02T01:12:33Z",
          "href": "https://aod-hls-ww-live.bbcfmt.hs.llnwd.net/usp/auth/vod/piff_abr_full_audio/9eea88-m0009r5t...",
          "transferFormat": "hls",
          "dpw": "50"
        }
      ]
    }
  ],
  "disclaimer": "This code and data form part of the BBC iPlayer content protection system. Tampering with, removal of, misuse of, or unauthorised use of this code or data constitutes circumvention of the BBC's content protection measures and may result in legal action. BBC (C) 2020."
}

Wowfunhappy · 2020-02-19T01:45:01Z

I ran into this independently a few months ago.

As a possible temporary workaround: one thing I noticed is that my browser and youtube-dl were grabbing different streams. Take a look at what streams are being downloaded by your browser using your browser's inspector, and try downloading those with youtube-dl. This worked for me.

OmarWKH · 2020-02-19T07:00:51Z

my browser and youtube-dl were grabbing different streams

Seems so. And to give YoutubeDL the opportunity to select the same stream as the browser, I added these URLs to BBCCoUkIE._MEDIASELECTOR_URLS in extractor/bbc.py:

https://open.live.bbc.co.uk/mediaselector/6/select/version/2.0/mediaset/iptv-all/vpid/%s/format/xml
https://open.live.bbc.co.uk/mediaselector/6/select/version/2.0/mediaset/pc/vpid/%s/format/xml

Someone left a comment on _MEDIASELECTOR_URLS indicating they are aware of the inconsistency in geolocation errors.

It seems there are two variables in the URLs:

mediaselector/x: Can be either mediaselector/5 or mediaselector/6. YoutubeDL uses mediaselector/5.
mediaset/x: Can be either mediaset/iptv-all or mediaset/pc. YoutubeDL uses both and tries iptv-all before pc (see the comment for more info).

For https://www.bbc.co.uk/sounds/play/m0009r5v, all combinations of mediaselector and mediaset give a gelocation error except mediaselector/6 and mediaset/pc. The media returned by the working URL has the service field set to [deprecated].

Someone with more experience in BBC's API could shed more light, but it might be too rare of a problem to worry about.

OmarWKH · 2020-02-19T07:29:09Z

More inconsistency. mediaselector/6 URLs could return JSON instead of XML even if you add /format/xml to the end. Example:

These correspond to a link which is geo-restricted in the browser: https://www.bbc.co.uk/sounds/play/m000c4s4.

mistotebe · 2020-05-18T08:54:32Z

I've just tried to download https://www.bbc.co.uk/news/av/health-52494495/coronavirus-r0-what-is-the-r-number-and-why-does-it-matter and had mixed success when I added the two mediaselector/6 URLs above.

I say mixed because using the above URL directly results in a failure, but resolves the programme id p08c0klq correctly, retrying with https://www.bbc.co.uk/programmes/p08c0klq then succeeds (needs the new URLs added to mediaselector URL list).

dirkf · 2020-10-06T12:06:13Z

The code in _download_media_selector that retrieves the playlist XML only returns the first successfully parsed playlist. If that contains no video or audio (say, only captions), the user will be disappointed.

Instead?

    def _download_media_selector(self, programme_id):
        last_exception = None
        formats = []
        subtitles = []
        for mediaselector_url in self._MEDIASELECTOR_URLS:
            try:
                formatsAndSubtitles = self._download_media_selector_url(
                    mediaselector_url % programme_id, programme_id)
                formats += formatsAndSubtitles[0]
                subtitles += formatsAndSubtitles[1]
            except BBCCoUkIE.MediaSelectionError as e:
                if e.id in ('notukerror', 'geolocation', 'selectionunavailable'):
                    last_exception = e
                    continue
                self._raise_extractor_error(e)
        if last_exception:
                self._raise_extractor_error(last_exception)
        return formats, subtitles

dirkf · 2020-10-06T14:44:00Z

As observed above the .../6/... playlists are JSON by default while the .../5/... ones are XML. Their parsed contents may not always be the same 5<->6.

The .../6/.../format/xml playlists quoted do come as XML for me.

dirkf · 2020-10-07T02:01:03Z

#26821

dirkf mentioned this issue Oct 6, 2020

Try all possible mediaselectors even if one succeeds #26821

Open

6 of 9 tasks complete

ytdl-org / youtube-dl

[bbc.co.uk] geolocation error for some episodes that play in the browser #23933

[bbc.co.uk] geolocation error for some episodes that play in the browser #23933

OmarWKH commented Feb 1, 2020

Wowfunhappy commented Feb 19, 2020 •

edited

OmarWKH commented Feb 19, 2020

OmarWKH commented Feb 19, 2020

mistotebe commented May 18, 2020

dirkf commented Oct 6, 2020

dirkf commented Oct 6, 2020

dirkf commented Oct 7, 2020

ytdl-org / youtube-dl

Join GitHub today

GitHub is where the world builds software

[bbc.co.uk] geolocation error for some episodes that play in the browser #23933

[bbc.co.uk] geolocation error for some episodes that play in the browser #23933

Comments

OmarWKH commented Feb 1, 2020

Checklist

Verbose log

Description

Wowfunhappy commented Feb 19, 2020 • edited

OmarWKH commented Feb 19, 2020

OmarWKH commented Feb 19, 2020

mistotebe commented May 18, 2020

dirkf commented Oct 6, 2020

dirkf commented Oct 6, 2020

dirkf commented Oct 7, 2020

Essential cookies

Always active

Analytics cookies

Wowfunhappy commented Feb 19, 2020 •

edited