[generic] None: Unable to download webpage: HTTP Error 403: Forbidden

[tinybug]

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

• I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

• I'm reporting a bug unrelated to a specific site
• I've verified that I'm running yt-dlp version 2023.03.04 (update instructions) or later (specify commit)
• I've checked that all provided URLs are playable in a browser with the same IP and same login details
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
• I've read the guidelines for opening an issue

Provide a description that is worded well enough to be understood

yt-dlp --extractor-args "generic:hls_key=https://hls2.videos.sproutvideo.com/a3f063147006cc49918f18f5f40e209e/ac7f47322156a22444b8957c99719a62/video/240.key?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9obHMyLnZpZGVvcy5zcHJvdXR2aWRlby5jb20vYTNmMDYzMTQ3MDA2Y2M0OTkxOGYxOGY1ZjQwZTIwOWUvYWM3ZjQ3MzIyMTU2YTIyNDQ0Yjg5NTdjOTk3MTlhNjIvKi5rZXk~c2Vzc2lvbklEPTVhNWE5MDdlLWUxYzItNGEyNC05ZWMzLTNlNzAxYTNjMzUwYyIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTY3OTQ5NTg2OX19fV19&Signature=X9DwHbafzvP-4RrKTuYztmQtIFVXVon6v4-xLilhvk8Za1fQ-JLjAQoSgMPBrcryfX4WqsYmf4vfgc-zBi1vYcZO8vxcz-GIaAgzN8VlhYZYwb1v1-ZAWEFI-k6voCWGGF04c~rQmxlJJboG5y5dM0ZL6Oves5UVxTa3fm-NSibyNqXdJKto0ADAKd6UV6H3tHI-x87IOnoqaUvhx2JND5SYJUWUVqGmIW1cBMUkDrSo4l~gJrdNlNLJrVMK5P9e0rnjggJzqwO4pAM3pPgxbZNt0~0dMuWlEi2WfiZL5lZwZcFjTFrkogYOaBA-fFtXTlCD5njQngeiqJUg5hHKKA__&Key-Pair-Id=APKAIB5DGCGAQJ4GGIUQ&sessionID=5a5a907e-e1c2-4a24-9ec3-3e701a3c350c" "https://hls2.videos.sproutvideo.com/a3f063147006cc49918f18f5f40e209e/ac7f47322156a22444b8957c99719a62/video/240.m3u8?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9obHMyLnZpZGVvcy5zcHJvdXR2aWRlby5jb20vYTNmMDYzMTQ3MDA2Y2M0OTkxOGYxOGY1ZjQwZTIwOWUvYWM3ZjQ3MzIyMTU2YTIyNDQ0Yjg5NTdjOTk3MTlhNjIvKi5tM3U4P3Nlc3Npb25JRD01ZjE4ZDY1ZS0yNzRjLTQ3YzYtOTFhMS0zMGQwZTk1ODgyZWIiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE2Nzk0OTc3MDN9fX1dfQ__&Signature=ZbWNlPzMD9oAk7XmG6MrmLnNN7OwpzKYMGLj~2L9LTXafsaq5DHALky5lcJ7-PtmkfBnKleIvuA-DKDWRNCeAtLZULoYV-WQAlNMiv1qKv5hpbpbz5qekJTygl7d-C6G-7Y7QF1SxueGnKHy9AO8QCzOVHLyo78od0LSa0RDUQmfxc1OXny-Waj28C~d8iOStY8-ygYZ8GAg1kLxjjuoolMltm8CisAi07xzJQUY-0O7~bpZHyPFrudn4Tg-QvNQunXFsMEQFbLz7Nq-9Bjw0jz89r58fHbHoqL96-UkAbCXijXdTHgPLH7ndThhK2LvhH9aZ5XclskQVi9wrHeC3Q__&Key-Pair-Id=APKAIB5DGCGAQJ4GGIUQ&sessionID=5f18d65e-274c-47c6-91a1-30d0e95882eb"

get this error: [generic] None: Unable to download webpage: HTTP Error 403: Forbidden

I write a simple demo for this url, it response 200

import urllib.request

url = 'https://hls2.videos.sproutvideo.com/a3f063147006cc49918f18f5f40e209e/ac7f47322156a22444b8957c99719a62/video/240.m3u8?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9obHMyLnZpZGVvcy5zcHJvdXR2aWRlby5jb20vYTNmMDYzMTQ3MDA2Y2M0OTkxOGYxOGY1ZjQwZTIwOWUvYWM3ZjQ3MzIyMTU2YTIyNDQ0Yjg5NTdjOTk3MTlhNjIvKi5tM3U4P3Nlc3Npb25JRD01ZjE4ZDY1ZS0yNzRjLTQ3YzYtOTFhMS0zMGQwZTk1ODgyZWIiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE2Nzk0OTc3MDN9fX1dfQ__&Signature=ZbWNlPzMD9oAk7XmG6MrmLnNN7OwpzKYMGLj~2L9LTXafsaq5DHALky5lcJ7-PtmkfBnKleIvuA-DKDWRNCeAtLZULoYV-WQAlNMiv1qKv5hpbpbz5qekJTygl7d-C6G-7Y7QF1SxueGnKHy9AO8QCzOVHLyo78od0LSa0RDUQmfxc1OXny-Waj28C~d8iOStY8-ygYZ8GAg1kLxjjuoolMltm8CisAi07xzJQUY-0O7~bpZHyPFrudn4Tg-QvNQunXFsMEQFbLz7Nq-9Bjw0jz89r58fHbHoqL96-UkAbCXijXdTHgPLH7ndThhK2LvhH9aZ5XclskQVi9wrHeC3Q__&Key-Pair-Id=APKAIB5DGCGAQJ4GGIUQ&sessionID=5f18d65e-274c-47c6-91a1-30d0e95882eb'

req = urllib.request.Request(url)
response = urllib.request.urlopen(req)

print(response.code)

for the simple demo, it will respose 200.

the m3u8 url can get from this page: https://www.wanderlustentrepreneur.com/rysr-replays/

Provide verbose output that clearly demonstrates the problem

• Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
• If using API, add 'verbose': True to YoutubeDL params instead
• Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

[generic] Extracting URL: https://hls2.videos.sproutvideo.com/a3f063147006cc49918f18f5f40e209e/ac7f47322156a22444b8957c9971...c6-91a1-30d0e95882eb 
kxOGYxOGY1ZjQwZTIwOWUvYWM3ZjQ3MzIyMTU2YTIyNDQ0Yjg5NTdjOTk3MTlhNjIvKi5rZXk~c2Vzc2lvbklEPTVhNWE5MDdlLWUxYzItNGEyNC05ZWMzLTNlNzAxYTNjMzUwYyIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTY3OTQ5NTg2OX19fV19&Signature=X9DwHbafzvP-4RrKTuYztmQtIFVXVon6v4-xLilhvk8Za1fQ-JLjAQoSgMPBrcryfX4WqsYmf4vfgc-zBi1vYcZO8vxcz-GIaAgzN8VlhYZYwb1v1-ZAWEFI-k6voCWGGF04c~rQmxlJJboG5y5dM0ZL6Oves5UVxTa3fm-NSibyNqXdJKto0ADAKd6UV6H3tHI-x87IOnoqaUvhx2JND5SYJUWUVqGmIW1cBMUkDrSo4l~gJrdNlNLJrVMK5P9e0rnjggJzqwO4pAM3pPgxbZNt0~0dMuWlEi2WfiZL5lZwZcFjTFrkogYOaBA-fFtXTlCD5njQngeiqJUg5hHKKA__&Key-Pair-Id=APKAIB5DGCGAQJ4GGIUQ&sessionID=5a5a907e-e1c2-4a24-9ec3-3e701a3c350c' 'https://hls2.videos.sproutvideo.com/a3f063147006cc49918f18f5f40e209e/ac7f47322156a22444b8957c99719a62/video/240.m3u8?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9obHMyLnZpZGVvcy5zcHJvdXR2aWRlby5jb20vYTNmMDYzMTQ3MDA2Y2M0OTkxOGYxOGY1ZjQwZTIwOWUvYWM3ZjQ3MzIyMTU2YTIyNDQ0Yjg5NTdjOTk3MTlhNjIvKi5tM3U4P3Nlc3Npb25JRD01ZjE4ZDY1ZS0yNzRjLTQ3YzYtOTFhMS0zMGQwZTk1ODgyZWIiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE2Nzk0OTc3MDN9fX1dfQ__&Signature=ZbWNlPzMD9oAk7XmG6MrmLnNN7OwpzKYMGLj~2L9LTXafsaq5DHALky5lcJ7-PtmkfBnKleIvuA-DKDWRNCeAtLZULoYV-WQAlNMiv1qKv5hpbpbz5qekJTygl7d-C6G-7Y7QF1SxueGnKHy9AO8QCzOVHLyo78od0LSa0RDUQmfxc1OXny-Waj28C~d8iOStY8-ygYZ8GAg1kLxjjuoolMltm8CisAi07xzJQUY-0O7~bpZHyPFrudn4Tg-QvNQunXFsMEQFbLz7Nq-9Bjw0jz89r58fHbHoqL96-UkAbCXijXdTHgPLH7ndThhK2LvhH9aZ5XclskQVi9wrHeC3Q__&Key-Pair-Id=APKAIB5DGCGAQJ4GGIUQ&sessionID=5f18d65e-274c-47c6-91a1-30d0e95882eb'[generic] Extracting URL: https://hls2.videos.sproutvideo.com/a3f063147006cc49918f18f5f40e209e/ac7f47322156a22444b8957c9971...c6-91a1-30d0e95882eb[generic] 240: Downloading webpageERROR: [generic] None: Unable to download webpage: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U

[bashonly]

Seems that the server is blocking based on user-agent. Adding this allows me to download the m3u8:

--add-header "User-Agent:Mozilla/5.0 (Windows NT 10.0; rv:111.0) Gecko/20100101 Firefox/111.0"

But then the fragments give 403 errors, even with the fragment_query extractor-arg.

The fragments 403 in browser too, with a message of Missing Key-Pair-Id query parameter or cookie value if I don't add the query string, and simply AccessDenied if I do add the query string.

(hls_key extractor-arg is irrelevant since yt-dlp never gets to the point of attempting to download the key)

[tinybug]

I use fragment_query for it, it still get 403 error for the fragments

"--extractor-args" "generic:fragment_query=https://hls2.videos.sproutvideo.com/a3f063147006cc49918f18f5f40e209e/ac7f47322156a22444b8957c99719a62/video/240_00000.ts?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9obHMyLnZpZGVvcy5zcHJvdXR2aWRlby5jb20vYTNmMDYzMTQ3MDA2Y2M0OTkxOGYxOGY1ZjQwZTIwOWUvYWM3ZjQ3MzIyMTU2YTIyNDQ0Yjg5NTdjOTk3MTlhNjIvKi50cz9zZXNzaW9uSUQ9MzcwYzc3NzUtMTg0MC00ZmFiLTg2OGMtOTE1MjM2OWI1ZGFlIiwiQ29uZGl0aW9uIjp7IkRhdGVMZXNzVGhhbiI6eyJBV1M6RXBvY2hUaW1lIjoxNjc5NTA1Mzk0fX19XX0_&Signature=rKaHTLmmhdYdkJwcfRpT-6QsoK-Ldeo5wrZhANLoGuNGTtAmPCXAe1V2w~11p8XgJqZxuA5uVtP0qNyCtvULjjBx1l2IdWaGgWT3ROSvWClrsPCG33Qnl3EBQpZ0pxeUZdWUL42SvqxHlBq3ar3vCzP4V04OVo6ZqZGfJaKBG6GNP1wE~AsbToOh~zzaaLL9W78fkgE3U6odR0lzHxZSzgQXsxWyUJJ1iVpRPCsS~niT-0Hrt8b3gPMqy9Bom-4jhWm7x5GdGDLCEmOapmGM5ABa8QYLeuv7XHv23ianxYKBjyuaRQcpTWt~dnHsvM8xnySCJLK-WXJt-faylj5vdQ__&Key-Pair-Id=APKAIB5DGCGAQJ4GGIUQ&sessionID=370c7775-1840-4fab-868c-9152369b5dae"

[bashonly]

fragment_query currently doesn't take an argument, it's a boolean-like switch that only passes through the m3u8 URL's query to the fragment URLs

Though maybe it should be improved so that it can take an optional argument

[dirkf]

Repro with yt-dl, Which punts the HLS to ffmpeg for the download and fails in a similar way. The manifest can't be played by mpv. A trace of the fragments being downloaded might help.

The containing page has a slightly interesting feature that could be a target if it didn't have a JS-enabled download link: the embed <iframe> has an element <script>var dat = '{base64_encoded_js_object}';</script>. dat is passed as an argument to the player.js. The leading encoded text eyJ in the dat value indicates {" followed by an alphabetic character.

Just Mozilla/5.0 as UA gets the manifest itself.

[bashonly]

@tinybug with nightly/master you should be able to download that m3u8 now, using --extractor-args "generic:hls_key=URI;fragment_query=QUERY" --user-agent "Mozilla/5.0"

[tinybug]

@bashonly Big Thanks to you!

[muhamed-didovic]

@bashonly can you give example with link how to run it, thanks

[bashonly]

@muhamed-didovic
the full command would follow this template:

yt-dlp --user-agent "Mozilla/5.0" --extractor-args "generic:hls_key=URI;fragment_query=QUERY" "URL"

so, using the URLs/values from OP, it would look like this:

yt-dlp --user-agent "Mozilla/5.0" --extractor-args "generic:hls_key=https://hls2.videos.sproutvideo.com/a3f063147006cc49918f18f5f40e209e/ac7f47322156a22444b8957c99719a62/video/240.key?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9obHMyLnZpZGVvcy5zcHJvdXR2aWRlby5jb20vYTNmMDYzMTQ3MDA2Y2M0OTkxOGYxOGY1ZjQwZTIwOWUvYWM3ZjQ3MzIyMTU2YTIyNDQ0Yjg5NTdjOTk3MTlhNjIvKi5rZXk~c2Vzc2lvbklEPTVhNWE5MDdlLWUxYzItNGEyNC05ZWMzLTNlNzAxYTNjMzUwYyIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTY3OTQ5NTg2OX19fV19&Signature=X9DwHbafzvP-4RrKTuYztmQtIFVXVon6v4-xLilhvk8Za1fQ-JLjAQoSgMPBrcryfX4WqsYmf4vfgc-zBi1vYcZO8vxcz-GIaAgzN8VlhYZYwb1v1-ZAWEFI-k6voCWGGF04c~rQmxlJJboG5y5dM0ZL6Oves5UVxTa3fm-NSibyNqXdJKto0ADAKd6UV6H3tHI-x87IOnoqaUvhx2JND5SYJUWUVqGmIW1cBMUkDrSo4l~gJrdNlNLJrVMK5P9e0rnjggJzqwO4pAM3pPgxbZNt0~0dMuWlEi2WfiZL5lZwZcFjTFrkogYOaBA-fFtXTlCD5njQngeiqJUg5hHKKA__&Key-Pair-Id=APKAIB5DGCGAQJ4GGIUQ&sessionID=5a5a907e-e1c2-4a24-9ec3-3e701a3c350c;fragment_query=Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9obHMyLnZpZGVvcy5zcHJvdXR2aWRlby5jb20vYTNmMDYzMTQ3MDA2Y2M0OTkxOGYxOGY1ZjQwZTIwOWUvYWM3ZjQ3MzIyMTU2YTIyNDQ0Yjg5NTdjOTk3MTlhNjIvKi50cz9zZXNzaW9uSUQ9MzcwYzc3NzUtMTg0MC00ZmFiLTg2OGMtOTE1MjM2OWI1ZGFlIiwiQ29uZGl0aW9uIjp7IkRhdGVMZXNzVGhhbiI6eyJBV1M6RXBvY2hUaW1lIjoxNjc5NTA1Mzk0fX19XX0_&Signature=rKaHTLmmhdYdkJwcfRpT-6QsoK-Ldeo5wrZhANLoGuNGTtAmPCXAe1V2w~11p8XgJqZxuA5uVtP0qNyCtvULjjBx1l2IdWaGgWT3ROSvWClrsPCG33Qnl3EBQpZ0pxeUZdWUL42SvqxHlBq3ar3vCzP4V04OVo6ZqZGfJaKBG6GNP1wE~AsbToOh~zzaaLL9W78fkgE3U6odR0lzHxZSzgQXsxWyUJJ1iVpRPCsS~niT-0Hrt8b3gPMqy9Bom-4jhWm7x5GdGDLCEmOapmGM5ABa8QYLeuv7XHv23ianxYKBjyuaRQcpTWt~dnHsvM8xnySCJLK-WXJt-faylj5vdQ__&Key-Pair-Id=APKAIB5DGCGAQJ4GGIUQ&sessionID=370c7775-1840-4fab-868c-9152369b5dae" "https://hls2.videos.sproutvideo.com/a3f063147006cc49918f18f5f40e209e/ac7f47322156a22444b8957c99719a62/video/240.m3u8?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9obHMyLnZpZGVvcy5zcHJvdXR2aWRlby5jb20vYTNmMDYzMTQ3MDA2Y2M0OTkxOGYxOGY1ZjQwZTIwOWUvYWM3ZjQ3MzIyMTU2YTIyNDQ0Yjg5NTdjOTk3MTlhNjIvKi5tM3U4P3Nlc3Npb25JRD01ZjE4ZDY1ZS0yNzRjLTQ3YzYtOTFhMS0zMGQwZTk1ODgyZWIiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE2Nzk0OTc3MDN9fX1dfQ__&Signature=ZbWNlPzMD9oAk7XmG6MrmLnNN7OwpzKYMGLj~2L9LTXafsaq5DHALky5lcJ7-PtmkfBnKleIvuA-DKDWRNCeAtLZULoYV-WQAlNMiv1qKv5hpbpbz5qekJTygl7d-C6G-7Y7QF1SxueGnKHy9AO8QCzOVHLyo78od0LSa0RDUQmfxc1OXny-Waj28C~d8iOStY8-ygYZ8GAg1kLxjjuoolMltm8CisAi07xzJQUY-0O7~bpZHyPFrudn4Tg-QvNQunXFsMEQFbLz7Nq-9Bjw0jz89r58fHbHoqL96-UkAbCXijXdTHgPLH7ndThhK2LvhH9aZ5XclskQVi9wrHeC3Q__&Key-Pair-Id=APKAIB5DGCGAQJ4GGIUQ&sessionID=5f18d65e-274c-47c6-91a1-30d0e95882eb"

though you could be getting a generic extractor 403 error for any number of reasons. Without seeing more details from your end I can't say if this info will actually be helpful to you. If you weren't specifically asking about the generic extractor-args, please open a new issue

[muhamed-didovic]

@bashonly thank you very much for quick response, I am having issue with 403 (cloudflare) and how to bypass it