Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Smotrim.ru does not download videos in MP4 format due to 403 Forbidden #9372

Open
10 of 11 tasks
zubrRB opened this issue Mar 5, 2024 · 12 comments
Open
10 of 11 tasks

Smotrim.ru does not download videos in MP4 format due to 403 Forbidden #9372

zubrRB opened this issue Mar 5, 2024 · 12 comments
Labels
patch-available There is patch available that should fix this issue. Someone needs to make a PR with it site-bug Issue with a specific website

Comments

@zubrRB
Copy link

zubrRB commented Mar 5, 2024

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

  • I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

Region

Russia

Provide a description that is worded well enough to be understood

I use to download Download Master 7.1.2.1729
A month ago due to changes on the site stopped downloading videos because it gives a link to a blocked server.
Simply changing the domain allows you to download the video. https://smotrim.ru/video/2754644

cdn-v.smotrim.ru — 403 Forbidden
https://cdn-v.smotrim.ru/_cdn_auth/secure/v/vh/mp4/fhd-wide/002/946/936.mp4?auth=mh&vid=2946936

cdn-v.rtr-vesti.ru — download available
https://cdn-v.rtr-vesti.ru/_cdn_auth/secure/v/vh/mp4/fhd-wide/002/946/936.mp4?auth=mh&vid=2946936

Provide verbose output that clearly demonstrates the problem

  • Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
  • If using API, add 'verbose': True to YoutubeDL params instead
  • Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

1 05.03.2024 17:48:56 Download state changed to Downloading
2 05.03.2024 17:48:56 Start get new link with ytdl
3 05.03.2024 17:49:03 Set new URL data from ytdl, and start download
4 05.03.2024 17:49:03 Start new section [1]
5 05.03.2024 17:49:03 Start connection to cdn-v.smotrim.ru:443 [1]
6 05.03.2024 17:49:03 Connecting to cdn-v.smotrim.ru:443 [1]
7 05.03.2024 17:49:03 Certificate is self-signed
8 05.03.2024 17:49:03 Version: 3
                      Issuer: GlobalSign
                      - organization unit: GlobalSign Root CA - R3
                      - common name: GlobalSign
                      Subject: GlobalSign
                      Validity period: 18.03.2009 10:00:00 - 18.03.2029 10:00:00
9 05.03.2024 17:49:03 Certificate is self-signed
10 05.03.2024 17:49:03 Version: 3
                       Issuer: GlobalSign
                       - organization unit: GlobalSign Root CA - R3
                       - common name: GlobalSign
                       Subject: GlobalSign GCC R3 DV TLS CA 2020
                       Validity period: 28.07.2020 - 18.03.2029
11 05.03.2024 17:49:03 Certificate is self-signed
12 05.03.2024 17:49:03 Version: 3
                       Issuer: GlobalSign nv-sa
                       - country: BE
                       - common name: GlobalSign GCC R3 DV TLS CA 2020
                       Subject: *.smotrim.ru
                       Validity period: 06.04.2023 14:07:22 - 07.05.2024 14:07:21
13 05.03.2024 17:49:04 Connected [1]
14 05.03.2024 17:49:04 GET /_cdn_auth/secure/v/vh/mp4/fhd-wide/002/961/688.mp4?auth=mh&vid=2961688 HTTP/1.1
                       Accept: */*
                       Accept-Encoding: identity
                       Referer: https://smotrim.ru/brand/62417
                       Cache-Control: no-cache
                       Host: cdn-v.smotrim.ru
15 05.03.2024 17:49:04 HTTP/1.1 403 Forbidden
                       Server: nginx
                       Date: Tue, 05 Mar 2024 14:49:02 GMT
                       Content-Type: text/html
                       Content-Length: 146
                       Connection: keep-alive
                       Set-Cookie: ngx_uid=wKgcM2XnMN5H6tTHf9uMAg==; expires=Thu, 31-Dec-37 23:55:55 GMT; path=/
...
50 05.03.2024 17:49:07 Download state changed to Erroring
51 05.03.2024 17:49:07 Closing socket [1]
52 05.03.2024 17:49:07 Socket disconnected [1]
53 05.03.2024 17:49:07 Start delete section [1]
54 05.03.2024 17:49:07 Section deleted [1] (t)
55 05.03.2024 17:49:07 Download state changed to Error
@zubrRB zubrRB added site-bug Issue with a specific website triage Untriaged issue labels Mar 5, 2024
@bashonly
Copy link
Member

bashonly commented Mar 5, 2024

I use to download Download Master 7.1.2.1729

I don't know what this means

Complete Verbose Output

What is this???? Use yt-dlp if you want yt-dlp support

@bashonly bashonly added invalid This doesn't seem right incomplete Further information is needed and removed triage Untriaged issue labels Mar 5, 2024
@zubrRB
Copy link
Author

zubrRB commented Mar 5, 2024

This is a well-known download manager https://downloadMaster.com/download/, using yt-dlp, which can be seen from the log.

@bashonly
Copy link
Member

bashonly commented Mar 5, 2024

You need to share an actual yt-dlp verbose log or else I am closing this issue

@zubrRB
Copy link
Author

zubrRB commented Mar 5, 2024

He uses yt-dlp to get a link to download the video MP4.

@bashonly bashonly closed this as not planned Won't fix, can't repro, duplicate, stale Mar 5, 2024
@qzmtch
Copy link

qzmtch commented Mar 5, 2024

You need to share an actual yt-dlp verbose log or else I am closing this issue

yt-dlp -f http-1080 -v https://smotrim.ru/video/2754644
[debug] Command-line config: ['-f', 'http-1080', '-v', 'https://smotrim.ru/video/2754644']
[debug] Encodings: locale cp1251, fs utf-8, pref cp1251, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version nightly@2024.02.17.232706 from yt-dlp/yt-dlp-nightly-builds [974d44403] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.19043-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg N-113649-ga2cfd6062c-20240215 (setts), ffprobe N-113649-ga2cfd6062c-20240215
[debug] Optional libraries: Cryptodome-3.20.0, brotli-1.1.0, certifi-2024.02.02, mutagen-1.47.0, requests-2.31.0, sqlite3-3.35.5, urllib3-2.2.0, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1833 extractors
[Smotrim] Extracting URL: https://smotrim.ru/video/2754644
[RUTV] Extracting URL: https://player.vgtrk.com/iframe/video/id/2754644
[RUTV] 2754644: Downloading JSON
[RUTV] 2754644: Downloading m3u8 information
[debug] Sort order given by extractor: source
[debug] Formats sorted by: hasvid, ie_pref, source, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, size, br, asr, proto, vext, aext, hasaud, id
[info] 2754644: Downloading 1 format(s): http-1080
[debug] Invoking http downloader on "https://cdn-v.smotrim.ru/_cdn_auth/secure/v/vh/mp4/fhd-wide/002/946/936_3.mp4?auth=mh&vid=2946936"
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "yt_dlp\YoutubeDL.py", line 3417, in process_info
  File "yt_dlp\YoutubeDL.py", line 3138, in dl
  File "yt_dlp\downloader\common.py", line 455, in download
  File "yt_dlp\downloader\http.py", line 364, in real_download
  File "yt_dlp\downloader\http.py", line 120, in establish_connection
  File "yt_dlp\YoutubeDL.py", line 4081, in urlopen
  File "yt_dlp\networking\common.py", line 114, in send
  File "yt_dlp\networking\_helper.py", line 204, in wrapper
  File "yt_dlp\networking\common.py", line 325, in send
  File "yt_dlp\networking\_requests.py", line 351, in _send
yt_dlp.networking.exceptions.HTTPError: HTTP Error 403: Forbidden

This site gives such links. This is a site problem.
He wants you to change the domain on HTTP links because some videos become available on HTTP links when you change the domain. But this does not work for all links and this makes no sense since all videos are available on HLS. This does not work only in the download master. They need to write this so that the developers of the download master add downloading using the m3u8 protocol. They use yt-dlp with their internal config

@bashonly
Copy link
Member

bashonly commented Mar 5, 2024

using the m3u8 protocol

--check-formats would work too

It looks like yt-dlp downloads HLS by default

@bashonly bashonly reopened this Mar 5, 2024
@bashonly bashonly removed invalid This doesn't seem right incomplete Further information is needed labels Mar 5, 2024
@qzmtch
Copy link

qzmtch commented Mar 5, 2024

using the m3u8 protocol

--check-formats would work too

It looks like yt-dlp downloads HLS by default

The fact is that the download wizard does not allow you to add your own keys. You need to write this to the developers of the download wizard, or rather the plugin, and ask them to add settings. Or to make it possible to add custom keys to launch the yt-dlp

@zubrRB
Copy link
Author

zubrRB commented Mar 5, 2024

A similar problem with the inability to download the video mp4 on the site https://belsat.eu/programs using TVPlayer (HTML5).
Although they are supported by yt-dlp.

@bashonly
Copy link
Member

bashonly commented Mar 5, 2024

Agreed with @qzmtch that replacing the hostname is not a viable solution, since many sites/extractors return url results to the RUTV extractor.

We could manually check the https formats like this:

diff --git a/yt_dlp/extractor/rutv.py b/yt_dlp/extractor/rutv.py
index d7f9a7337..50b9464cd 100644
--- a/yt_dlp/extractor/rutv.py
+++ b/yt_dlp/extractor/rutv.py
@@ -1,6 +1,7 @@
 import re
 
 from .common import InfoExtractor
+from ..networking import HEADRequest
 from ..utils import (
     ExtractorError,
     int_or_none,
@@ -156,6 +157,7 @@ def _real_extract(self, url):
 
         for transport, links in media['sources'].items():
             for quality, url in links.items():
+                format_id = f'{transport}-{quality}'
                 preference = -1 if priority_transport == transport else -2
                 if transport == 'rtmp':
                     mobj = re.search(r'^(?P<url>rtmp://[^/]+/(?P<app>.+))/(?P<playpath>.+)$', url)
@@ -178,13 +180,17 @@ def _real_extract(self, url):
                     self._merge_subtitles(subs, target=subtitles)
                     continue
                 else:
+                    try:
+                        self._request_webpage(HEADRequest(url), video_id, f'Checking {format_id} URL')
+                    except ExtractorError:
+                        continue
                     fmt = {
                         'url': url
                     }
                 fmt.update({
                     'width': int_or_none(quality, default=height, invscale=width, scale=height),
                     'height': int_or_none(quality, default=height),
-                    'format_id': '%s-%s' % (transport, quality),
+                    'format_id': format_id,
                     'source_preference': preference,
                 })
                 formats.append(fmt)

But that would be rather inconvenient if the http formats work fine for other sites/users.

Alternatively, we could make use of the private __needs_testing format field that we currently use with youtube m3u8 formats to force --check-formats (but then we should probably document it, too):

diff --git a/yt_dlp/extractor/rutv.py b/yt_dlp/extractor/rutv.py
index d7f9a7337..fed7229f9 100644
--- a/yt_dlp/extractor/rutv.py
+++ b/yt_dlp/extractor/rutv.py
@@ -179,7 +179,8 @@ def _real_extract(self, url):
                     continue
                 else:
                     fmt = {
-                        'url': url
+                        'url': url,
+                        '__needs_testing': True,
                     }
                 fmt.update({
                     'width': int_or_none(quality, default=height, invscale=width, scale=height),

@bashonly
Copy link
Member

bashonly commented Mar 5, 2024

While those patches would work as a yt-dlp solution, it looks like the 3rd party download manager is forcing https formats, so @zubrRB will likely need to raise an issue with the devs of that app in order to get this working for them.

@qzmtch
Copy link

qzmtch commented Mar 5, 2024

While those patches would work as a yt-dlp solution, it looks like the 3rd party download manager is forcing https formats, so @zubrRB will likely need to raise an issue with the devs of that app in order to get this working for them.

DM does not use --ignore-config switch
the config file with the key that you suggested --check-formats DM downloads normally

@zubrRB
Copy link
Author

zubrRB commented Mar 8, 2024

DM developers will try to fix.

@bashonly bashonly added the patch-available There is patch available that should fix this issue. Someone needs to make a PR with it label Apr 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
patch-available There is patch available that should fix this issue. Someone needs to make a PR with it site-bug Issue with a specific website
Projects
None yet
Development

No branches or pull requests

3 participants