[xhamster] No video format found again with patch #26254 and possible workaround

[TheRealDude2]

Checklist

• I'm reporting a broken site support
• I've verified that I'm running youtube-dl version 2020.07.28
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar issues including closed ones

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-c', '--batch-file=/media/dl/dl.txt', '-o', '/media/dl/%(title)s %(id)s.%(ext)s', '-v']
[debug] Batch file urls: ['https://de.xhamster.com/videos/jeny-smith-xhR2sPF'']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2020.07.28
[debug] Git HEAD: a4ed50bb8
[debug] Python version 3.7.3 (CPython) - Linux-5.4.51-v7+-armv7l-with-debian-10.4
[debug] exe versions: ffmpeg 4.1.6-1, ffprobe 4.1.6-1
[debug] Proxy map: {}
[XHamster] xhR2sPF: Downloading webpage
WARNING: [XHamster] xhR2sPF: Failed to parse JSON Unterminated string starting at: line 1 column 18367 (char 18366)
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/home/apadmin/git/youtube-dl_master/youtube_dl/YoutubeDL.py", line 797, in extract_info
    ie_result = ie.extract(url)
  File "/home/apadmin/git/youtube-dl_master/youtube_dl/extractor/common.py", line 530, in extract
    ie_result = self._real_extract(url)
  File "/home/apadmin/git/youtube-dl_master/youtube_dl/extractor/xhamster.py", line 242, in _real_extract
    self._sort_formats(formats)
  File "/home/apadmin/git/youtube-dl_master/youtube_dl/extractor/common.py", line 1350, in _sort_formats
    raise ExtractorError('No video formats found')
youtube_dl.utils.ExtractorError: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description

I noticed a problem again with the RegEx for the site initials. As it looks like the web server has problems to deliver the line completely. On the other hand, the character ; appears more often in the line now. Maybe just a temporary problem.

Workaround:
I changed the line 141 of the xhamster.py from:
r'window.initials\s*=\s*({.+?})\s*;', webpage, 'initials',
to
r'window.initials\s*=\s*({.+?})\s*;<', webpage, 'initials',

But this time it is really only a workaround. I think it would be better to adjust the regular expression to parse everything from <script id='initials-script'> to the next </script> and not to use the ; character at the end of the regex.

[hitam4450]

Hi, i checked my xhamster.py and could not find any reference of window.initials to fix the problem !

I am on UBUNTU 18.04 LTS !
regards, hitam4450

[TheRealDude2]

Hi hitam4450. It's line 141 in the latest xhamster.py. Some changes have been made here in the last weeks. Search in the line range 130-150.

Search for the block in the def _real_extract(self, url):
initials = self._parse_json(
self._search_regex(
r'window.initials\s*=\s*({.+?})\s*;', webpage, 'initials',
default='{}'),
video_id, fatal=false)

[hitam4450]

This i found on line 117

r'window.initials\s*=\s*({.+?})\s*;\s*\n', webpage, 'initials',

[hitam4450]

line 115 to 123
11 initials = self._parse_json(
116 self._search_regex(
117 r'window.initials\s*=\s*({.+?})\s*;\s*\n', webpage, 'initials',
118 default='{}'),
119 video_id, fatal=False)
120 if initials:
121 video = initials['videoModel']
122 title = video['title']
123 formats = []

[TheRealDude2]

Yes that is the line without patch #26254.

[hitam4450]

I applied the patch..but NO Success!

[TheRealDude2]

Do you have the no video format found OR the unsupported url error.
If it is the last and your url ends with letter you have to apply #25804 also.

[TheRealDude2]

Seems to have really been a temporary problem. I just messed up with the original xhamster.py and it's working again for now.

[hitam4450]

I applied the patch..but NO Success!

[TheRealDude2]

Maybe you have another problem. I think it will be best if you open a new issue with your own log.

[TheRealDude2]

Today the error occurs again. If there are more cases, maybe a patch should be written.
It is noticeable in the log that the line error is always at a different character position and sometimes no problem occurs. But all very arbitrary. For example:

WARNING: [XHamster] xhJqAVs: Failed to parse JSON Unterminated string starting at: line 1 column 21345 (char 21344)

Abort of the script with error.
Next run:
WARNING: [XHamster] xhJqAVs: Failed to parse JSON Unterminated string starting at: line 1 column 22785 (char 22784)

With the above mentioned workaround it works again.