ValueError: Padding is incorrect.

[rootxran]

Checklist

• This is a plugin issue and not a different kind of issue
• I have read the contribution guidelines
• I have checked the list of open and recently closed plugin issues
• I have checked the commit log of the master branch

Streamlink version

Latest stable release

Description

streamlink --http-header "User-Agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36" --http-header "Referer=https://www.log2base2.com/" "https://d2bzjah96pmxyc.cloudfront.net/C/Functions/Recursion/index.m3u8" best -l debug -o out1.ts

hls_audio_160k_v4.m3u8 has these #EXT-X-BYTERANGE lines.

#EXT-X-BYTERANGE:228432@0
#EXT-X-BYTERANGE:224288
#EXT-X-BYTERANGE:224864

...

And streamlink is setting these Range headers for HTTP GET requests.

Range: bytes=0-228431
Range: bytes=0-224287
Range: bytes=0-224863
...

Comment by:
@kikuyan

Debug log

streamlink --http-header "User-Agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36" --http-header "Referer=https://www.log2base2.com/" "https://d2bzjah96pmxyc.cloudfront.net/C/Functions/Recursion/index.m3u8" best -l debug -o out1.ts
[cli][debug] OS: Windows 10
[cli][debug] Python: 3.9.8
[cli][debug] Streamlink: 3.0.3
[cli][debug] Requests(2.26.0), Socks(1.7.1), Websocket(1.2.1)
[cli][debug] Arguments:
[cli][debug] url=https://d2bzjah96pmxyc.cloudfront.net/C/Functions/Recursion/index.m3u8
[cli][debug] stream=['best']
[cli][debug] --loglevel=debug
[cli][debug] --output=out1.ts
[cli][debug] --ffmpeg-ffmpeg=C:\Program Files (x86)\Streamlink\ffmpeg\ffmpeg.exe
[cli][debug] --http-header=[('User-Agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36'), ('Referer', 'https://www.log2base2.com/')]
[cli][info] Found matching plugin hls for URL https://d2bzjah96pmxyc.cloudfront.net/C/Functions/Recursion/index.m3u8
[plugins.hls][debug] URL=https://d2bzjah96pmxyc.cloudfront.net/C/Functions/Recursion/index.m3u8; params={}
[utils.l10n][debug] Language code: en_GB
[stream.hls][debug] Using external audio tracks for stream 1080p (language=None, name=Default)
[stream.hls][debug] Using external audio tracks for stream 1080p_alt (language=None, name=Default)
[stream.hls][debug] Using external audio tracks for stream 576p (language=None, name=Default)
[stream.hls][debug] Using external audio tracks for stream 360p (language=None, name=Default)
[stream.hls][debug] Using external audio tracks for stream 224p (language=None, name=Default)
[stream.hls][debug] Using external audio tracks for stream 73k (language=None, name=Default)
[cli][info] Available streams: 73k (worst), 224p, 360p, 576p, 1080p_alt, 1080p (best)
[cli][info] Opening stream: 1080p (hls-multi)
[stream.ffmpegmux][debug] Opening hls substream
[stream.hls][debug] Reloading playlist
[stream.ffmpegmux][debug] Opening hls substream
[stream.hls][debug] Reloading playlist
[utils.named_pipe][info] Creating pipe streamlinkpipe-6436-1-825
[utils.named_pipe][info] Creating pipe streamlinkpipe-6436-2-8256
[stream.ffmpegmux][debug] ffmpeg command: C:\Program Files (x86)\Streamlink\ffmpeg\ffmpeg.exe -nostats -y -i \.\pipe\streamlinkpipe-6436-1-825 -i \.\pipe\streamlinkpipe-6436-2-8256 -c:v copy -c:a copy -map 0:v? -map 0:a? -map 1:a -f mpegts pipe:1
[stream.ffmpegmux][debug] Starting copy to pipe: \.\pipe\streamlinkpipe-6436-1-825
[stream.ffmpegmux][debug] Starting copy to pipe: \.\pipe\streamlinkpipe-6436-2-8256
[cli][debug] Pre-buffering 8192 bytes
[stream.hls][debug] Segments in this playlist are encrypted
[stream.hls][debug] First Sequence: 0; Last Sequence: 24
[stream.hls][debug] Start offset: 0; Duration: None; Start Sequence: 0; End Sequence: 24
[stream.hls][debug] Adding segment 0 to queue
[stream.hls][debug] Adding segment 1 to queue
[stream.hls][debug] Adding segment 2 to queue
[stream.hls][debug] Adding segment 3 to queue
[stream.hls][debug] Adding segment 4 to queue
[stream.hls][debug] Adding segment 5 to queue
[stream.hls][debug] Adding segment 6 to queue
[stream.hls][debug] Adding segment 7 to queue
[stream.hls][debug] Adding segment 8 to queue
[stream.hls][debug] Adding segment 9 to queue
[stream.hls][debug] Adding segment 10 to queue
[stream.hls][debug] Adding segment 11 to queue
[stream.hls][debug] Adding segment 12 to queue
[stream.hls][debug] Adding segment 13 to queue
[stream.hls][debug] Adding segment 14 to queue
[stream.hls][debug] Adding segment 15 to queue
[stream.hls][debug] Adding segment 16 to queue
[stream.hls][debug] Adding segment 17 to queue
[stream.hls][debug] Adding segment 18 to queue
[stream.hls][debug] Adding segment 19 to queue
[stream.hls][debug] Adding segment 20 to queue
[stream.hls][debug] Adding segment 21 to queue
[stream.hls][debug] Segments in this playlist are encrypted
[stream.hls][debug] First Sequence: 0; Last Sequence: 25
[stream.hls][debug] Start offset: 0; Duration: None; Start Sequence: 0; End Sequence: 25
[stream.hls][debug] Adding segment 0 to queue
[stream.hls][debug] Adding segment 1 to queue
[stream.hls][debug] Adding segment 2 to queue
[stream.hls][debug] Adding segment 3 to queue
[stream.hls][debug] Adding segment 4 to queue
[stream.hls][debug] Adding segment 5 to queue
[stream.hls][debug] Adding segment 6 to queue
[stream.hls][debug] Adding segment 7 to queue
[stream.hls][debug] Adding segment 8 to queue
[stream.hls][debug] Adding segment 9 to queue
[stream.hls][debug] Adding segment 10 to queue
[stream.hls][debug] Adding segment 11 to queue
[stream.hls][debug] Adding segment 12 to queue
[stream.hls][debug] Adding segment 13 to queue
[stream.hls][debug] Adding segment 14 to queue
[stream.hls][debug] Adding segment 15 to queue
[stream.hls][debug] Adding segment 16 to queue
[stream.hls][debug] Adding segment 17 to queue
[stream.hls][debug] Adding segment 18 to queue
[stream.hls][debug] Adding segment 19 to queue
[stream.hls][debug] Adding segment 20 to queue
[stream.hls][debug] Adding segment 21 to queue
[stream.hls][debug] Segment 0 complete
[stream.hls][debug] Adding segment 22 to queue
[stream.hls][debug] Segment 0 complete
[stream.hls][debug] Adding segment 22 to queue
[cli][debug] Checking file output
[cli][debug] Writing stream to output
Exception in thread Thread-HLSStreamWriter:
Traceback (most recent call last):
File "threading.py", line 973, in _bootstrap_inner
File "C:\Program Files (x86)\Streamlink\pkgs\streamlink\stream\segmented.py", line 200, in run
self.write(segment, result, *data)
File "C:\Program Files (x86)\Streamlink\pkgs\streamlink\stream\hls.py", line 162, in write
return self._write(sequence, *args, **kwargs)
File "C:\Program Files (x86)\Streamlink\pkgs\streamlink\stream\hls.py", line 192, in _write
chunk = unpad(decrypted_chunk, AES.block_size, style="pkcs7")
File "C:\Program Files (x86)\Streamlink\pkgs\Crypto\Util\Padding.py", line 92, in unpad
raise ValueError("Padding is incorrect.")
ValueError: Padding is incorrect.
Exception in thread Thread-HLSStreamWriter:
Traceback (most recent call last):
File "threading.py", line 973, in _bootstrap_inner
File "C:\Program Files (x86)\Streamlink\pkgs\streamlink\stream\segmented.py", line 200, in run
self.write(segment, result, *data)
File "C:\Program Files (x86)\Streamlink\pkgs\streamlink\stream\hls.py", line 162, in write
return self._write(sequence, *args, **kwargs)
File "C:\Program Files (x86)\Streamlink\pkgs\streamlink\stream\hls.py", line 192, in _write
chunk = unpad(decrypted_chunk, AES.block_size, style="pkcs7")
File "C:\Program Files (x86)\Streamlink\pkgs\Crypto\Util\Padding.py", line 92, in unpad
raise ValueError("Padding is incorrect.")
ValueError: Padding is incorrect.

[bastimeyer]

There's a bug in the HLS playlist parser:
https://github.com/streamlink/streamlink/blob/3.0.3/src/streamlink/stream/hls_playlist.py#L217

The BYTERANGE tag's offset value is optional but the parser always sets the value to 0 if it doesn't exist, which is obviously wrong.

The create_request_params code where the Range header gets set stems from the Livestreamer era in 2013 and hasn't been changed since then apart from some minor stuff. Unfortunately nobody has written tests for HLS byteranges, so a side effect in the HLS playlist parser has broken the byterange requests.

However, there's another issue with how the Range header gets set. The HLSStreamWriter has an infinitely growing cache for offsets of previous segments (byterange_offsets). This cache depends on the segment's URL, which is incorrect. The HLS spec says that only the previous segment should be used for missing offset values, and this previous segment can have a different URL.

Another problem is that the HLSStreamWriter.fetch method (which calls create_request_params) is run by other threads from the writer's thread-pool, so a sequential order of requests is not guaranteed. The individual Segment objects don't have a reference to their previous segment, and the Sequence object which holds the Segment object and its sequence number currently doesn't get passed to the create_request_params method, so a single "previous segment byterange offset cache" value can't be used without changing that.

Let me quickly fix the HLS playlist parser first though before I take a look at the other issue.