[BUG] JSONDecodeError when attempting to download the chat of any twitch VOD

[chaosaudit]

Basic information

• Program version: 0.2.2
• Python version: 3.10
• Operating system: Windows 11

Describe the bug

JSONDecodeError when attempting to download the chat of any twitch VOD

Command/Code used

from chat_downloader import ChatDownloader
chat = ChatDownloader().get_chat(vod_url)
for message in chat:
    print(message)

2. Output, traceback or other information relating to the bug:

Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\site-packages\requests\models.py", line 910, in json
    return complexjson.loads(self.text, **kwargs)
  File "C:\Program Files\Python310\lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "C:\Program Files\Python310\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Program Files\Python310\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\site-packages\chat_downloader\sites\twitch.py", line 1175, in _get_chat_messages_by_vod_id
    info = self._session_get_json(url)
  File "C:\Program Files\Python310\lib\site-packages\chat_downloader\sites\common.py", line 512, in _session_get_json
    return self._session_get(url, **kwargs).json()
  File "C:\Program Files\Python310\lib\site-packages\requests\models.py", line 917, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Expecting value] : 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\chaos\PycharmProjects\AdventOfCode\scratch.py", line 5, in <module>
    for message in chat:
  File "C:\Program Files\Python310\lib\site-packages\chat_downloader\sites\common.py", line 283, in __next__
    item = next(self.chat)
  File "C:\Program Files\Python310\lib\site-packages\chat_downloader\sites\twitch.py", line 1178, in _get_chat_messages_by_vod_id
    self.retry(attempt_number, error=e, **params)
  File "C:\Program Files\Python310\lib\site-packages\chat_downloader\sites\common.py", line 640, in retry
    page_title = get_title_of_webpage(error.doc)
AttributeError: 'JSONDecodeError' object has no attribute 'doc'

Process finished with exit code 1

[xenova]

It finally happened :') Looks like they discontinued the /v5/videos/.../comments endpoint. (Must have happened in the past 24 hours since the last unit tests succeeded).

I imagine every other tool that downloads Twitch chat is currently broken.

Thanks for the report! I will fix ASAP

[gadi-playstream-gg]

Thanks for the update! Please keep us posted :)

[xenova]

That should fix it 👍 Just running unit tests then will publish package.

[xenova]

Published 👍 You can install using
pip install chat-downloader --upgrade

[wordlebot-uxy]

@xenova I see that you solved it by moving to gql endpoint. Is it pssible that twitch will close that endpoint as well, as they just closed the v5? And if so, would your package have another way of extracting the chat data? Wondering bc of future proof. Thanks for your work!

[xenova]

@wordlebot-uxy While they may eventually decide to switch to another protocol, they have been slowly transitioning to GQL (GraphQL) over the past few years... so, I do not imagine that they will switch any time soon.

That being said, if any breaking changes do come, I will be ready to fix them :)

[wordlebot-uxy]

Got it, thanks! One more q: do they require you to use a token to communicate with that endpoint? If so, couldn't they just kill your token?

[xenova]

Got it, thanks! One more q: do they require you to use a token to communicate with that endpoint? If so, couldn't they just kill your token?

The client id (token) that is used throughout the program (kimne78kx3ncx6brgo4mv6wki5h1ko) is public and is used for all unauthenticated users (you can test it out by opening an incognito tab and viewing the request that is sent). This token has been in use for many years, so, they shouldn't just kill it. If they do... it can just be replaced.

[wordlebot-uxy]

Thanks a lot! You got a patreon or Ko-Fi or similar?

[xenova]

Thanks a lot! You got a patreon or Ko-Fi or similar?

No worries! 👍 I have a Ko-Fi: https://ko-fi.com/xenova 😊