[Bug] Comments won't load on all videos

[DestinyofYeet]

Describe the bug

Comments won't load on some videos

Steps to Reproduce

1. Open video with video ID q0zCZpyv8HI
2. Scroll down to the comments
3. Be greeted by an infinite loading circle instead of the comment section

Logs

The GET request to api/v1/comments/VIDEOID?format=html&hl=en-US&thin_mode=false fails with a 500 error code.

{
	"error": "Missing hash key: \"commentRenderer\"",
	"errorBacktrace": "Missing hash key: \"commentRenderer\" (KeyError)\n  from /usr/share/crystal/src/hash.cr:1191:11 in '[]'\n  from /usr/share/crystal/src/json/any.cr:115:7 in '[]'\n  from src/invidious/comments/youtube.cr:137:34 in 'parse_youtube'\n  from src/invidious/comments/youtube.cr:60:3 in 'parse_youtube'\n  from src/invidious/comments/youtube.cr:16:12 in 'fetch_youtube'\n  from src/invidious/routes/api/v1/videos.cr:325:20 in 'comments'\n  from src/invidious/routing.cr:229:5 in '->'\n  from lib/kemal/src/kemal/route.cr:12:26 in '->'\n  from src/invidious/helpers/handlers.cr:30:5 in 'process_request'\n  from lib/kemal/src/kemal/route_handler.cr:17:7 in 'call'\n  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'\n  from lib/kemal/src/kemal/websocket_handler.cr:13:14 in 'call'\n  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'\n  from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call'\n  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'\n  from src/invidious/helpers/handlers.cr:157:5 in 'call'\n  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'\n  from src/invidious/helpers/handlers.cr:94:12 in 'call'\n  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'\n  from src/invidious/helpers/handlers.cr:146:5 in 'call'\n  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'\n  from src/invidious/helpers/handlers.cr:70:5 in 'call'\n  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'\n  from src/ext/kemal_static_file_handler.cr:162:16 in 'call'\n  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'\n  from lib/kemal/src/kemal/exception_handler.cr:8:7 in 'call'\n  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'\n  from src/invidious/helpers/logger.cr:17:35 in 'call'\n  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'\n  from lib/kemal/src/kemal/init_handler.cr:12:7 in 'call'\n  from /usr/share/crystal/src/http/server/request_processor.cr:51:11 in 'process'\n  from /usr/share/crystal/src/http/server.cr:521:5 in 'handle_client'\n  from /usr/share/crystal/src/http/server.cr:451:5 in '->'\n  from /usr/share/crystal/src/fiber.cr:146:11 in 'run'\n  from /usr/share/crystal/src/fiber.cr:98:34 in '->'\n  from ???\n"
}

Screenshots

Additional context

Browser: Firefox 124.0
OS: Arch linux

[YeapGuy]

It's still broken. My healthcheck keeps failing because of this.

[ashley0143]

youtube is a/b testing a new comments response iirc

[limbuscode]

Yes, the response is different now, the comments are in another property "frameworkUpdates.entityBatchUpdate.mutations".

I wonder if it will be a permanent update...

[RundownRhino]

YCD seems to already have a fix for it: egbertbouman/youtube-comment-downloader@9850cf4
so this can be ported to invidious.

[limbuscode]

I've noted that sometimes (bare rarely) the instances output the comments correctly but most of the time it doesn't, it seems not a permanent update to the response just yet. @ChunkyProgrammer

[RundownRhino]

That's worrying if so, because if this is still in A/B testing, then moving to the new format will make it fail to parse the responses whenever youtube sends them in the old format.

[ChunkyProgrammer]

My PR doesnt remove parsing of the old format, it just adds code for parsing the new format 🙂 @limbuscode @RundownRhino

[Yetangitu]

FYI: Just built current master with @ChunkyProgrammer 's PR and tested with a number of video's, comments loaded on every attempt.