Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Refresh channels throws several exceptions - Youtube probably broke something again #3820

Closed
Sommerwiesel opened this issue May 25, 2023 · 9 comments
Closed

[Bug] Refresh channels throws several exceptions - Youtube probably broke something again #3820

Sommerwiesel opened this issue May 25, 2023 · 9 comments
Labels
bug Something isn't working

Comments

@Sommerwiesel
Copy link

Sommerwiesel commented May 25, 2023 •
edited by SamantazFox

Describe the bug
Some Channel feeds are not getting updated anymore.
Invidious logs is full of exceptions related to the refresh cahnnels job.

Steps to Reproduce
Tested on my own instance https://invidious.nerdvpn.de

  1. Go to any feed (i.e. /feed/subscriptions)
  2. See old results
  3. Check logs and see errors

Environment info
2023.05.23-e238c08 @ master
Runs via docker-compose file from official invidious docs

Logs
Taken from docker logs <docker_container>

Exception: Index out of bounds (IndexError)
  from /usr/share/crystal/src/indexable.cr:914:29 in '[]'
  from /usr/share/crystal/src/indexable.cr:897:8 in 'decode_date'
  from src/invidious/yt_backend/extractors.cr:74:83 in 'process'
  from src/invidious/yt_backend/extractors.cr:420:15 in 'process'
  from src/invidious/yt_backend/extractors.cr:783:17 in 'parse_item'
  from src/invidious/yt_backend/extractors.cr:830:14 in 'extract_items'
  from src/invidious/channels/videos.cr:106:12 in 'videos'
  from lib/kemal/src/kemal/route.cr:13:9 in '->'
  from src/invidious/helpers/handlers.cr:30:37 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/init_handler.cr:12:7 in 'process'
  from /usr/share/crystal/src/http/server.cr:500:5 in '->'
  from /usr/share/crystal/src/fiber.cr:146:11 in 'run'
  from ???

And some more:

2023-05-25 17:47:16 UTC [error] YoutubeAPI: Got error 500 when requesting /youtubei/v1/browse
2023-05-25 17:47:16 UTC [error] YoutubeAPI: RET_CHECK failure (video/youtube/channels/frontend/shared/feed_filler/feed_filler.cc:200) failed_aggregates < aggregates.size() (1 vs. 1) All aggregates failed
Exception: Could not extract JSON. Youtube API returned error 500 with message:<br>"RET_CHECK failure (video/youtube/channels/frontend/shared/feed_filler/feed_filler.cc:200) failed_aggregates < aggregates.size() (1 vs. 1) All aggregates failed" (InfoException)
  from /usr/share/crystal/src/http/client/response.cr:131:7 in '_post_json'
  from src/invidious/channels/videos.cr:106:12 in 'videos'
  from lib/kemal/src/kemal/route.cr:13:9 in '->'
  from src/invidious/helpers/handlers.cr:30:37 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/init_handler.cr:12:7 in 'process'
  from /usr/share/crystal/src/http/server.cr:500:5 in '->'
  from /usr/share/crystal/src/fiber.cr:146:11 in 'run'
  from ???

Exception: Index out of bounds (IndexError)
  from /usr/share/crystal/src/indexable.cr:914:29 in '[]'
  from /usr/share/crystal/src/indexable.cr:897:8 in 'decode_date'
  from src/invidious/yt_backend/extractors.cr:74:83 in 'process'
  from src/invidious/yt_backend/extractors.cr:420:15 in 'process'
  from src/invidious/yt_backend/extractors.cr:783:17 in 'parse_item'
  from src/invidious/yt_backend/extractors.cr:830:14 in 'extract_items'
  from src/invidious/channels/videos.cr:106:12 in 'videos'
  from lib/kemal/src/kemal/route.cr:13:9 in '->'
  from src/invidious/helpers/handlers.cr:30:37 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/init_handler.cr:12:7 in 'process'
  from /usr/share/crystal/src/http/server.cr:500:5 in '->'
  from /usr/share/crystal/src/fiber.cr:146:11 in 'run'
  from ???

  from /usr/share/crystal/src/http/server/response.cr:216:9 in 'unbuffered_write'
  from /usr/share/crystal/src/compress/deflate/writer.cr:93:7 in 'consume_output'
  from /usr/share/crystal/src/compress/deflate/writer.cr:73:11 in 'close'
  from /usr/share/crystal/src/compress/gzip/writer.cr:104:5 in 'close'
  from /usr/share/crystal/src/compress/gzip/writer.cr:55:25 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/init_handler.cr:12:7 in 'process'
  from /usr/share/crystal/src/http/server.cr:500:5 in '->'
  from /usr/share/crystal/src/fiber.cr:146:11 in 'run'
  from ???

Caused by: Error writing to socket: Broken pipe (IO::Error)
  from /usr/share/crystal/src/io/evented.cr:156:7 in 'unbuffered_write'
  from /usr/share/crystal/src/http/server/response.cr:233:11 in 'unbuffered_write'
  from /usr/share/crystal/src/compress/deflate/writer.cr:93:7 in 'consume_output'
  from /usr/share/crystal/src/compress/deflate/writer.cr:73:11 in 'close'
  from /usr/share/crystal/src/compress/gzip/writer.cr:104:5 in 'close'
  from /usr/share/crystal/src/compress/gzip/writer.cr:55:25 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/init_handler.cr:12:7 in 'process'
  from /usr/share/crystal/src/http/server.cr:500:5 in '->'
  from /usr/share/crystal/src/fiber.cr:146:11 in 'run'
  from ???
@Sommerwiesel Sommerwiesel added the bug Something isn't working label May 25, 2023
@Sommerwiesel Sommerwiesel changed the title [Bug] Refresh channels throws several exceptions [Bug] Refresh channels throws several exceptions - Youtube probably broke something again May 25, 2023
@SamantazFox
Copy link
Member

Are we really breaking Youtube's C++ that often now?!

@garoto
Copy link

garoto commented May 26, 2023

yewtu.be seems to somewhat still successfully be able to refresh my subs feed, which I share between three other instances, nerdvpn being one of them.

@Sommerwiesel
Copy link
Author

Sommerwiesel commented May 27, 2023 •
edited

Today, I started getting errors in the Frontend for a lot of Videos.

Title: The video returned by YouTube isn't the requested one. (WEB client) (VideoNotAvailableException)
Date: 2023-05-27T06:48:55Z
Route: /watch?v=MKk1u5RMTn4
Version: 2023.05.25-381a0e3 @ master

Backtrace 

The video returned by YouTube isn't the requested one. (WEB client) (VideoNotAvailableException)
  from /usr/share/crystal/src/array.cr:114:31 in 'extract_video_info:video_id'
  from src/invidious/videos.cr:377:10 in 'fetch_video'
  from src/invidious/videos.cr:365:13 in 'get_video:region'
  from src/invidious/routes/watch.cr:63:15 in 'handle'
  from lib/kemal/src/kemal/route.cr:13:9 in '->'
  from src/invidious/helpers/handlers.cr:30:37 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/init_handler.cr:12:7 in 'process'
  from /usr/share/crystal/src/http/server.cr:500:5 in '->'
  from /usr/share/crystal/src/fiber.cr:146:11 in 'run'
  from ???

Please help my instance is slowly becoming completely broken :(

Also, it doesn't seem to affect other instances.

@unixfox
Copy link
Member

unixfox commented May 27, 2023

Today, I started getting errors in the Frontend for a lot of Videos.

Title: The video returned by YouTube isn't the requested one. (WEB client) (VideoNotAvailableException) Date: 2023-05-27T06:48:55Z Route: /watch?v=MKk1u5RMTn4 Version: 2023.05.25-381a0e3 @ master
Backtrace

Please help my instance is slowly becoming completely broken :(

Also, it doesn't seem to affect other instances.

For this bug let's keep discussing in #3822.

@delendum
Copy link

My instance (invidious.pufe.org) has been occasionally experiencing this too:

Title: Index out of bounds (IndexError)Date:2023-05-29T08:25:08ZRoute:/feed/trendingVersion:2023.05.25-381a0e32 @ master`

Backtrace 

Index out of bounds (IndexError)
  from /usr/share/crystal/src/array.cr:114:31 in '[]'
  from /usr/share/crystal/src/indexable.cr:949:8 in 'decode_date'
  from src/invidious/yt_backend/extractors.cr:74:83 in 'process'
  from src/invidious/yt_backend/extractors.cr:783:17 in 'parse_item'
  from src/invidious/yt_backend/extractors.cr:363:18 in 'process'
  from src/invidious/yt_backend/extractors.cr:783:17 in 'parse_item'
  from src/invidious/yt_backend/extractors.cr:830:14 in 'extract_items'
  from src/invidious/trending.cr:21:14 in 'trending'
  from src/invidious/helpers/handlers.cr:30:37 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:94:12 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:151:29 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from src/ext/kemal_static_file_handler.cr:106:14 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/request_processor.cr:51:11 in 'process'
  from /usr/share/crystal/src/http/server.cr:502:5 in '->'
  from /usr/share/crystal/src/fiber.cr:146:11 in 'run'
  from ???

`

I've noticed health checks are showing it down with status 500 quite often now too.

`Up 2023-05-29 20:29:02 200 - OK
Down 2023-05-29 20:27:59 Request failed with status code 500
Up 2023-05-29 19:05:08 200 - OK
Down 2023-05-29 19:04:02 Request failed with status code 500
Up 2023-05-29 15:31:54 200 - OK
Down 2023-05-29 15:30:50 Request failed with status code 500
Up 2023-05-29 13:45:46 200 - OK
Down 2023-05-29 13:44:42 Request failed with status code 500
Up 2023-05-29 13:18:06 200 - OK
Down 2023-05-29 13:17:00 Request failed with status code 500
Up 2023-05-29 12:05:17 200 - OK
Down 2023-05-29 12:04:11 Request failed with status code 500
Up 2023-05-29 08:40:19 200 - OK
Down 2023-05-29 08:39:15 Request failed with status code 500
Up 2023-05-29 08:27:08 200 - OK
Down 2023-05-29 08:26:05 Request failed with status code 500
Up 2023-05-29 08:03:52 200 - OK
Down 2023-05-29 08:02:45 Request failed with status code 500
Up 2023-05-29 06:29:52 200 - OK
Down 2023-05-29 06:28:47 Request failed with status code 500
Up 2023-05-29 06:02:30 200 - OK
Down 2023-05-29 06:01:26 Request failed with status code 500
Up 2023-05-29 04:06:17 200 - OK
Down 2023-05-29 04:05:13 Request failed with status code 500
Up 2023-05-29 01:28:31 200 - OK`

@omar-elnaggar

This comment was marked as outdated.

Sign in to view
@SamantazFox SamantazFox mentioned this issue May 30, 2023
Merged
@Sommerwiesel
Copy link
Author

Problems have been fixed on my instance by switching to ipv6.
Please add my instance back to the list and close this issue when done.

@unixfox

Thanks :)

@omar-elnaggar
Copy link

#3825 has fixed the channel refresh job on my instance.

Thank you!

@delendum
Copy link

delendum commented Jun 1, 2023

Yes this seems to have fixed it, nice work!

@unixfox unixfox closed this as completed Jun 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@unixfox @garoto @omar-elnaggar @SamantazFox @delendum @Sommerwiesel