Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slow performance on ARM64 docker image #3740

Closed
Tan-tan-san opened this issue Apr 12, 2023 · 18 comments
Closed

Slow performance on ARM64 docker image #3740

Tan-tan-san opened this issue Apr 12, 2023 · 18 comments

Comments

@Tan-tan-san
Copy link

I have recently changed from an AMD64 docker image to an ARM64 for my invidious instance. Everything ran great on the x86 server deployment in docker using docker-compose. I've copied over the exact config, but changed the image to:

quay.io/invidious/invidious:latest-arm64

I am experiencin terrible performance since the switch. The thumbnails don't really load, but it shows the text for latest and trending. Search function is also fairly slow. When I open a video I can eventually get it to load and play, but it buffers constantly or may never play at all. I have tried accessing it through my swag nginx reverse proxy or on my internal network with multiple devices and can replicate the same issue. The hardware should be more than enough to run invidious. Specs are below:

_Orange Pi 5 16G memory unit
RK3588 CPU
500G M.2 NVME
Docker version 23.0.3, build 3e7cbfd

ls_release -a
Distributor ID: Debian
Description: Debian GNU/Linux 11 (bullseye)
Release: 11
Codename: bullseye_

docker-compose.yml file:

version: "3.4"
services:
  postgres:
    image: docker.io/library/postgres:13
    restart: always
    volumes:
    - /home/<user>/invidious/postgresdata:/var/lib/postgresql/data
    - /home/<user>/invidious/config/sql:/config/sql
    - /home/<user>/invidious/docker/init-invidious-db.sh:/docker-entrypoint-initdb.d/init-invidious-db.sh
    environment:
      POSTGRES_DB: invidious
      POSTGRES_USER: kemal
      POSTGRES_PASSWORD: kemal
  invidious:
    image: quay.io/invidious/invidious:latest-arm64
    restart: always
    # I had to publish port 9872 because port 3000 was already in use on the server, but you could go with the standard 3000:3000 otherwise
    ports:
    - 3005:3000
    environment:
      INVIDIOUS_CONFIG: |
        domain: <hiding domain> #comment out if you don't publish out
        dark_mode: true
        channel_threads: 1
        check_tables: true
        feed_threads: 1
        db:
          dbname: invidious
          user: kemal
          password: kemal
          host: postgres
          port: 5432
          full_refresh: false
          registration_enabled: false
          https_only: true
          hsts: true
          admins: [adminuser]
          depends_on:
          - postgres
volumes:
  postgresdata:

invdious container log snippet:

2023-04-12 19:18:13 UTC [info] 200 GET /feed/popular 2.35ms
2023-04-12 19:18:13 UTC [info] 200 GET /css/pure-min.css?v=961cae2 1.14ms
2023-04-12 19:18:13 UTC [info] 200 GET /css/grids-responsive-min.css?v=961cae2 552.99µs
2023-04-12 19:18:13 UTC [info] 200 GET /js/themes.js?v=961cae2 233.04µs
2023-04-12 19:18:13 UTC [info] 200 GET /css/default.css?v=961cae2 771.15µs
2023-04-12 19:18:13 UTC [info] 200 GET /css/ionicons.min.css?v=961cae2 2.49ms
2023-04-12 19:18:13 UTC [info] 200 GET /js/_helpers.js?v=961cae2 641.36µs
2023-04-12 19:18:13 UTC [info] 200 GET /js/handlers.js?v=961cae2 375.66µs
2023-04-12 19:18:13 UTC [info] 200 GET /fonts/ionicons.woff2?v=4.6.3 5.32ms
2023-04-12 19:18:13 UTC [info] 200 GET /opensearch.xml 598.49µs
2023-04-12 19:18:13 UTC [info] 200 GET /favicon-32x32.png?v=961cae2 52.21µs
2023-04-12 19:18:13 UTC [info] 200 GET /site.webmanifest?v=961cae2 95.37µs
2023-04-12 19:18:20 UTC [info] 200 GET /feed/popular 2.08ms
2023-04-12 19:18:28 UTC [info] 200 GET /search 4763.68ms
2023-04-12 19:18:28 UTC [info] 200 GET /css/search.css 370.41µs
2023-04-12 19:20:42 UTC [info] 200 GET /ggpht/ytc/AGIKgqMCRAejMuaXQxxh2cFPT1jh6CCsVNaZnZ1jHan9mg=s176-c-k-c0x00ffffff-no-rj-mo 133703.78ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/cgBLF5LjRhs/mqdefault.jpg 129660.0ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/9WzHtujBRZQ/mqdefault.jpg 125615.53ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/JUzJ2uqehDg/mqdefault.jpg 121509.98ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/_-vt0AIE-KQ/mqdefault.jpg 117450.45ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/qcEJndbIcek/mqdefault.jpg 113409.1ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/vXM-0qZaZEM/mqdefault.jpg 109369.66ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/VM2MRdKclrs/mqdefault.jpg 105326.6ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/XtfL6AgAemA/mqdefault.jpg 101254.27ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/f4AOMat2GTI/mqdefault.jpg 97212.56ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/ydIpFsQVnpY/mqdefault.jpg 93163.93ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/N-Xn4Vy_WrE/mqdefault.jpg 89124.95ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/4ZYquP_Y-HQ/mqdefault.jpg 85085.76ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/V5mwkVp-_OQ/mqdefault.jpg 81040.4ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/9LwRgLVmod4/mqdefault.jpg 76994.29ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/cJ4iULqi53k/mqdefault.jpg 72876.8ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/XPCU5e8dBsQ/mqdefault.jpg 68859.58ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/PSSZY9JJU4M/mqdefault.jpg 64815.62ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/ybCs8bvxcS0/mqdefault.jpg 60760.56ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/y7tJ-KyMV9E/mqdefault.jpg 56702.31ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/XHHvaLgwnWY/mqdefault.jpg 52659.82ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/Bg8spPAxFSA/mqdefault.jpg 48618.33ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/-JO47GbsVvc/mqdefault.jpg 44577.18ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/sREPrq875P8/mqdefault.jpg 40515.91ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/BZuMer0hTXg/mqdefault.jpg 36477.31ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/xTKKxu8dfKo/mqdefault.jpg 32438.01ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/fw5z8uUIgrg/mqdefault.jpg 28399.96ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/x8JSCZg-mts/mqdefault.jpg 24357.19ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/1AvaJq3Zlpc/mqdefault.jpg 20313.97ms
2023-04-12 19:20:42 UTC [info] 200 GET /vi/SQlg0Ukfjmo/mqdefault.jpg 16273.55ms
2023-04-12 19:21:02 UTC [info] 200 GET /vi/yU08TSe0bQc/mqdefault.jpg 32346.14ms
2023-04-12 19:21:02 UTC [info] 200 GET /vi/UiAMFecwUi8/mqdefault.jpg 28385.4ms
2023-04-12 19:21:02 UTC [info] 200 GET /vi/RM9zD8t5lOI/mqdefault.jpg 24299.38ms
2023-04-12 19:21:03 UTC [info] 200 GET /watch?v=cgBLF5LjRhs 20955.76ms
2023-04-12 19:21:03 UTC [info] 200 GET /watch?v=cgBLF5LjRhs 21175.09ms
2023-04-12 19:21:59 UTC [info] 302 GET / 3.91ms
2023-04-12 19:21:59 UTC [info] 200 GET /feed/popular 2.56ms

postgresdb container log snippet:

2023-04-12 15:58:43.622 UTC [1] LOG:  database system is ready to accept connections
2023-04-12 15:59:52.935 UTC [34] ERROR:  relation "channels" does not exist at character 15
2023-04-12 15:59:52.935 UTC [34] STATEMENT:  SELECT * FROM channels LIMIT 0
2023-04-12 15:59:52.959 UTC [34] ERROR:  relation "channel_videos" does not exist at character 15
2023-04-12 15:59:52.959 UTC [34] STATEMENT:  SELECT * FROM channel_videos LIMIT 0
2023-04-12 15:59:52.972 UTC [34] ERROR:  relation "playlists" does not exist at character 15
2023-04-12 15:59:52.972 UTC [34] STATEMENT:  SELECT * FROM playlists LIMIT 0
2023-04-12 15:59:52.985 UTC [34] ERROR:  relation "playlist_videos" does not exist at character 15
2023-04-12 15:59:52.985 UTC [34] STATEMENT:  SELECT * FROM playlist_videos LIMIT 0
2023-04-12 15:59:52.996 UTC [34] ERROR:  relation "nonces" does not exist at character 15
2023-04-12 15:59:52.996 UTC [34] STATEMENT:  SELECT * FROM nonces LIMIT 0
2023-04-12 15:59:53.008 UTC [34] ERROR:  relation "session_ids" does not exist at character 15
2023-04-12 15:59:53.008 UTC [34] STATEMENT:  SELECT * FROM session_ids LIMIT 0
2023-04-12 15:59:53.021 UTC [34] ERROR:  relation "users" does not exist at character 15
2023-04-12 15:59:53.021 UTC [34] STATEMENT:  SELECT * FROM users LIMIT 0
2023-04-12 15:59:53.034 UTC [34] ERROR:  relation "videos" does not exist at character 15
2023-04-12 15:59:53.034 UTC [34] STATEMENT:  SELECT * FROM videos LIMIT 0

Screenshot of experience:
Capture

@blackandcold
Copy link

blackandcold commented Apr 13, 2023

Hi, this is not ARM related - I have spun up an instance yesterday and it is so slow, that thumbnails take hours to load or don't load at all. A click takes minutes... x64 system

Edit: it also is the same on other instances which worked well until now
https://invidious.tiekoetter.com/

@unixfox
Copy link
Member

unixfox commented Apr 13, 2023

Do you hourly restart your instance?

@Tan-tan-san
Copy link
Author

Do you hourly restart your instance?

I do not. I had been restarting the instance a few times to capture logs and data with different config settings.

@blackandcold
Copy link

blackandcold commented Apr 13, 2023

It does not matter if I start it fresh or it runs 5 hours - same thing happening. I have no useful logs for now either.

After restarts it seems even slower.
200 GET /preferences?referer=%2Ffeed%2Fsubscriptions 44202.24ms

I have 32GB RAM, 2400G on Arch and 300MBit Downstream with 7ms latency.
Could it be European CDN or something?

invidious-invidious-1     | [development] Invidious is ready to lead at http://0.0.0.0:3000
invidious-invidious-1     | 2023-04-13 19:21:59 UTC [info] jobs: running ClearExpiredItems job
invidious-invidious-1     | 2023-04-13 19:22:35 UTC [error] RefreshChannelsJob: UC9-y-6csu5WGm29I7JiwpnA : **SSL_connect: Unexpected EOF**
invidious-invidious-1     | 2023-04-13 19:22:35 UTC [error] RefreshChannelsJob: UC9-y-6csu5WGm29I7JiwpnA fiber : backing off for 00:02:00s
invidious-invidious-1     | 2023-04-13 19:22:35 UTC [error] RefreshChannelsJob: UC5UAwBUum7CPN5buc-_N1Fw : **SSL_connect: Unexpected EOF**
invidious-invidious-1     | 2023-04-13 19:22:35 UTC [error] RefreshChannelsJob: UC5UAwBUum7CPN5buc-_N1Fw fiber : backing off for 00:02:00s
invidious-invidious-1     | 2023-04-13 19:22:51 UTC [info] jobs: ClearExpiredItems done.
invidious-invidious-1     | 2023-04-13 19:22:55 UTC [error] RefreshChannelsJob: UCdBK94H6oZT2Q7l0-b0xmMg : **Unexpected end of http response**
invidious-invidious-1     | 2023-04-13 19:22:55 UTC [error] RefreshChannelsJob: UCdBK94H6oZT2Q7l0-b0xmMg fiber : backing off for 00:02:00s
invidious-invidious-1     | 2023-04-13 19:22:55 UTC [info] **200 GET /feed/subscriptions 16081.02ms**
invidious-invidious-db-1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
...
invidious-invidious-1  | 2023-04-13 19:28:41 UTC [error] **YoutubeAPI: Got error 503 when requesting /youtubei/v1/next**
invidious-invidious-1  | 2023-04-13 19:28:41 UTC [error] **YoutubeAPI: The service is currently unavailable.**
...
invidious-invidious-1     | 2023-04-13 19:38:19 UTC [info] 200 GET /feed/subscriptions 46.39ms
invidious-invidious-1     | 2023-04-13 19:38:43 UTC [info] 200 GET /vi/1vpepaQ-VQQ/mqdefault.jpg 24200.87ms
invidious-invidious-1     | 2023-04-13 19:38:43 UTC [info] 200 GET /vi/1NAmLp5i4Ps/mqdefault.jpg 20176.23ms
invidious-invidious-1     | 2023-04-13 19:38:43 UTC [info] 200 GET /vi/91IPJ6LFmto/mqdefault.jpg 16160.5ms
invidious-invidious-1     | 2023-04-13 19:38:55 UTC [info] 200 GET /vi/hCnPENkfaiw/mqdefault.jpg 16090.59ms
invidious-invidious-1     | 2023-04-13 19:39:03 UTC [info] 200 GET /api/v1/comments/jNQXAC9IVRw 20129.42ms
invidious-invidious-1     | 2023-04-13 19:39:03 UTC [info] 200 GET /vi/zy3XmyfTB44/mqdefault.jpg 20129.58ms
invidious-invidious-1     | 2023-04-13 19:39:03 UTC [info] 200 GET /vi/nQ2A30cD3Q8/mqdefault.jpg 16107.42ms
invidious-invidious-1     | 2023-04-13 19:39:07 UTC [info] 200 GET /vi/F-N3PMmHEMo/mqdefault.jpg 16110.5ms
invidious-invidious-1     | 2023-04-13 19:39:19 UTC [info] 200 GET /vi/_rGXIXyNqpk/mqdefault.jpg 16077.2ms
invidious-invidious-1     | 2023-04-13 19:39:19 UTC [info] 200 GET /vi/jJ9tqG7I7s4/mqdefault.jpg 48354.33ms
invidious-invidious-1     | 2023-04-13 19:39:19 UTC [info] 200 GET /vi/l9dItRUjQ0k/mqdefault.jpg 44284.2ms
invidious-invidious-1     | 2023-04-13 19:39:31 UTC [info] 200 GET /vi/KCPuw7UDNag/mqdefault.jpg 16071.68ms
invidious-invidious-1     | 2023-04-13 19:39:39 UTC [info] 200 GET /api/v1/comments/jNQXAC9IVRw 20105.27ms
invidious-invidious-1     | 2023-04-13 19:39:39 UTC [info] 200 GET /vi/kZ9ntfjytTI/mqdefault.jpg 20105.24ms
invidious-invidious-1     | 2023-04-13 19:39:39 UTC [info] 200 GET /vi/6NoTuqDAkfg/mqdefault.jpg 16091.84ms
invidious-invidious-1     | 2023-04-13 19:39:44 UTC [info] 200 GET /vi/GgKeWLy8cno/mqdefault.jpg 16095.33ms
invidious-invidious-1     | 2023-04-13 19:39:56 UTC [info] 200 GET /vi/ODXbN-KGY6A/mqdefault.jpg 16100.86ms
invidious-invidious-1     | 2023-04-13 19:39:56 UTC [info] 200 GET /vi/GDSf2h9_39I/mqdefault.jpg 48266.55ms
invidious-invidious-1     | 2023-04-13 19:39:56 UTC [info] 200 GET /vi/U2Yw-UbS_88/mqdefault.jpg 44252.96ms
invidious-invidious-1     | 2023-04-13 19:40:08 UTC [info] 200 GET /vi/LxY_oIohm-c/mqdefault.jpg 16116.76ms
invidious-invidious-1     | 2023-04-13 19:40:16 UTC [info] 200 GET /api/v1/comments/jNQXAC9IVRw 20111.33ms
invidious-invidious-1     | 2023-04-13 19:40:16 UTC [info] 200 GET /vi/1ugJ1BJx0HE/mqdefault.jpg 20111.02ms
invidious-invidious-1     | 2023-04-13 19:40:16 UTC [info] 200 GET /vi/eUL8EvZkfEY/mqdefault.jpg 16086.97ms
invidious-invidious-1     | 2023-04-13 19:40:20 UTC [info] 200 GET /vi/rQ6YDCCH1Nc/mqdefault.jpg 16083.07ms

@Tan-tan-san
Copy link
Author

I'm in North America, so it doesn't seem like its a CDN issue.

@blackandcold
Copy link

blackandcold commented Apr 14, 2023

DB version from 13 to 14 (development yml has 13 in it!) and fetch the image, don't build it from scratch.

Need to test some stuff and I'll come back in a few days.

@blackandcold
Copy link

Working now quicker than public instances.

I changed my DNS from 1.0.0.1 and 1.1.1.1 to ISP defaults (maybe also other DNS filters like AdGuard will interfere?).
Also a subdomain does now exist (in the config invidious.f.q.d.com made up one) with valid certificate and Apache 2.4 reverse proxy.

Apache settings:

<VirtualHost *:443>
        SSLEngine on
        ServerName invidious.f.q.d.com
        SSLCertificateKeyFile /etc/letsencrypt/live/invidious.f.q.d.com/privkey.pem
        SSLCertificateFile /etc/letsencrypt/live/invidious.f.q.d.com/fullchain.pem

        <Location />
                Require all granted
                ProxyPreserveHost on
                ProxyPass http://192.168.0.2:3000/
                ProxyPassReverse http://192.168.0.2:3000/
        </Location>

        ErrorLog "/var/log/httpd/invidious.f.q.d.com-error_log"
        CustomLog "/var/log/httpd/invidious.f.q.d.com-access_log" common

</VirtualHost>

<Virtualhost *:80>
    ServerName invidious.f.q.d.com
    Redirect "/" "https://invidious.f.q.d.com/"
</Virtualhost>

When switching from Postgresql 13 to 14 you need to delete the volume "postgresdata" volume because the data is not compatible. Do a full export of your subs and watches before and reimport later.

docker volumes ls
docker volumes rm ..._postgresdata

Invidious docker-compose:


version: "3"
services:
  invidious:
    image: quay.io/invidious/invidious:latest
    restart: unless-stopped
    ports:
      - "3000:3000"
    environment:
      INVIDIOUS_CONFIG: |
        admins: ["admin"]
        #log_level: Debug
        registration_enabled: false
        db:
          dbname: invidious
          user: kemal
          password: kemal
          host: invidious-db
          port: 5432
        channel_threads: 20
        feed_threads: 8
        check_tables: true
        external_port: 443
        domain: invidious.f.q.d.com
        https_only: true
        statistics_enabled: true
        hmac_key: XXX
        use_pubsub_feeds: true
    healthcheck:
      test: wget -nv --tries=1 --spider http://127.0.0.1:3000/api/v1/comments/jNQXAC9IVRw || exit 1
      interval: 30s
      timeout: 5s
      retries: 2
    logging:
      options:
        max-size: "1G"
        max-file: "4"
    depends_on:
      - invidious-db

  invidious-db:
    image: docker.io/library/postgres:14
    restart: unless-stopped
    volumes:
      - postgresdata:/var/lib/postgresql/data
      - ./config/sql:/config/sql
      - ./docker/init-invidious-db.sh:/docker-entrypoint-initdb.d/init-invidious-db.sh
    environment:
      POSTGRES_DB: invidious
      POSTGRES_USER: kemal
      POSTGRES_PASSWORD: kemal
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"]

volumes:
  postgresdata:

Request are now in the <500ms range instead of 30s +


invidious-invidious-1     | 2023-04-15 10:03:22 UTC [info] 200 GET /vi/9-jIplX6Wjw/mqdefault.jpg 158.56ms
invidious-invidious-1     | 2023-04-15 10:03:22 UTC [info] 200 GET /vi/nkh9VGCY8as/mqdefault.jpg 133.37ms
invidious-invidious-1     | 2023-04-15 10:03:22 UTC [info] 200 GET /vi/qbIk7-JPB2c/mqdefault.jpg 129.14ms
invidious-invidious-1     | 2023-04-15 10:03:22 UTC [info] 200 GET /vi/-IV1LL3Y17Q/mqdefault.jpg 126.14ms
invidious-invidious-1     | 2023-04-15 10:03:23 UTC [info] 200 GET /vi/W-UPEG64Yn8/mqdefault.jpg 146.46ms
invidious-invidious-1     | 2023-04-15 10:03:23 UTC [info] 200 GET /vi/bVu_X1lELY4/mqdefault.jpg 116.39ms
invidious-invidious-1     | 2023-04-15 10:03:23 UTC [info] 200 GET /vi/kyuAJgMsRsM/maxres.jpg 966.89ms
invidious-invidious-1     | 2023-04-15 10:03:23 UTC [info] 200 GET /api/v1/comments/kyuAJgMsRsM?format=html&hl=de&thin_mode=false 207.88ms
invidious-invidious-1     | 2023-04-15 10:03:23 UTC [info] 200 GET /ggpht/ytc/AGIKgqNFHMuFr70o6Nt2uFCIIwA1EzeYGAJDvPAStLSktg=s176-c-k-c0x00ffffff-no-rj 133.23ms
invidious-invidious-1     | 2023-04-15 10:03:23 UTC [info] 200 GET /ggpht/Twt-pwgo6NumklvAKSkbFrkOs0axhLw9jLzkscqHHMIQC51A1RHLcy6TzLrO8PL0jkGwFf5Q=s176-c-k-c0x00ffffff-no-rj 120.13ms
invidious-invidious-1     | 2023-04-15 10:03:23 UTC [info] 200 GET /ggpht/ytc/AGIKgqORyuzRDr_wKRBX9foz-RKduIfXZ_aqBAwod-w9=s176-c-k-c0x00ffffff-no-rj 135.35ms
invidious-invidious-1     | 2023-04-15 10:03:23 UTC 

@Tan-tan-san
Copy link
Author

I completely removed all Postgres info, changed the DB to v14 and rebuilt it from scratch. The site is faster, but the images still don't load and video watching/downloading is terribly slow.

@unixfox
Copy link
Member

unixfox commented Apr 15, 2023

Did you try what @blackandcold said? using a different DNS server

@blackandcold
Copy link

Can it be the pubsubhub?

    hmac_key: XXX
    use_pubsub_feeds: true

438 - 465
https://github.com/iv-org/invidious/blob/master/config/config.example.yml#L438

Since it requires a callback your server would also need to be reachable from the internet.

@Tan-tan-san
Copy link
Author

I completely removed all Postgres info, changed the DB to v14 and rebuilt it from scratch. The site is faster, but the images still don't load and video watching/downloading is terribly slow.

I've also tried Quad9, Cloudflare and Google DNS with the same results. Example after changes:

023-04-15 19:11:58 UTC [info] jobs: running ClearExpiredItems job
2023-04-15 19:11:58 UTC [info] jobs: ClearExpiredItems done.
2023-04-15 19:12:12 UTC [info] 302 GET / 722.74µs
2023-04-15 19:12:12 UTC [info] 200 GET /feed/popular 3.82ms
2023-04-15 19:12:22 UTC [info] 200 GET /search 4832.39ms
2023-04-15 19:14:00 UTC [info] 200 GET /vi/97q30JjEq9Y/mqdefault.jpg 97759.91ms
2023-04-15 19:14:00 UTC [info] 200 GET /vi/UsaiOEFdfs0/mqdefault.jpg 93657.28ms
2023-04-15 19:14:00 UTC [info] 200 GET /vi/PxERvbbzmb8/hqdefault.jpg 89540.76ms
2023-04-15 19:14:00 UTC [info] 200 GET /vi/eOBq__h4OJ4/mqdefault.jpg 85420.8ms
2023-04-15 19:14:00 UTC [info] 200 GET /vi/SslOHC-qji4/mqdefault.jpg 81375.35ms
2023-04-15 19:14:00 UTC [info] 200 GET /vi/seaBeltaKhw/mqdefault.jpg 77324.68ms
2023-04-15 19:14:00 UTC [info] 200 GET /vi/CgY2uxQPflY/hqdefault.jpg 73159.15ms
2023-04-15 19:14:00 UTC [info] 200 GET /vi/0_jfH6qijVY/mqdefault.jpg 69020.49ms
2023-04-15 19:14:00 UTC [info] 200 GET /vi/NZVNAEK6shc/mqdefault.jpg 64942.99ms
2023-04-15 19:14:00 UTC [info] 200 GET /vi/2tmsyweB3k4/mqdefault.jpg 60899.04ms
2023-04-15 19:14:00 UTC [info] 200 GET /vi/PiQ_eZFO2GU/mqdefault.jpg 56854.06ms
2023-04-15 19:14:00 UTC [info] 200 GET /vi/HJNR_dX8g8c/mqdefault.jpg 52745.29ms
2023-04-15 19:14:00 UTC [info] 200 GET /vi/JIbIYCM48to/mqdefault.jpg 48701.97ms
2023-04-15 19:14:00 UTC [info] 200 GET /vi/EBSdyoO3goc/mqdefault.jpg 44583.49ms
2023-04-15 19:14:00 UTC [info] 200 GET /vi/3Ar1ABlD_Vs/mqdefault.jpg 40539.94ms
2023-04-15 19:14:00 UTC [info] 200 GET /vi/sBkoT6_xK1Q/mqdefault.jpg 36496.59ms
2023-04-15 19:14:00 UTC [info] 200 GET /vi/-y4A6Q8yVx8/mqdefault.jpg 32450.85ms
2023-04-15 19:14:00 UTC [info] 200 GET /vi/JNz7tBIPrJY/mqdefault.jpg 28408.08ms
2023-04-15 19:14:00 UTC [info] 200 GET /vi/0p7oQjOIiTs/mqdefault.jpg 24350.89ms
2023-04-15 19:14:00 UTC [info] 200 GET /vi/SG6_oy72hh4/mqdefault.jpg 20304.45ms
2023-04-15 19:14:00 UTC [info] 200 GET /vi/D5HsgNtqStk/mqdefault.jpg 16255.05ms
2023-04-15 19:14:33 UTC [info] 200 GET /vi/SGudMFFa3Gw/mqdefault.jpg 36575.32ms
2023-04-15 19:14:41 UTC [info] 200 GET /vi/VhHJUXzl9ZU/mqdefault.jpg 40513.87ms
2023-04-15 19:14:41 UTC [info] 200 GET /vi/pqC-t1kcTL4/mqdefault.jpg 36464.1ms
2023-04-15 19:14:41 UTC [info] 200 GET /vi/fEZE3rm8Ma8/mqdefault.jpg 32401.22ms
2023-04-15 19:14:41 UTC [info] 200 GET /vi/7bfOllAyxlg/mqdefault.jpg 28346.38ms
2023-04-15 19:14:41 UTC [info] 200 GET /vi/Qv_Tr_BCFCQ/mqdefault.jpg 24243.98ms
2023-04-15 19:14:41 UTC [info] 200 GET /vi/2k2GINpO308/mqdefault.jpg 20204.4ms

@Tan-tan-san
Copy link
Author

After enabling 'hmac_key' and 'use_pub_sub_feeds: true' the performance is a little better, but images still wont load and searching is slow.

production] Invidious is ready to lead at http://0.0.0.0:3000
2023-04-15 20:22:53 UTC [info] jobs: running ClearExpiredItems job
2023-04-15 20:22:53 UTC [info] jobs: ClearExpiredItems done.
2023-04-15 20:23:15 UTC [info] 302 GET / 724.2µs
2023-04-15 20:23:16 UTC [info] 200 GET /feed/popular 3.6ms
2023-04-15 20:23:18 UTC [info] 200 GET /feed/popular 846.11µs
2023-04-15 20:23:25 UTC [info] 200 GET /feed/trending 4853.92ms
                2023-04-15 20:25:03 UTC [info] 200 GET /vi/STPzH0Uob54/mqdefault.jpg 93471.61ms
2023-04-15 20:25:07 UTC [info] 200 GET /vi/FSZYpDk9Xm8/mqdefault.jpg 93436.77ms
2023-04-15 20:25:07 UTC [info] 200 GET /vi/uHGShqcAHlQ/mqdefault.jpg 89322.66ms
2023-04-15 20:25:07 UTC [info] 200 GET /vi/SAheAwnUbcE/mqdefault.jpg 85272.5ms
2023-04-15 20:25:07 UTC [info] 200 GET /vi/7pYqbVztRtk/mqdefault.jpg 81186.81ms
2023-04-15 20:25:07 UTC [info] 200 GET /vi/LLKvNzHHe6k/mqdefault.jpg 77113.78ms
2023-04-15 20:25:07 UTC [info] 200 GET /vi/VdIWm3Vgiv0/mqdefault.jpg 73015.19ms
2023-04-15 20:25:07 UTC [info] 200 GET /vi/T3R8onC1yM0/mqdefault.jpg 68902.59ms
2023-04-15 20:25:07 UTC [info] 200 GET /vi/rWVAzS5duAs/mqdefault.jpg 64855.93ms
2023-04-15 20:25:07 UTC [info] 200 GET /vi/hRcQRwjVhP0/mqdefault.jpg 60739.97ms
2023-04-15 20:25:07 UTC [info] 200 GET /vi/ldwoiMHiyL8/mqdefault.jpg 56694.44ms
2023-04-15 20:25:07 UTC [info] 200 GET /vi/yzmSlPiaeRU/mqdefault.jpg 52579.13ms
2023-04-15 20:25:07 UTC [info] 200 GET /vi/IzPQ_jA00bk/mqdefault.jpg 48561.73ms
2023-04-15 20:25:07 UTC [info] 200 GET /vi/r-tTeTyi1Go/mqdefault.jpg 44520.58ms
2023-04-15 20:25:07 UTC [info] 200 GET /vi/SQymtMNE9BU/mqdefault.jpg 40504.23ms
2023-04-15 20:25:07 UTC [info] 200 GET /vi/1W31eskA2Ck/mqdefault.jpg 36455.75ms
2023-04-15 20:25:07 UTC [info] 200 GET /vi/tHxip2x-PLc/mqdefault.jpg 32406.76ms
2023-04-15 20:25:07 UTC [info] 200 GET /vi/x14_1gqeGvk/mqdefault.jpg 28389.71ms
2023-04-15 20:25:07 UTC [info] 200 GET /vi/9zvRzqO7yOA/mqdefault.jpg 24341.32ms
2023-04-15 20:25:07 UTC [info] 200 GET /vi/ZFjIMowsudA/mqdefault.jpg 20301.49ms
2023-04-15 20:25:07 UTC [info] 200 GET /vi/r7TYA7UcQlE/mqdefault.jpg 16257.26ms
2023-04-15 20:25:11 UTC [info] 200 GET /feed/popular 1.35ms
2023-04-15 20:25:15 UTC [info] 200 GET /vi/689R_j7sH_g/mqdefault.jpg 16374.21ms
2023-04-15 20:25:15 UTC [info] 200 GET /search 12376.61ms
2023-04-15 20:25:19 UTC [info] 200 GET /vi/gMq-I0dejjE/mqdefault.jpg 113970.85ms
2023-04-15 20:25:19 UTC [info] 200 GET /vi/ejYuPwhPJZg/mqdefault.jpg 24615.28ms
2023-04-15 20:25:20 UTC [info] 200 GET /feed/trending 12901.56ms
2023-04-15 20:25:20 UTC [info] 200 GET /watch?v=STPzH0Uob54 4794.73ms
2023-04-15 20:25:20 UTC [info] 200 GET /search 9000.64ms
2023-04-15 20:25:32 UTC [info] 302 GET /latest_version?id=STPzH0Uob54&itag=22 18.9ms
2023-04-15 20:25:32 UTC [info] 200 GET /ggpht/xyCDszZ4jHXdHCfVscyGsWO3Pc4k1yoVjMUI6FgIj_APgq6Tfa3j5GKQM5XfcCntv0axVPJB=s48-c-k-c0x00ffffff-no-rj 12265.07ms

@Tan-tan-san
Copy link
Author

While searching, trending, popular and general operations are slow, downloads are much faster now after the recent change. It just takes a long time for the search and page load of the video with no thumbnail.

@blackandcold
Copy link

well, all that is left are the logging block in the config and the thread counts. neither should influence search, thumbnails or general operations.
Is in the beginning of the log somewhere "development" written? Maybe the ARM image is running in dev mode?
Other than that we would need some developer fluent in crystal and the framework here.

@Tan-tan-san
Copy link
Author

Want to confirm after playing with some config settings. Still the same issue with slowness, but at least when I click to play the video, it plays flawlessly. Just still no thumbnails and terribly slow navigation between pages. Not sure what else to check. Logs aren't reporting anything really other than very long load times.

@Tan-tan-san
Copy link
Author

well, all that is left are the logging block in the config and the thread counts. neither should influence search, thumbnails or general operations. Is in the beginning of the log somewhere "development" written? Maybe the ARM image is running in dev mode? Other than that we would need some developer fluent in crystal and the framework here.

I have confirmed that development is not in any logs. If I knew where to look at the code, I would investigate myself.

@SamantazFox
Copy link
Member

I'd highly recommend running multiple docker containers in parallel.
Invidious is currently single threaded for multiple reasons, so if your instance is getting a lot of requests, it could spend quite a lot of time switching between concurrent branches (known as fibers in Crystal).

Here is as an example the docker-compose config I'm using on my instance.
It's spawning a total of 7 replicas, which are auto-restarted as needed.
Note: the ports section doesn't work with podman-compose (See: containers/podman-compose#244)

version: "3"
services:

  invidious:
    image: quay.io/invidious/invidious:latest
    restart: unless-stopped
    deploy:
      mode: replicated
      replicas: 7
      endpoint_mode: vip
      restart_policy:
        condition: any
        delay: 30s
        window: 30s
      resources:
        limits:
          cpus: '0.75'
          memory: 1500M
        reservations:
          cpus: '0.15'
          memory: 400M
    ports:
      - "127.0.0.1:3000-3006:3000"
    # The rest of your config goes here

@Tan-tan-san
Copy link
Author

I reinstalled the same OS on a new nvme and deployed the same exact docker-compose files and it worked flawlessly. Not sure what the original problem was, but all works great now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants