Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Postgres role hardcoded in initial tables #1772

Closed
JuniorJPDJ opened this issue Feb 14, 2021 · 8 comments · Fixed by #1678
Closed

[Bug] Postgres role hardcoded in initial tables #1772

JuniorJPDJ opened this issue Feb 14, 2021 · 8 comments · Fixed by #1678
Labels
bug Something isn't working type:documentation

Comments

@JuniorJPDJ
Copy link
Contributor

Describe the bug
Postgres role/username seems to be hardcoded as kemal even if set to another name in compose/config.

Steps to Reproduce

$ cat docker-compose.yml                                                                                                                                                                                                                     
version: '3'                                                                                                                                                                                                                                
services:                                                                                                                                                                                                                                   
  invidious:                                                                                                                                                                                                                                
    image: omarroth/invidious                                                                                                                                                                                                               
    restart: unless-stopped                                                                                                                                                                                                                 
    networks:                                                                                                                                                                                                                               
      - default                                                                                                                                                                                                                             
      - haproxy                                                                                                                                                                                                                             
      - postgres                                                                                                                                                                                                                            
    environment:                                                                                                                                                                                                                            
      INVIDIOUS_CONFIG: |                                                                                                                                                                                                                   
        channel_threads: 1                                                                                                                                                                                                                  
        check_tables: true                                                                                                                                                                                                                  
        feed_threads: 1                                                                                                                                                                                                                     
        db:                                                                                                                                                                                                                                 
          user: invidious                                                                                                                                                                                                                   
          password: <redacted>                                                                                                                                                                                        
          host: postgres                                                                                                                                                                                                                    
          port: 5432                                                                                                                                                                                                                        
          dbname: invidious                                                                                                                                                                                                                 
        full_refresh: false                                                                                                                                                                                                                 
        https_only: true                                                                                                                                                                                                                    
        domain: invidious.juniorjpdj.pl                                                                                                                                                                                                     
        external_port: 443                                                                                                                                                                                                                  
        force_resolve: ipv4                                                                                                                                                                                                                 
                                                                                                                                                                                                                                            
networks:                                                                                                                                                                                                                                   
  haproxy:                                                                                                                                                                                                                                  
    external:                                                                                                                                                                                                                               
      name: haproxy                                                                                                                                                                                                                         
  postgres:                                                                                                                                                                                                                                 
    external:
      name: postgres

Logs

$ dclf
Attaching to invidious_invidious_1
invidious_1  | 2021-02-14T00:13:29.649798505Z 2021-02-14 00:13:29 UTC [info] check_table: check_table: CREATE TABLE channels
invidious_1  | 2021-02-14T00:13:29.672997529Z Unhandled exception: role "kemal" does not exist (PQ::PQError)
invidious_1  | 2021-02-14T00:13:30.504870083Z   from lib/pg/src/pq/connection.cr:203:7 in 'handle_error'
invidious_1  | 2021-02-14T00:13:30.504889146Z   from lib/pg/src/pq/connection.cr:186:7 in 'handle_async_frames'
invidious_1  | 2021-02-14T00:13:30.504897304Z   from lib/pg/src/pq/connection.cr:162:7 in 'read'
invidious_1  | 2021-02-14T00:13:30.504901206Z   from lib/pg/src/pq/connection.cr:157:7 in 'read'
invidious_1  | 2021-02-14T00:13:30.504904671Z   from lib/pg/src/pq/query.cr:53:14 in 'initialize'
invidious_1  | 2021-02-14T00:13:30.504908077Z   from lib/pg/src/pq/query.cr:49:5 in 'new'
invidious_1  | 2021-02-14T00:13:30.504911423Z   from lib/pg/src/pg/connection.cr:30:7 in 'exec_all'
invidious_1  | 2021-02-14T00:13:30.504914780Z   from src/invidious/helpers/helpers.cr:415:7 in 'check_table'
invidious_1  | 2021-02-14T00:13:30.504918436Z   from src/invidious.cr:150:3 in '__crystal_main'
invidious_1  | 2021-02-14T00:13:30.504921721Z   from /usr/share/crystal/src/crystal/main.cr:110:5 in 'main_user_code'
invidious_1  | 2021-02-14T00:13:30.504925101Z   from /usr/share/crystal/src/crystal/main.cr:96:7 in 'main'
invidious_1  | 2021-02-14T00:13:30.504928384Z   from /usr/share/crystal/src/crystal/main.cr:119:3 in 'main'
invidious_1  | 2021-02-14T00:13:30.504931699Z   from src/env/__libc_start_main.c:94:2 in 'libc_start_main_stage2'
invidious_1  | 2021-02-14T00:13:31.541115269Z 2021-02-14 00:13:31 UTC [info] check_table: check_table: CREATE TABLE channels
invidious_1  | 2021-02-14T00:13:31.561962259Z Unhandled exception: role "kemal" does not exist (PQ::PQError)
invidious_1  | 2021-02-14T00:13:32.397669162Z   from lib/pg/src/pq/connection.cr:203:7 in 'handle_error'
invidious_1  | 2021-02-14T00:13:32.397700979Z   from lib/pg/src/pq/connection.cr:186:7 in 'handle_async_frames'
invidious_1  | 2021-02-14T00:13:32.397708048Z   from lib/pg/src/pq/connection.cr:162:7 in 'read'
invidious_1  | 2021-02-14T00:13:32.397713432Z   from lib/pg/src/pq/connection.cr:157:7 in 'read'
invidious_1  | 2021-02-14T00:13:32.397718746Z   from lib/pg/src/pq/query.cr:53:14 in 'initialize'
invidious_1  | 2021-02-14T00:13:32.397724013Z   from lib/pg/src/pq/query.cr:49:5 in 'new'
invidious_1  | 2021-02-14T00:13:32.397729162Z   from lib/pg/src/pg/connection.cr:30:7 in 'exec_all'
invidious_1  | 2021-02-14T00:13:32.397734264Z   from src/invidious/helpers/helpers.cr:415:7 in 'check_table'
invidious_1  | 2021-02-14T00:13:32.397739304Z   from src/invidious.cr:150:3 in '__crystal_main'
invidious_1  | 2021-02-14T00:13:32.397744438Z   from /usr/share/crystal/src/crystal/main.cr:110:5 in 'main_user_code'
invidious_1  | 2021-02-14T00:13:32.397749664Z   from /usr/share/crystal/src/crystal/main.cr:96:7 in 'main'
invidious_1  | 2021-02-14T00:13:32.397754742Z   from /usr/share/crystal/src/crystal/main.cr:119:3 in 'main'
invidious_1  | 2021-02-14T00:13:32.397759904Z   from src/env/__libc_start_main.c:94:2 in 'libc_start_main_stage2'
invidious_1  | 2021-02-14T00:13:33.484166990Z 2021-02-14 00:13:33 UTC [info] check_table: check_table: CREATE TABLE channels
invidious_1  | 2021-02-14T00:13:33.503798954Z Unhandled exception: role "kemal" does not exist (PQ::PQError)
invidious_1  | 2021-02-14T00:13:34.347686725Z   from lib/pg/src/pq/connection.cr:203:7 in 'handle_error'
invidious_1  | 2021-02-14T00:13:34.347712399Z   from lib/pg/src/pq/connection.cr:186:7 in 'handle_async_frames'
invidious_1  | 2021-02-14T00:13:34.347717301Z   from lib/pg/src/pq/connection.cr:162:7 in 'read'
invidious_1  | 2021-02-14T00:13:34.347720893Z   from lib/pg/src/pq/connection.cr:157:7 in 'read'
invidious_1  | 2021-02-14T00:13:34.347728170Z   from lib/pg/src/pq/query.cr:53:14 in 'initialize'
invidious_1  | 2021-02-14T00:13:34.347731771Z   from lib/pg/src/pq/query.cr:49:5 in 'new'
invidious_1  | 2021-02-14T00:13:34.347735073Z   from lib/pg/src/pg/connection.cr:30:7 in 'exec_all'
invidious_1  | 2021-02-14T00:13:34.347738454Z   from src/invidious/helpers/helpers.cr:415:7 in 'check_table'
invidious_1  | 2021-02-14T00:13:34.347741804Z   from src/invidious.cr:150:3 in '__crystal_main'
invidious_1  | 2021-02-14T00:13:34.347745161Z   from /usr/share/crystal/src/crystal/main.cr:110:5 in 'main_user_code'
invidious_1  | 2021-02-14T00:13:34.347748509Z   from /usr/share/crystal/src/crystal/main.cr:96:7 in 'main'
invidious_1  | 2021-02-14T00:13:34.347751833Z   from /usr/share/crystal/src/crystal/main.cr:119:3 in 'main'
invidious_1  | 2021-02-14T00:13:34.347755190Z   from src/env/__libc_start_main.c:94:2 in 'libc_start_main_stage2'
@JuniorJPDJ JuniorJPDJ added the bug Something isn't working label Feb 14, 2021
@Perflyst
Copy link
Contributor

The user is also hardcoded in the migration scripts, #976
@JuniorJPDJ please use "kemal" user for now

@gotmax23
Copy link

Funnily enough, I just had this same issue when setting this up a couple days ago. Fixing this is probably not the number one priority, but at least some documentation (for instance, a comment in docker-compose.yml) would be helpful.

@stranger-danger-zamu
Copy link

So I was able to get the $POSTGRES_USER environment variable to be used for configuring the database. After a bit of research I wasn't able to use Postgres' \set to read the environment variable.

However, since the .sql scripts are being run via shell scripts, I just used sed!

init-invidious-db.sh, migrate-db-3bcb98e.sh, and migrate-db-3646395.sh were the files I had to change to make it work.

Essentially, I used sed to replace kemal with $POSTGRES_USER.

#!/bin/bash
set -eou pipefail

psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
  CREATE USER postgres;
EOSQL

sed -r 's/kemal/'"$POSTGRES_USER"'/' config/sql/channels.sql | psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" 
sed -r 's/kemal/'"$POSTGRES_USER"'/' config/sql/videos.sql | psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" 
sed -r 's/kemal/'"$POSTGRES_USER"'/' config/sql/channel_videos.sql | psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" 
sed -r 's/kemal/'"$POSTGRES_USER"'/' config/sql/users.sql  | psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" 
sed -r 's/kemal/'"$POSTGRES_USER"'/' config/sql/session_ids.sql  | psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" 
sed -r 's/kemal/'"$POSTGRES_USER"'/' config/sql/nonces.sql | psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" 
sed -r 's/kemal/'"$POSTGRES_USER"'/' config/sql/annotations.sql  | psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" 
sed -r 's/kemal/'"$POSTGRES_USER"'/' config/sql/playlists.sql  | psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" 
sed -r 's/kemal/'"$POSTGRES_USER"'/' config/sql/playlist_videos.sql  | psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" 

Now, the reason I'm posting this instead of just doing a pull request with the changes is the migrate-db-*.sh files. If those are being generated as part of development then they might cause issues down the line. Should this be something that the devs do for us or should this just be documented?

@heylix
Copy link

heylix commented Mar 27, 2021

There could be a proper pgsql initialisation script which is inside the invidious container doing this.

Something like "if I can authenticate with the creds you gave me in the environment, but the tables don't exist, create them".

@gotmax23
Copy link

There could be a proper pgsql initialisation script which is inside the invidious container doing this.

Something like "if I can authenticate with the creds you gave me in the environment, but the tables don't exist, create them".

This would be a much better solution than adding the db init scripts as a volume mount to the postgres container. Also, this would allow running Invidious in Docker using the prebuilt images without a local copy of the repo.

@JuniorJPDJ
Copy link
Contributor Author

JuniorJPDJ commented Apr 6, 2021

@stranger-danger-zamu how about your PR?

@JuniorJPDJ
Copy link
Contributor Author

bump

@SamantazFox
Copy link
Member

Now, the reason I'm posting this instead of just doing a pull request with the changes is the migrate-db-*.sh files. If those are being generated as part of development then they might cause issues down the line. Should this be something that the devs do for us or should this just be documented?

Those files are maintained manually. They are here to help database migrations. We are aware that this is not the best solution, and that in the future, this should be done automatically, from the code, and not from the shell.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working type:documentation
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

7 participants