Better Config

.gitignore

@@ -1,7 +1,6 @@
 .DS_Store
 node_modules
 /dist
-/env
 /coverage
 /logs
 /ts-out

Procfile

@@ -1,2 +1,2 @@
-release: yarn workspace ott-server run sequelize db:migrate
+release: yarn workspace ott-server run sequelize db:migrate && cp env/heroku.base.toml env/base.toml
 web: yarn start

README.md

@@ -37,7 +37,7 @@ yarn
 
 4. Copy and fill out the configuration file
 ```
-cp env/example.env env/production.env
+cp env/example.toml env/production.toml
 ```
 
 **Please read the [config docs here](docs/config.md) for which options are required.**
@@ -87,21 +87,21 @@ npm install -g yarn
 yarn
 ```
 3. Next you need to set up your configuration. Start by copying the example
-config in the `env` folder to a new file called `development.env`
+config in the `env` folder to a new file called `development.toml`
 ```
-cp env/example.env env/development.env
+cp env/example.toml env/development.toml
 ```
 4. Create a new project on [Google Cloud](https://console.cloud.google.com)
 5. Add "YouTube Data API v3" and "Google Drive API" to the project
 6. Obtain a YouTube API key
 7. Obtain a Google Drive API key
 	- _Not necessary if you don't plan to stream videos from Google Drive, which you probably shouldn't do anyway because Google doesn't like that._
-8. Open `env/development.env` and replace `API_KEY_GOES_HERE` with the appropriate api key.
+8. Open `env/development.toml` and put in the appropriate api keys.
 9. Initialize your local database.
 ```
 yarn workspace ott-server run sequelize-cli db:migrate
 ```
-10. Install [redis](https://redis.io). This is used to store room state and user sessions across server restarts.
+10.  Install [redis](https://redis.io). This is used to store room state and user sessions across server restarts.
 
 ## Testing
 

docker-compose.yml

@@ -6,12 +6,13 @@ services:
     environment:
       - REDIS_HOST=redis_db
       # postgres
-      - POSTGRES_DB_USERNAME=opentogethertube
-      - POSTGRES_DB_NAME=opentogethertube
-      - POSTGRES_DB_HOST=postgres_db
-      - POSTGRES_DB_PASSWORD=postgres
+      - POSTGRES_USER=opentogethertube
+      - POSTGRES_DB=opentogethertube
+      - POSTGRES_HOST=postgres_db
+      - POSTGRES_PASSWORD=postgres
       - DOCKER=1
-    env_file: env/production.env # change the path to your env file
+    volumes:
+      - ./env:/usr/app/env
     ports:
       - 8080:8080
     links:

docker/Dockerfile

@@ -39,8 +39,6 @@ COPY --from=build-stage /usr/app/docker/scripts/wait_for_db.sh /usr/app/wait_for
 # Remove all the unnecessary files
 RUN rm -rf client/public client/src client/.browserslistrc .eslintrc.js .gitignore client/vite.config.js client/babel.config.js docker-compose.yml /root/.npm
 
-RUN mkdir env && touch env/production.env
-
 RUN apk update -q && apk --no-cache add curl ffmpeg -q
 
 # Healthcheck API, WEB, REDIS

docker/README.md

@@ -8,10 +8,10 @@ You can run the application using the development image or production/deployment
 
 ## Use Image
 
-Make sure to supply the `YOUTUBE_API_KEY` in `env/production.env`
+Make sure to supply the youtube api key in `env/production.toml`
 
 ```bash
-cp env/example.env env/production.env
+cp env/example.toml env/production.toml
 ```
 
 You run the docker container using the docker-compose file in the root directory by using the following command.
@@ -29,22 +29,22 @@ The production image uses docker-compose with redis and postgres in separate con
 To use the production image, please follow the steps below.
 
 1. Next you need to set up your configuration. Start by copying the example
-   config in the `env` folder to a new file called `production.env`
+   config in the `env` folder to a new file called `production.toml`
 
 ```bash
-cp env/example.env env/production.env
+cp env/example.toml env/production.toml
 ```
 
-2. Open `env/production.env` and replace `API_KEY_GOES_HERE` with the api keys.
+1. Open `env/production.toml` and put your youtube API key in.
 
-3. To build the image locally using the prod image make sure you are in the project root `/opentogethertube`
+2. To build the image locally using the prod image make sure you are in the project root `/opentogethertube`
    before runing the command below.
 
 ```bash
-docker-compose -f docker/docker-compose.yml --env-file env/production.env up -d
+docker-compose -f docker/docker-compose.yml up -d
 ```
 
-4. After Go to http://localhost:8080/
+4. Wait for everything to finish starting, and go to http://localhost:8080/
 
 ## Debug
 
@@ -71,12 +71,3 @@ You also rebuild the docker image with the following command.
 ```bash
 docker-compose -f docker/docker-compose.yml up -d --build
 ```
-
-# Developing for OTT using Docker
-
-`docker-compose-dev.yml` is a special docker compose that is intended to assist in debugging bugs that only appear in production (which are usually bugs that occur when interacting with postgres instead of sqlite).
-
-In order to use this setup, you **must** be able to run OTT outside of docker. After that, you can start it with:
-```bash
-docker-compose -f docker/docker-compose-dev.yml up --build
-```

docker/docker-compose.yml

@@ -10,12 +10,11 @@ services:
       - REDIS_HOST=redis_db
       - FFPROBE_PATH=/usr/bin/ffprobe
       # postgres
-      - POSTGRES_DB_USERNAME=opentogethertube
-      - POSTGRES_DB_NAME=opentogethertube
-      - POSTGRES_DB_HOST=postgres_db
-      - POSTGRES_DB_PASSWORD=postgres
+      - POSTGRES_USER=opentogethertube
+      - POSTGRES_DB=opentogethertube
+      - POSTGRES_HOST=postgres_db
+      - POSTGRES_PASSWORD=postgres
       - DOCKER=1
-    env_file: ../env/production.env
     ports:
       - 8080:8080
     links:
@@ -24,6 +23,8 @@ services:
     depends_on:
       - redis_db
       - postgres_db
+    volumes:
+      - "../env:/usr/app/env"
     restart: "${DOCKER_RESTART_POLICY:-unless-stopped}"
 
   redis_db:

docker/scripts/wait_for_db.sh

@@ -28,7 +28,7 @@ wait_for_db() {
       exit 3
     fi
 
-    nc -z "$POSTGRES_DB_HOST" "$POSTGRES_DB_PORT" > /dev/null 2>&1
+    nc -z "$POSTGRES_HOST" "$POSTGRES_PORT" > /dev/null 2>&1
 
     result=$?
     if [ $result -eq 0 ] ; then
@@ -51,8 +51,8 @@ while [ $# -gt 0 ]
 do
   case "$1" in
     *:* )
-    POSTGRES_DB_HOST=$(printf "%s\n" "$1"| cut -d : -f 1)
-    POSTGRES_DB_PORT=$(printf "%s\n" "$1"| cut -d : -f 2)
+    POSTGRES_HOST=$(printf "%s\n" "$1"| cut -d : -f 1)
+    POSTGRES_PORT=$(printf "%s\n" "$1"| cut -d : -f 2)
     shift 1
     ;;
     -q | --quiet)
@@ -82,10 +82,18 @@ do
   esac
 done
 
-POSTGRES_DB_HOST=${POSTGRES_DB_HOST:-localhost}
-POSTGRES_DB_PORT=${POSTGRES_DB_PORT:-5432}
-if [ "$POSTGRES_DB_HOST" = "" ] || [ "$POSTGRES_DB_PORT" = "" ]; then
-  echoerr "Error: you need to provide a host and port to test. Got: $POSTGRES_DB_HOST:$POSTGRES_DB_PORT"
+# backwards compatibility
+if [ "$POSTGRES_DB_HOST" != "" ]; then
+  POSTGRES_HOST=$POSTGRES_DB_HOST
+fi
+if [ "$POSTGRES_DB_PORT" != "" ]; then
+  POSTGRES_PORT=$POSTGRES_DB_PORT
+fi
+
+POSTGRES_HOST=${POSTGRES_HOST:-localhost}
+POSTGRES_PORT=${POSTGRES_PORT:-5432}
+if [ "$POSTGRES_HOST" = "" ] || [ "$POSTGRES_PORT" = "" ]; then
+  echoerr "Error: you need to provide a host and port to test. Got: $POSTGRES_HOST:$POSTGRES_PORT"
   usage 2
 fi
 

docs/config.md

@@ -1,45 +1,9 @@
 # Configuring OpenTogetherTube
 
-OTT is configured using `*.env` files located in the `env/` directory. The config that is read depends on the `NODE_ENV` environment variable. It's contents are series of key/value pairs, in the format `KEY=VALUE`.
+OTT is configured using `*.toml` files located in the `env/` directory.
 
-## All available options
+First, a base configuration is loaded from `env/base.toml`. Then, based on the value of `env` (which can also be set using the `NODE_ENV` environment variable), a configuration file is loaded from `env/<env>.toml`. For example, if `env` is set to `production`, then `env/production.toml` will be loaded.
 
-*Options marked as required are required for deployment*
+## All available options
 
-| Name | Description | Default | Expected Values |
-|-|-|-|-|
-| `OTT_HOSTNAME` | **REQUIRED** Specify the domain or IP address (with port, if necessary) of the host | None | `localhost:8080`, `opentogethertube.com` |
-| `PORT` | Specify the port the server listens on. | 8080 | 1-65535 |
-| `LOG_FILE` | Path to the log file | `./logs/ott.log` | Any valid, writable path to a file.
-| `LOG_LEVEL` | Only log messages with this log level or higher | `info` | `silly`, `debug`, `info`, `warn`, `error`
-| `OPENTOGETHERTUBE_API_KEY` | **REQUIRED** API key used to perform administrative actions via the OTT web API. | None | Any alpha-numeric string >= 40 characters
-| `SESSION_SECRET` | **REQUIRED** A secret used to make session cookies safer. | None | Any alpha-numeric string >= 80 characters
-| `YOUTUBE_API_KEY` | **REQUIRED** Youtube api key from [Google Cloud](https://console.cloud.google.com) | None | A Youtube API key
-| `GOOGLE_DRIVE_API_KEY` | **REQUIRED if you want google drive** Google drive api key from [Google Cloud](https://console.cloud.google.com). *Google drive really doesn't like to stream video, YMMV.* | None | A Google Drive API key
-| `DISCORD_CLIENT_ID` | Discord oauth client ID | None
-| `DISCORD_CLIENT_SECRET` | Discord oauth client secret | None
-| `GOOGLE_CLIENT_ID` | Currently unused
-| `GOOGLE_CLIENT_SECRET` | Currently unused
-| `REDIS_URL` | URI to your redis server. Overrides `REDIS_HOST`, and `REDIS_PORT`. If not supplied, it will try to connect to the redis server on the local machine. | None |
-| `REDIS_TLS_URL` | Works like, but prioritized over `REDIS_URL` | None |
-| `REDIS_HOST` | Hostname or IP of the redis server. If not supplied, it will try to connect to the redis server on the local machine. | `undefined` | Hostname or IP
-| `REDIS_PORT` | Port of the redis server. If not supplied, it will try to connect to the redis server on the default port. | `undefined` | 1-65535
-| `REDIS_DB` | Specify the redis DB that OTT should use. | `undefined` |
-| `REDIS_PASSWORD` | Password for the redis server. If not supplied, it will not be used. | `undefined` | string
-| `DB_MODE` | *Only if `NODE_ENV` is `production`* Force the server to use a database dialect. If not supplied, it will automatically use postgres if `DATABASE_URL`, `POSTGRES_DB_HOST`, `POSTGRES_DB_NAME`, `POSTGRES_DB_USERNAME`, `POSTGRES_DB_PASSWORD` are present. Otherwise, it will use SQLite. | None | `sqlite`, `postgres`
-| `DATABASE_URL` | Connection URI to your postgres database. |
-| `POSTGRES_DB_HOST` | Hostname or IP of the postgres server. | `127.0.0.1` | Hostname or IP
-| `POSTGRES_DB_NAME` | Postgres database name | `db_opentogethertube_prod` | string
-| `POSTGRES_DB_USERNAME` | Postgres username | `ott` | string
-| `POSTGRES_DB_PASSWORD` | Postgres password | None | string
-| `ADD_PREVIEW_SEARCH_MIN_LENGTH` | Minimum length of an add preview search query. Add preview queries shorter than this number are rejected. Does not affect URL add previews. | `3` | Integer >= 0
-| `ENABLE_SEARCH` | Enable searching for videos when no link is detected. | `true` | boolean
-| `SEARCH_PROVIDER` | Service adapter to use to provide video search results. | `youtube` | `youtube`
-| `ADD_PREVIEW_PLAYLIST_RESULTS_COUNT` | Limit the number of videos that appear in the results when a playlist is used. | `40` | Integer >= 0
-| `ADD_PREVIEW_SEARCH_RESULTS_COUNT` | Limit the number of videos that appear in the results when searching for a video. | `10` | Integer >= 0
-| `OTT_SHORT_URL_HOSTNAME` | The domain to use in the copyable "Share Invite" URL. This environment var must be present during building the client, otherwise it will not work. | undefined | `string` |
-| `TRUST_PROXY` | The number of reverse proxy layers to trust. | `1` | `number` |
-| `FFPROBE_PATH` | The path to the command `ffprobe`. You probably don't need to set this unless you are having problems with the ffprobe that is install automatically. | (see [@ffprobe-installer/ffprobe](https://github.com/SavageCore/node-ffprobe-installer)) | `string` |
-| `DIRECT_PREVIEW_MAX_BYTES` | The number of bytes to download when trying to preview direct playback videos. | Infinity | Integer >= 0 |
-| `RATE_LIMIT_KEY_PREFIX` | The prefix to use for rate limit keys, which are stored in redis. | `rateLimit` | `string` |
-| `ENABLE_RATE_LIMIT` | Enable rate limiting. | `true` | `boolean` |
+The config schema is defined in [ott-config.ts](../server/ott-config.ts). Most config options are also overridable using environment variables, which are also included in the schema.

env/.gitignore

@@ -0,0 +1,7 @@
+base.toml
+development.toml
+test.toml
+production.toml
+staging.toml
+
+*.env

env/docker.base.toml

@@ -0,0 +1,4 @@
+docker = true
+
+[db]
+mode = "postgres"

env/example.toml

@@ -0,0 +1,9 @@
+hostname="localhost:8080"
+
+log = { level="info" }
+
+api_key="" # generate your own >= 40 characters
+session_secret="" # generate your own >= 80 characters
+
+[info_extractor.youtube]
+api_key="" # get your own key from https://console.developers.google.com/apis/credentials

env/heroku.base.toml

@@ -0,0 +1,4 @@
+heroku = true
+
+[db]
+mode = "postgres"

server/.sequelizerc

@@ -0,0 +1,3 @@
+module.exports = {
+	config: "config/config.mjs",
+};