redis error

[ekokir]

Docker container outlinewiki/outline crashes periodically, system has down in 502 error.
There is error in those logs:

/opt/outline/node_modules/redis-parser/lib/parser.js:179
    return new ReplyError(string)
           ^

ReplyError: READONLY You can't write against a read only replica.
    at parseError (/opt/outline/node_modules/redis-parser/lib/parser.js:179:12)
    at parseType (/opt/outline/node_modules/redis-parser/lib/parser.js:302:14) {
  command: { name: 'del', args: [ 'UPDATES_KEY' ] }
}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

There is my docker-compose.yml :

version: "3"
services:

  outline:
    image: outlinewiki/outline
    env_file: ./docker.env
    ports:
      - "3000:3000"
    depends_on:
      - postgres
      - redis
      - storage

  redis:
    image: redis
    env_file: ./docker.env
    ports:
      - "6379:6379"
    volumes:
      - ./redis.conf:/redis.conf
    command: ["redis-server", "/redis.conf"]
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 30s
      retries: 3

  postgres:
    image: postgres
    env_file: ./docker.env
    ports:
      - "5432:5432"
    volumes:
      - database-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD", "pg_isready -U user"]
      interval: 30s
      timeout: 20s
      retries: 3
    environment:
      POSTGRES_USER: user
      POSTGRES_PASSWORD: pass
      POSTGRES_DB: outline

  storage:
    image: minio/minio
    env_file: ./docker.env
    ports:
      - "9000:9000"
    entrypoint: sh
    command: -c 'minio server'
    deploy:
      restart_policy:
        condition: on-failure
    volumes:
      - storage-data:/data
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  https-portal:
    image: steveltn/https-portal
    env_file: ./docker.env
    ports:
      - '80:80'
      - '443:443'
    links:
      - outline
      - storage
    restart: always
    volumes:
      - https-portal-data:/var/lib/https-portal
    healthcheck:
      test: ["CMD", "service", "nginx", "status"]
      interval: 30s
      timeout: 20s
      retries: 3

volumes:
  https-portal-data:
  storage-data:
  database-data:

There is my docker.env :

# –––––––––––––––– REQUIRED ––––––––––––––––

# Generate a hex-encoded 32-byte random key. You should use `openssl rand -hex 32`
# in your terminal to generate a random value.
SECRET_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# Generate a unique random key. The format is not important but you could still use
# `openssl rand -hex 32` in your terminal to produce this.
UTILS_SECRET=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# For production point these at your databases, in development the default
# should work out of the box.
DATABASE_URL=postgres://user:pass@postgres:5432/outline
DATABASE_URL_TEST=postgres://user:pass@postgres:5432/outline-test
DATABASE_CONNECTION_POOL_MIN=
DATABASE_CONNECTION_POOL_MAX=
# Uncomment this to disable SSL for connecting to Postgres
PGSSLMODE=disable

# For redis you can either specify an ioredis compatible url like this
REDIS_URL=redis://redis:6379
# or alternatively, if you would like to provide addtional connection options,
# use a base64 encoded JSON connection option object. Refer to the ioredis documentation
# for a list of available options.
# Example: Use Redis Sentinel for high availability
# {"sentinels":[{"host":"sentinel-0","port":26379},{"host":"sentinel-1","port":26379}],"name":"mymaster"}
# REDIS_URL=ioredis://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# URL should point to the fully qualified, publicly accessible URL. If using a
# proxy the port in URL and PORT may be different.
URL=https://xxxxxxxxxxxxx.com
PORT=3000

#https-portal config
DOMAINS='xxxxxxxx.com -> http://outline:3000'
STAGE=production
WEBSOCKET=true
#FORCE_RENEW=true

# See [documentation](docs/SERVICES.md) on running a separate collaboration
# server, for normal operation this does not need to be set.
#COLLABORATION_URL=

# To support uploading of images for avatars and document attachments an
# s3-compatible storage must be provided. AWS S3 is recommended for redundency
# however if you want to keep all file storage local an alternative such as
# minio (https://github.com/minio/minio) can be used.

# A more detailed guide on setting up S3 is available here:
# => https://wiki.generaloutline.com/share/125de1cc-9ff6-424b-8415-0d58c809a40f
#
#AWS_ACCESS_KEY_ID=get_a_key_from_aws
#AWS_SECRET_ACCESS_KEY=get_the_secret_of_above_key
#AWS_REGION=xx-xxxx-x
#AWS_S3_ACCELERATE_URL=
#AWS_S3_UPLOAD_BUCKET_URL=http://s3:4569
#AWS_S3_UPLOAD_BUCKET_NAME=bucket_name_here
#AWS_S3_UPLOAD_MAX_SIZE=26214400
#AWS_S3_FORCE_PATH_STYLE=true
#AWS_S3_ACL=private


# –––––––––––––– AUTHENTICATION ––––––––––––––

# Third party signin credentials, at least ONE OF EITHER Google, Slack,
# or Microsoft is required for a working installation or you'll have no sign-in
# options.

# To configure Slack auth, you'll need to create an Application at
# => https://api.slack.com/apps
#
# When configuring the Client ID, add a redirect URL under "OAuth & Permissions":
# https://<URL>/auth/slack.callback
#SLACK_CLIENT_ID=
#SLACK_CLIENT_SECRET=

# To configure Google auth, you'll need to create an OAuth Client ID at
# => https://console.cloud.google.com/apis/credentials
#
# When configuring the Client ID, add an Authorized redirect URI:
# https://<URL>/auth/google.callback
GOOGLE_CLIENT_ID=xxxxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com
GOOGLE_CLIENT_SECRET=xxxxxxxxxxxxxxxxxxxxxxxxx

# To configure Microsoft/Azure auth, you'll need to create an OAuth Client. See
# the guide for details on setting up your Azure App:
# => https://wiki.generaloutline.com/share/dfa77e56-d4d2-4b51-8ff8-84ea6608faa4
#AZURE_CLIENT_ID=
#AZURE_CLIENT_SECRET=
#AZURE_RESOURCE_APP_ID=

# To configure generic OIDC auth, you'll need some kind of identity provider.
# See documentation for whichever IdP you use to acquire the following info:
# Redirect URI is https://<URL>/auth/oidc.callback
#OIDC_CLIENT_ID=
#OIDC_CLIENT_SECRET=
#OIDC_AUTH_URI=
#OIDC_TOKEN_URI=
#OIDC_USERINFO_URI=

# Specify which claims to derive user information from
# Supports any valid JSON path with the JWT payload
#OIDC_USERNAME_CLAIM=preferred_username

# Display name for OIDC authentication
#OIDC_DISPLAY_NAME=OpenID

# Space separated auth scopes.
#OIDC_SCOPES=openid profile email


# –––––––––––––––– OPTIONAL ––––––––––––––––

# Base64 encoded private key and certificate for HTTPS termination. This is only
# required if you do not use an external reverse proxy. See documentation:
# https://wiki.generaloutline.com/share/1c922644-40d8-41fe-98f9-df2b67239d45
#SSL_KEY=
#SSL_CERT=

# If using a Cloudfront/Cloudflare distribution or similar it can be set below.
# This will cause paths to javascript, stylesheets, and images to be updated to
# the hostname defined in CDN_URL. In your CDN configuration the origin server
# should be set to the same as URL.
#CDN_URL=

# Auto-redirect to https in production. The default is true but you may set to
# false if you can be sure that SSL is terminated at an external loadbalancer.
FORCE_HTTPS=true

# Have the installation check for updates by sending anonymized statistics to
# the maintainers
ENABLE_UPDATES=true

# How many processes should be spawned. As a reasonable rule divide your servers
# available memory by 512 for a rough estimate
WEB_CONCURRENCY=1

# Override the maxium size of document imports, could be required if you have
# especially large Word documents with embedded imagery
MAXIMUM_IMPORT_SIZE=5120000

# You can remove this line if your reverse proxy already logs incoming http
# requests and this ends up being duplicative
DEBUG=http

# For a complete Slack integration with search and posting to channels the
# following configs are also needed, some more details
# => https://wiki.generaloutline.com/share/be25efd1-b3ef-4450-b8e5-c4a4fc11e02a
#
#SLACK_VERIFICATION_TOKEN=your_token
#SLACK_APP_ID=A0XXXXXXX
#SLACK_MESSAGE_ACTIONS=true

# Optionally enable google analytics to track pageviews in the knowledge base
#GOOGLE_ANALYTICS_ID=

# Optionally enable Sentry (sentry.io) to track errors and performance
#SENTRY_DSN=

# To support sending outgoing transactional emails such as "document updated" or
# "you've been invited" you'll need to provide authentication for an SMTP server
#SMTP_HOST=
#SMTP_PORT=
#SMTP_USERNAME=
#SMTP_PASSWORD=
#SMTP_FROM_EMAIL=
#SMTP_REPLY_EMAIL=
#SMTP_TLS_CIPHERS=
#SMTP_SECURE=true

# Custom logo that displays on the authentication screen, scaled to height: 60px
# TEAM_LOGO=https://example.com/images/logo.png

# The default interface language. See translate.getoutline.com for a list of
# available language codes and their rough percentage translated.
DEFAULT_LANGUAGE=en_US

But I don't see the file redis.conf, just an empty folder.
why does the system crash and how fix it?

[almereyda]

This sounds like your hard drive is full, and the redis container has no space left on the device for intermediarily writing to the volatile database file during container run time.

[ekokir]

Thanks for the version, but the disk space is fine. Where would you advise to check, in addition to docker logs?

[almereyda]

It appears you have added an additional redis.conf, which is currently unavailable to us for further debugging.

Did you maybe intend to activate Redis cluster mode at some point? https://redis.io/docs/manual/replication/

Else I can't see why your Redis "node" would turn itself into READONLY mode.

[ekokir]

I tryred set custom redis.conf, but the same problem occurs exactly as described here https://stackoverflow.com/questions/72906936/why-does-redis-forcibly-demote-master-to-slave

[dapperdrop]

I encountered the same issue in a recent docker container deployment. My cloud server would randomly spike to 100% CPU which caused a 502, requiring a reboot to rectify.

Some research revealed that the docker-compose port key is allowing redis (and postgresql for that matter) to be publicly accessible on the specified ports. Note that some users may not be affected by this if they setup a firewall layer on their hosting/server, but I didn't do this initially.

With redis being publicly accessible, I believe it was being sniffed out and attacked by scripts/port scanners (see: https://medium.com/threat-intel/cryptojacking-coin-mining-cybercrime-234895bec6e1), which was causing the CPU spikes and subsequent 502s.

To resolve this issue, replace the port key with expose (expose makes the port only accessible to link services). e.g.

#ports:
#      - "6379:6379"
expose:
      - 6379

Make sure to apply changes to docker and restart: docker-compose down --remove-orphans && docker-compose up -d

Also, if you can, setup a firewall via your server hosting platform. Good luck and I hope this resolves your issue!

[thomasgriffin]

Thanks so much for sharing this! I updated my tutorial on installing Outline to make sure that both the Postgres and Redis configurations use expose rather than ports to prevent spam attacks.

[heyvaldemar]

hey guys, I wrote a detailed guide after I was able to configure Outline with MinIO and Keycloak.

Hopefully, it will help others.