Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ERROR when starting mattermost , postgres and nginx read_only file system #39

Closed
craph opened this issue Sep 20, 2021 · 6 comments
Closed

Comments

@craph
Copy link
Contributor

craph commented Sep 20, 2021

Hello,

I just followed all the steps to start with a fresh installation of MM with the version 5.38.2 and when I ran docker-compose up -d I have errors with the database and nginx.

Here are the errors:

postgres_mattermost | 2021-09-20 10:09:21.430 UTC [1] LOG:  starting PostgreSQL 13.4 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.3.1_git20210424) 10.3.1 20210424, 64-bit
postgres_mattermost | 2021-09-20 10:09:21.430 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres_mattermost | 2021-09-20 10:09:21.430 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres_mattermost | 2021-09-20 10:09:21.430 UTC [1] FATAL:  could not create lock file "/var/run/postgresql/.s.PGSQL.5432.lock": Read-only file system
postgres_mattermost | 2021-09-20 10:09:21.432 UTC [1] LOG:  database system is shut down
mattermost    | {"level":"info","ts":1632132609.2597942,"caller":"sqlstore/store.go:308","msg":"Pinging SQL","database":"master"}
mattermost    | {"level":"error","ts":1632132609.2736187,"caller":"sqlstore/store.go:320","msg":"Failed to ping DB","error":"dial tcp: lookup postgres on 127.0.0.11:53: server misbehaving","retrying in seconds":10}
mattermost    | {"level":"info","ts":1632132619.2740848,"caller":"sqlstore/store.go:308","msg":"Pinging SQL","database":"master"}
mattermost    | {"level":"error","ts":1632132619.280617,"caller":"sqlstore/store.go:320","msg":"Failed to ping DB","error":"dial tcp: lookup postgres on 127.0.0.11:53: server misbehaving","retrying in seconds":10}
postgres_mattermost exited with code 1

and for nginx :

/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [warn] "ssl_stapling" ignored, issuer certificate not found for certificate "/cert.pem"
nginx: [emerg] open() "/var/run/nginx.pid" failed (30: Read-only file system)

I have existing certificate and key from a previous installation of MM 5.31.0 so I moved the file to the new directory.

Here is the .env file :

# Domain of service
DOMAIN=mymattermost

# Container settings
## Timezone inside the containers. The value needs to be in the form 'Europe/Berlin'.
## A list of these tz database names can be looked up at Wikipedia
## https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
TZ=UTC
RESTART_POLICY=unless-stopped

# Postgres settings
## Documentation for this image and available settings can be found on hub.docker.com
## https://hub.docker.com/_/postgres
## Please keep in mind this will create a superuser and it's recommended to use a less privileged
## user to connect to the database.
## A guide on how to change the database user to a nonsuperuser can be found in docs/creation-of-nonsuperuser.md
POSTGRES_IMAGE_TAG=13-alpine
POSTGRES_DATA_PATH=./volumes/db/var/lib/postgresql/data

POSTGRES_USER=mmuser
POSTGRES_PASSWORD=mmuser_password
POSTGRES_DB=mattermost

# Nginx
## The nginx container will use a configuration found at the NGINX_MATTERMOST_CONFIG. The config aims
## to be secure and uses a catch-all server vhost which will work out-of-the-box. For additional settings
## or changes ones can edit it or provide another config. Important note: inside the container, nginx sources
## every config file inside */etc/nginx/conf.d* ending with a *.conf* file extension.

## Inside the container the uid and gid is 101. The folder owner can be set with
## `sudo chown -R 101:101 ./nginx` if needed.
NGINX_IMAGE_TAG=alpine

## The folder containing server blocks and any additional config to nginx.conf
NGINX_CONFIG_PATH=./nginx/conf.d
NGINX_DHPARAMS_FILE=./nginx/dhparams4096.pem

CERT_PATH=./volumes/web/cert/cert.pem
KEY_PATH=./volumes/web/cert/key-no-password.pem
GITLAB_PKI_CHAIN_PATH=/root/certpki/my_pki_chain.pem
#CERT_PATH=./certs/etc/letsencrypt/live/${DOMAIN}/fullchain.pem
#KEY_PATH=./certs/etc/letsencrypt/live/${DOMAIN}/privkey.pem

## Exposed ports to the host. Inside the container 80 and 443 will be used
HTTPS_PORT=4434
HTTP_PORT=8008

# Mattermost settings
## Inside the container the uid and gid is 2000. The folder owner can be set with
## `sudo chown -R 2000:2000 ./volumes/app/mattermost`.
MATTERMOST_CONFIG_PATH=./volumes/app/mattermost/config
MATTERMOST_DATA_PATH=./volumes/app/mattermost/data
MATTERMOST_LOGS_PATH=./volumes/app/mattermost/logs
MATTERMOST_PLUGINS_PATH=./volumes/app/mattermost/plugins
MATTERMOST_CLIENT_PLUGINS_PATH=./volumes/app/mattermost/client/plugins

## This will be 'mattermost-enterprise-edition' or 'mattermost-team-edition' based on the version of Mattermost you're installing.
MATTERMOST_IMAGE=mattermost-team-edition
MATTERMOST_IMAGE_TAG=5.38.2

## The app port is only relevant for using Mattermost without the nginx container as reverse proxy. This is not meant
## to be used with the internal HTTP server exposed but rather in case one wants to host several services on one host
## or for using it behind another existing reverse proxy.
APP_PORT=8065

## Configuration settings for Mattermost. Documentation on the variables and the settings itself can be found at
## https://docs.mattermost.com/administration/config-settings.html
## Keep in mind that variables set here will take precedence over the same setting in config.json. This includes
## the system console as well and settings set with env variables will be greyed out.

## Below one can find necessary settings to spin up the Mattermost container
MM_SQLSETTINGS_DRIVERNAME=postgres
MM_SQLSETTINGS_DATASOURCE=postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}?sslmode=disable&connect_timeout=10

## Example settings (any additional setting added here also needs to be introduced in the docker-compose.yml)
MM_SERVICESETTINGS_SITEURL=https://${DOMAIN}:4434
@craph craph changed the title ERROR when starting mattermost , postgres and nginx ERROR when starting mattermost , postgres and nginx read_only file system Sep 20, 2021
@craph
Copy link
Contributor Author

craph commented Sep 20, 2021

After more investigations, it seems the errors are due to the tmpfs entries in docker-compose files.

Postgres and nginx are both based on alpine. In alpine, /var/run is a symlink on /run so the entry in compose files should be like that for Postgres & Nginx :

tmpfs:
  - /run # instead of /var/run

If you want, I can do a PR for that ?

What do you think @mrckndt , @jasonblais

@craph
Copy link
Contributor Author

craph commented Oct 20, 2021

Hello @mrckndt , @metanerd , Please could you have a look to my PR ?
Thank you very much.
Best regards,

@craph
Copy link
Contributor Author

craph commented Oct 26, 2021

Hi @jasonblais is it possible to have some updates on this issue and PR ?
Thank you very much.
Best regards,

@craph
Copy link
Contributor Author

craph commented Dec 18, 2021

@mrckndt, @cpanato, @hanzei, After more investigations, the errors are due to the tmpfs entries in docker-compose files.

Postgres and nginx are both based on alpine. In alpine, /var/run is a symlink on /run so the entry in compose files should be like that for Postgres & Nginx :

tmpfs:
  - /run # instead of /var/run

Here are the steps to reproduce the behavior :

# lsb_release -d
Description:    CentOS Linux release 7.7.1908 (Core)
# docker --version
Docker version 19.03.6, build 369ce74a3c
# docker-compose version
docker-compose version 1.26.0, build unknown
docker-py version: 4.2.1
CPython version: 3.6.8
OpenSSL version: OpenSSL 1.0.2k-fips  26 Jan 2017
  1. git clone https://github.com/mattermost/docker.git mattermost-docker-debug
  2. cp env.example .env
    Here is the sample of my .env file:
# cat .env
# Domain of service
DOMAIN=mymmserver.com

# Container settings
## Timezone inside the containers. The value needs to be in the form 'Europe/Berlin'.
## A list of these tz database names can be looked up at Wikipedia
## https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
TZ=UTC
RESTART_POLICY=unless-stopped

# Postgres settings
## Documentation for this image and available settings can be found on hub.docker.com
## https://hub.docker.com/_/postgres
## Please keep in mind this will create a superuser and it's recommended to use a less privileged
## user to connect to the database.
## A guide on how to change the database user to a nonsuperuser can be found in docs/creation-of-nonsuperuser.md
POSTGRES_IMAGE_TAG=13-alpine
POSTGRES_DATA_PATH=./volumes/db/var/lib/postgresql/data

POSTGRES_USER=mmuser
POSTGRES_PASSWORD=mmuser_password
POSTGRES_DB=mattermost

# Nginx
## The nginx container will use a configuration found at the NGINX_MATTERMOST_CONFIG. The config aims
## to be secure and uses a catch-all server vhost which will work out-of-the-box. For additional settings
## or changes ones can edit it or provide another config. Important note: inside the container, nginx sources
## every config file inside */etc/nginx/conf.d* ending with a *.conf* file extension.

## Inside the container the uid and gid is 101. The folder owner can be set with
## `sudo chown -R 101:101 ./nginx` if needed.
NGINX_IMAGE_TAG=alpine

## The folder containing server blocks and any additional config to nginx.conf
NGINX_CONFIG_PATH=./nginx/conf.d
NGINX_DHPARAMS_FILE=./nginx/dhparams4096.pem

CERT_PATH=./volumes/web/cert/cert.pem
KEY_PATH=./volumes/web/cert/key-no-password.pem
#GITLAB_PKI_CHAIN_PATH=<path_to_your_gitlab_pki>/pki_chain.pem
#CERT_PATH=./certs/etc/letsencrypt/live/${DOMAIN}/fullchain.pem
#KEY_PATH=./certs/etc/letsencrypt/live/${DOMAIN}/privkey.pem

## Exposed ports to the host. Inside the container 80 and 443 will be used
HTTPS_PORT=4434
HTTP_PORT=8008

# Mattermost settings
## Inside the container the uid and gid is 2000. The folder owner can be set with
## `sudo chown -R 2000:2000 ./volumes/app/mattermost`.
MATTERMOST_CONFIG_PATH=./volumes/app/mattermost/config
MATTERMOST_DATA_PATH=./volumes/app/mattermost/data
MATTERMOST_LOGS_PATH=./volumes/app/mattermost/logs
MATTERMOST_PLUGINS_PATH=./volumes/app/mattermost/plugins
MATTERMOST_CLIENT_PLUGINS_PATH=./volumes/app/mattermost/client/plugins

## This will be 'mattermost-enterprise-edition' or 'mattermost-team-edition' based on the version of Mattermost you're installing.
MATTERMOST_IMAGE=mattermost-team-edition
MATTERMOST_IMAGE_TAG=6.1

## Make Mattermost container readonly. This interferes with the regeneration of root.html inside the container. Only use
## it if you know what you're doing.
## See https://github.com/mattermost/docker/issues/18
MATTERMOST_CONTAINER_READONLY=false

## The app port is only relevant for using Mattermost without the nginx container as reverse proxy. This is not meant
## to be used with the internal HTTP server exposed but rather in case one wants to host several services on one host
## or for using it behind another existing reverse proxy.
APP_PORT=8065

## Configuration settings for Mattermost. Documentation on the variables and the settings itself can be found at
## https://docs.mattermost.com/administration/config-settings.html
## Keep in mind that variables set here will take precedence over the same setting in config.json. This includes
## the system console as well and settings set with env variables will be greyed out.

## Below one can find necessary settings to spin up the Mattermost container
MM_SQLSETTINGS_DRIVERNAME=postgres
MM_SQLSETTINGS_DATASOURCE=postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}?sslmode=disable&connect_timeout=10

## Example settings (any additional setting added here also needs to be introduced in the docker-compose.yml)
MM_SERVICESETTINGS_SITEURL=https://${DOMAIN}:4434
  1. mkdir -p ./volumes/app/mattermost/{config,data,logs,plugins,client/plugins,bleve-indexes}
  2. chown -R 2000:2000 ./volumes/app/mattermost
  3. mkdir -p ./volumes/web/cert
  4. cp PATH-TO-CERT.PEM ./volumes/web/cert/cert.pem; cp PATH-TO-KEY.PEM ./volumes/web/cert/key-no-password.pem
  5. docker-compose -f docker-compose.nginx.yml -f docker-compose.yml up -d && docker-compose -f docker-compose.nginx.yml -f docker-compose.yml logs -f
  6. Here you can see the errors in the logs
postgres_mattermost | 2021-12-18 21:10:32.076 UTC [1] FATAL:  could not create lock file "/var/run/postgresql/.s.PGSQL.5432.lock": Read-only file system
postgres_mattermost | 2021-12-18 21:10:32.078 UTC [1] LOG:  database system is shut down
postgres_mattermost exited with code 1
...
nginx_mattermost | nginx: [emerg] open() "/var/run/nginx.pid" failed (30: Read-only file system)
nginx_mattermost exited with code 1

Here a screenshot
image
and you can see that the container are not running
image

  1. docker-compose -f docker-compose.nginx.yml -f docker-compose.yml down
  2. rm -rf volumes/db needed because the database was not created due to the read_only filesystem
  3. Change /var/run -> /run in docker-compose.nginx.yml and /var/run/postgresql -> /run/postgresql in docker-compose.yml
    image
  4. docker-compose -f docker-compose.nginx.yml -f docker-compose.yml up -d && docker-compose -f docker-compose.nginx.yml -f docker-compose.yml logs -f
  5. everything goes well and the logs are OK and all the container are running
    image

@craph
Copy link
Contributor Author

craph commented Feb 1, 2022

Hello @mrckndt,

Do you have any news about my PR ?
You can find in my previous comment and in the PR the full explanation about the issue and the fix :)

Thank you very much.
Best regards,

@spirosoik
Copy link
Member

Closing as it's not valid anymore. Check here #53 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants