Sites volume permission error

[timemaster5]

Description of the issue

I follow this guide: https://github.com/frappe/frappe_docker/blob/main/docs/single-server-example.md
And this step is failing every time, even though I have all the environment variables correctly set :

docker compose --project-name erpnext-one exec backend \
  bench new-site one.example.com --mariadb-root-password changeit --install-app erpnext --admin-password changeit

The problem is with the mariadb, bench is trying to connect to 127.0.0.1 instead of to mariadb-database container

Context information (for bug reports)

Ubuntu 21.10, Docker 20.10.12 with compose 2.5.0

Steps to reproduce the issue

1. follow the steps in https://github.com/frappe/frappe_docker/blob/main/docs/single-server-example.md

Observed result

Mariadb connection error

Expected result

Correctly created site

Stacktrace / full error message if available

pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on '127.0.0.1' ([Errno 111] Connection refused)")

[revant]

wait for few seconds for Mariadb container to start with configurations and env variables.

wait for the configuration container to stop successfully.

[timemaster5]

Thank you for your response. I believe all mentioned is fine.

MySQL container seems fine

⠿ Container mariadb-database Started

2022-05-10 07:40:08+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.6.7+maria~focal started.
2022-05-10 07:40:08+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2022-05-10 07:40:08+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.6.7+maria~focal started.
2022-05-10 07:40:08+00:00 [Note] [Entrypoint]: MariaDB upgrade not required
2022-05-10  7:40:08 0 [Note] mariadbd (server 10.6.7-MariaDB-1:10.6.7+maria~focal) starting as process 1 ...
2022-05-10  7:40:08 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-05-10  7:40:08 0 [Note] InnoDB: Number of pools: 1
2022-05-10  7:40:08 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-05-10  7:40:08 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2022-05-10  7:40:08 0 [Note] InnoDB: Using Linux native AIO
2022-05-10  7:40:08 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2022-05-10  7:40:08 0 [Note] InnoDB: Completed initialization of buffer pool
2022-05-10  7:40:08 0 [Note] InnoDB: 128 rollback segments are active.
2022-05-10  7:40:08 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2022-05-10  7:40:08 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2022-05-10  7:40:08 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2022-05-10  7:40:08 0 [Note] InnoDB: 10.6.7 started; log sequence number 42353; transaction id 14
2022-05-10  7:40:08 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2022-05-10  7:40:08 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-05-10  7:40:08 0 [Note] InnoDB: Buffer pool(s) load completed at 220510  7:40:08
2022-05-10  7:40:08 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
2022-05-10  7:40:08 0 [Note] Server socket created on IP: '0.0.0.0'.
2022-05-10  7:40:08 0 [Note] Server socket created on IP: '::'.
2022-05-10  7:40:08 0 [Note] mariadbd: ready for connections.
Version: '10.6.7-MariaDB-1:10.6.7+maria~focal'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution

Here is the rest:

[+] Running 10/10
 ⠿ Network erpnext-one                    Created
 ⠿ Container erpnext-one-redis-1          Started
 ⠿ Container erpnext-one-configurator-1   Exited 
 ⠿ Container erpnext-one-scheduler-1      Started
 ⠿ Container erpnext-one-queue-short-1    Started
 ⠿ Container erpnext-one-backend-1        Started
 ⠿ Container erpnext-one-queue-long-1     Started
 ⠿ Container erpnext-one-websocket-1      Started
 ⠿ Container erpnext-one-queue-default-1  Started
 ⠿ Container erpnext-one-frontend-1       Started

redis seems to be fine the same way as mariadb

1:C 10 May 2022 07:44:16.220 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 10 May 2022 07:44:16.220 # Redis version=6.2.7, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 10 May 2022 07:44:16.220 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
1:M 10 May 2022 07:44:16.222 * monotonic clock: POSIX clock_gettime
1:M 10 May 2022 07:44:16.222 # A key '__redis__compare_helper' was added to Lua globals which is not on the globals allow list nor listed on the deny list.
1:M 10 May 2022 07:44:16.222 * Running mode=standalone, port=6379.
1:M 10 May 2022 07:44:16.222 # Server initialized
1:M 10 May 2022 07:44:16.222 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 10 May 2022 07:44:16.223 * Loading RDB produced by version 6.2.7
1:M 10 May 2022 07:44:16.223 * RDB age 1850 seconds
1:M 10 May 2022 07:44:16.223 * RDB memory usage when created 0.77 Mb
1:M 10 May 2022 07:44:16.223 # Done loading RDB, keys loaded: 0, keys expired: 0.
1:M 10 May 2022 07:44:16.224 * DB loaded from disk: 0.001 seconds
1:M 10 May 2022 07:44:16.224 * Ready to accept connections

but other containers can't connect to it, this is the same for erpnext-one-queue-short-1 and erpnext-one-queue-long-1 and erpnext-one-queue-default-1:

Traceback (most recent call last):
  File "/usr/local/bin/patched_bench_helper.py", line 48, in <module>
    raise SystemExit(main())
  File "/usr/local/bin/patched_bench_helper.py", line 43, in main
    frappe.utils.bench_helper.main()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 20, in main
    click.Group(commands=commands)(prog_name="bench")
  File "/home/frappe/frappe-bench/env/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/frappe/frappe-bench/env/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/frappe/frappe-bench/env/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/frappe/frappe-bench/env/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/frappe/frappe-bench/env/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/frappe/frappe-bench/env/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/commands/scheduler.py", line 203, in start_worker
    start_worker(queue, quiet=quiet)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/background_jobs.py", line 176, in start_worker
    redis_connection = get_redis_conn()
  File "/home/frappe/frappe-bench/env/lib/python3.9/site-packages/tenacity/__init__.py", line 324, in wrapped_f
    return self(f, *args, **kw)
  File "/home/frappe/frappe-bench/env/lib/python3.9/site-packages/tenacity/__init__.py", line 404, in __call__
    do = self.iter(retry_state=retry_state)
  File "/home/frappe/frappe-bench/env/lib/python3.9/site-packages/tenacity/__init__.py", line 349, in iter
    return fut.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/home/frappe/frappe-bench/env/lib/python3.9/site-packages/tenacity/__init__.py", line 407, in __call__
    result = fn(*args, **kwargs)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/background_jobs.py", line 284, in get_redis_conn
    raise Exception("redis_queue missing in common_site_config.json")
Exception: redis_queue missing in common_site_config.json

here is a websocket container error log

listening on *: 3000
node:events:505
      throw er; // Unhandled 'error' event
      ^

Error: connect ECONNREFUSED 127.0.0.1:12311
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1195:16)
Emitted 'error' event on RedisClient instance at:
    at RedisClient.on_error (/home/frappe/frappe-bench/apps/frappe/node_modules/redis/index.js:342:14)
    at Socket.<anonymous> (/home/frappe/frappe-bench/apps/frappe/node_modules/redis/index.js:223:14)
    at Socket.emit (node:events:527:28)
    at emitErrorNT (node:internal/streams/destroy:164:8)
    at emitErrorCloseNT (node:internal/streams/destroy:129:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 12311
}

Node.js v17.9.0

and this one is also interesting erpnext-one-frontend-1:

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/frappe-entrypoint.sh
touch: /usr/share/nginx/html/sites/.build: Permission denied

[timemaster5]

Also, networking between containers seems to be fine as well, as I can connect to MariaDB from the backend container using mysql -h mariadb-database -p.. so I guess it is really some missing piece of configuration or non-working environment variables..

I have one last clue to try, I will post my results in a few mins

[timemaster5]

Hm, nothing. I've tried again from the beginning, which means I have re-created ~/gitops/erpnext-one.yaml from scratch and this way, the configurator container didn't even run correctly:

Traceback (most recent call last):
  File "/usr/local/bin/configure.py", line 56, in <module>
    raise SystemExit(main())
  File "/usr/local/bin/configure.py", line 44, in main
    update_config(
  File "/usr/local/bin/configure.py", line 12, in update_config
    with open(fname, "a") as f:
PermissionError: [Errno 13] Permission denied: 'common_site_config.json'

So I guess my original approach of changing paths in the .yaml file to some accessible directories on my system was right.

I've been changing /home/frappe/frappe-bench/sites to /var/erpnext/{subdirs}, and had created this empty dir before. Here is a snippet of the erpnext-one.yaml to show what paths I am referring to.

name: erpnext-one
services:
  backend:
    depends_on:
      configurator:
        condition: service_completed_successfully
    image: frappe/erpnext-worker:v13.28.0
    networks:
      mariadb-network: null
    volumes:
    - type: volume
      source: sites
      target: /home/frappe/frappe-bench/sites
      volume: {}
    - type: volume
      source: assets
      target: /home/frappe/frappe-bench/sites/assets
      read_only: true
      volume: {}
  configurator:
    command:
    - configure.py
    depends_on:
      redis:
        condition: service_started
    environment:
      DB_HOST: mariadb-database
      DB_PORT: "3306"
      REDIS_CACHE: redis:6379/0
      REDIS_QUEUE: redis:6379/1
      REDIS_SOCKETIO: redis:6379/2
      SOCKETIO_PORT: "9000"
    image: frappe/erpnext-worker:v13.28.0
    networks:
      mariadb-network: null
    volumes:
    - type: volume
      source: sites
      target: /home/frappe/frappe-bench/sites
      volume: {}

[revant]

changing paths won't solve the problem.

the permission to docker volumes need to be fixed.

the sites directory needs to be owned by uid:gid 1000:1000

manually chown the volumes from /var/lib/docker/volumes or use additional container with root privileges to chown the volume

[timemaster5]

Thank you for all the clues. I've been successful today with adding a user statement to some service configurations. Please check my PR #804

[revant]

follow discussion on PR

[revant]

https://discuss.erpnext.com/t/how-can-i-get-erpnext-running-under-docker-for-a-quick-look/90288

PWD setup working locally on my linux (non-root user, docker compose v2):

pwd-setup-locally.mp4

[revant]

Make sure only 1 docker daemon is installed and it has appropriate access. docker installed through snap may cause problem.

Check this thread https://discuss.erpnext.com/t/how-can-i-get-erpnext-running-under-docker-for-a-quick-look/90288/18

[revant]

I guess this happens when I try to setup a server as root user directly instead of using non-root user with docker permission.

[timemaster5]

I guess this happens when I try to setup a server as root user directly instead of using non-root user with docker permission.

This could be the cause, yes. I’ve used sudo for sure, but that could result in the same behaviour as using root account directly.

[sdfateh]

@timemaster5 can you access your database from backend container cli using:

mysql -u root -p -h <DB_CONTANER_NAME>

[revant]

https://docs.docker.com/engine/install/linux-postinstall/

I use docker as non root user added to docker group as mentioned in docker docs.

[revant]

I'll close this.

open new issue if you still face the problem.