Unifi Controller does not Produce Backups #512

Jeb-InnoVision · 2022-01-10T20:07:17Z

Operating system

Ubuntu Linux 20.04.3 LTS

UniFi Tag

6.2.23

Docker run

docker-compose up -d

Bug description

The Unifi Controller is not creating backups, autobackups, nor will it download/create a backup when requested within the controller itself.

Backup files are not being generated in the volumes of the OS.

Placing old backups (created before moving to Docker Unifi) in the volumes directory are not visible as well.

Compose File
docker-unifi-backup-issue_docker-compose.txt

Steps to reproduce

[ Setting up Autobackup ]
Log into Unifi Controller as Super User
Settings > Backup > [Configure settings for autobackup]
*No backups are created as time progresses (based on settings)

[ Downloading Backup File ]
Log into Unifi Controller as Super User
Settings > Backup
Select "Download Backup" button
*Button just spins (for hours) with no resolution
*Refreshing the page allows me to press the button again

Relevant log output

None. No errors or issues appear in the logs related to creating backups.

mountaindude · 2022-01-20T12:34:47Z

Fwiw, this bug is also occurring in 6.5.55 (which is latest as of writing this).

jacobalberty · 2022-01-20T16:23:01Z

@mountaindude can you provide a copy of your docker-compose or docker run command line?

This works on my setup so I want to look at narrowing down the common difference between my setup and you guys.

mountaindude · 2022-01-20T16:37:21Z

Of course!

Running Docker 20.10.12, build e91ed57 on Ubuntu 18.04.6 LTS

docker-compose is pretty much standard except I hardcoded the container names, plus a few extra labels (should not affect things).

version: '2.3'
services:
  mongo:
    image: mongo:3.6
    # container_name: ${COMPOSE_PROJECT_NAME}_mongo
    container_name: unifi_mongo
    networks:
      - unifi
    restart: always
    volumes:
      - db:/data/db
      - dbcfg:/data/configdb
  controller:
    # image: "jacobalberty/unifi:${TAG:-latest}"
    image: jacobalberty/unifi:latest
    # container_name: ${COMPOSE_PROJECT_NAME}_controller
    container_name: unifi_controller
    depends_on:
      - mongo
    init: true
    networks:
      - unifi
    restart: always
    volumes:
      - dir:/unifi
      - data:/unifi/data
      - log:/unifi/log
      - cert:/unifi/cert
      - init:/unifi/init.d
      - run:/var/run/unifi
      # Mount local folder for backups and autobackups
      - ./backup:/unifi/data/backup
    user: unifi
    sysctls:
      net.ipv4.ip_unprivileged_port_start: 0
    environment:
      DB_URI: mongodb://mongo/unifi
      STATDB_URI: mongodb://mongo/unifi_stat
      DB_NAME: unifi
      TZ: Europe/Stockholm
    ports:
      - "3478:3478/udp" # STUN
      - "6789:6789/tcp" # Speed test
      - "8080:8080/tcp" # Device/ controller comm.
      - "8443:8443/tcp" # Controller GUI/API as seen in a web browser
      - "8880:8880/tcp" # HTTP portal redirection
      - "8843:8843/tcp" # HTTPS portal redirection
      - "10001:10001/udp" # AP discovery
    labels:
      - "diun.enable=true"
      - "diun.watch_repo=true"
      - "diun.exclude_tags=.*arm32v7;.*arm64v8"

  logs:
    image: bash
    # container_name: ${COMPOSE_PROJECT_NAME}_logs
    container_name: unifi_logs
    depends_on:
      - controller
    command: bash -c 'tail -F /unifi/log/*.log'
    restart: always
    volumes:
      - log:/unifi/log

volumes:
  db:
  dbcfg:
  data:
  log:
  cert:
  init:
  dir:
  run:

networks:
  unifi:

I also notice some log errors relating to the scheduled backups.
Not sure if it's related.

[2022-01-20T14:00:01,290] <schedule-backup> ERROR system - zipDir error
java.io.FileNotFoundException: /usr/lib/unifi/data/backup/autobackup/autobackup_6.5.55_20220120_1300_1642683600011.unf (No such file or directory)
	at java.io.FileOutputStream.open0(Native Method) ~[?:1.8.0_312]
	at java.io.FileOutputStream.open(FileOutputStream.java:270) ~[?:1.8.0_312]
	at java.io.FileOutputStream.<init>(FileOutputStream.java:213) ~[?:1.8.0_312]
	at java.io.FileOutputStream.<init>(FileOutputStream.java:101) ~[?:1.8.0_312]
	at com.ubnt.ace.B.o00000(Unknown Source) [ace.jar:?]
	at com.ubnt.service.system.returnObject.o00000(Unknown Source) [ace.jar:?]
	at com.ubnt.service.system.backup.for.o00000(Unknown Source) [ace.jar:?]
	at com.ubnt.service.system.backup.for.o00000(Unknown Source) [ace.jar:?]
	at com.ubnt.service.system.backup.for.o00000(Unknown Source) [ace.jar:?]
	at com.ubnt.service.system.backup.oO0O.o00000(Unknown Source) [ace.jar:?]
	at com.ubnt.service.system.backup.oO0O.o00000(Unknown Source) [ace.jar:?]
	at com.ubnt.service.system.backup.oO0O$_OOo.super(Unknown Source) [ace.jar:?]
	at com.ubnt.service.schedule.L$_Oo.execute(Unknown Source) [ace.jar:?]
	at it.sauronsoftware.cron4j.TaskExecutor$Runner.run(Unknown Source) [cron4j-2.2.5.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
[2022-01-20T14:00:01,291] <schedule-backup> ERROR schedule - Failed to execute[backup]: /usr/lib/unifi/data/backup/autobackup/autobackup_6.5.55_20220120_1300_1642683600011.unf does not exist
java.lang.IllegalArgumentException: /usr/lib/unifi/data/backup/autobackup/autobackup_6.5.55_20220120_1300_1642683600011.unf does not exist
	at org.apache.commons.io.FileUtils.sizeOf(FileUtils.java:2541) ~[commons-io-2.6.jar:2.6]
	at com.ubnt.service.system.returnObject.o00000(Unknown Source) ~[ace.jar:?]
	at com.ubnt.service.system.backup.for.o00000(Unknown Source) ~[ace.jar:?]
	at com.ubnt.service.system.backup.for.o00000(Unknown Source) ~[ace.jar:?]
	at com.ubnt.service.system.backup.for.o00000(Unknown Source) ~[ace.jar:?]
	at com.ubnt.service.system.backup.oO0O.o00000(Unknown Source) ~[ace.jar:?]
	at com.ubnt.service.system.backup.oO0O.o00000(Unknown Source) ~[ace.jar:?]
	at com.ubnt.service.system.backup.oO0O$_OOo.super(Unknown Source) ~[ace.jar:?]
	at com.ubnt.service.schedule.L$_Oo.execute(Unknown Source) [ace.jar:?]
	at it.sauronsoftware.cron4j.TaskExecutor$Runner.run(Unknown Source) [cron4j-2.2.5.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]

mountaindude · 2022-01-20T16:45:58Z

@jacobalberty Do you want to track these issues in separate posts?
I.e. the inability to download backups and the auto-backup errors in the logs?

I checked and it looks like the auto backup errors in the log are related to the fact that /usr/lib/unifi/data/backup/autobackup/ does not exist.

unifi@64602e3bbf81:/unifi$ ls -la /usr/lib/unifi/data/backup/
total 8
drwxr-xr-x 2 root  root  4096 Jan 20 13:14 .
drwxr-xr-x 4 unifi unifi 4096 Jan 20 13:42 ..
unifi@64602e3bbf81:/unifi$

Digging a bit deeper, permissions on that backup directory looks funky.
Not sure what they should be though...unifi maybe?

unifi@64602e3bbf81:/usr/lib/unifi/data$ pwd
/usr/lib/unifi/data
unifi@64602e3bbf81:/usr/lib/unifi/data$ ls -la
total 52
drwxr-xr-x 4 unifi unifi  4096 Jan 20 13:42 .
drwxr-xr-x 6 unifi unifi  4096 Jan 20 13:14 ..
drwxr-xr-x 2 root  root   4096 Jan 20 13:14 backup
drwxr-xr-x 2 unifi unifi  4096 Jan 20 13:14 db
-rw-r--r-- 1 unifi unifi 18344 Jan 20 13:43 firmware.json
-rw-r--r-- 1 unifi unifi  2248 Jan 20 13:14 keystore
-rw-r--r-- 1 unifi unifi  1424 Jan 20 13:42 model_lifecycles.json
-rw-r--r-- 1 unifi unifi  1461 Jan 20 13:42 system.properties
-rw-r--r-- 1 unifi unifi  1461 Jan 20 13:42 system.properties.bk
unifi@64602e3bbf81:/usr/lib/unifi/data$

From within the running container, these are the non-standard/Linux-out-of-the-box users available:

unifi:x:999:999::/home/unifi:/bin/sh
mongodb:x:101:102::/var/lib/mongodb:/usr/sbin/nologin

On the host we instead have an old account no longer used on that id

netdata:x:999:998:netdata:/var/lib/netdata:

...and the directory on the host where docker-compose lives looks like this.
Feels more and more like a permissions issue..

root@abc1:/opt/docker/container/unifi-controller# ls -la
total 16
drwxr-xr-x  3 root root 4096 Jan 20 13:14 .
drwxr-xr-x 47 root root 4096 Nov 21 20:04 ..
drwxr-xr-x  2 root root 4096 Jan 20 13:14 backup
-rw-r--r--  1 root root 2590 Jan 20 13:10 docker-compose.yaml
root@abc1:/opt/docker/container/unifi-controller#

jacobalberty · 2022-01-20T16:57:33Z

In your configuration yeah it should be unifi.unifi as the owner user/group . So that whole /usr/lib/unifi/data folder should be a symlink to /unifi/data

So the problem you're running into is the same one I've been struggling to come up with a solution to in #362 / #373 I do not have a good solution as of yet. On my setup I manually chown those directories to be owned by the unifi uid/gid (default 999/999) I've investigated using a setuid binary to change permissions style solution but I haven't found one that felt right to me yet.

mountaindude · 2022-01-20T17:16:51Z

Ok makes sense.
I think - got to admit I am not 100% comfortable with the Linux user/group model... But good enough I think.

I'll move things around on this server, freeing up 999:999 on the host etc.
Then we'll if either or even both the issues go away.

mountaindude · 2022-01-20T17:38:14Z

Yup, that did the trick.

I can now do manual backup downloads as expected.
There was an initial error in the logs, complaining about /usr/lib/unifi/data/backup/meta.json not existing:

[2022-01-20T18:31:16,860] <backup> ERROR system - Failed to load file[/usr/lib/unifi/data/backup/meta.json] as JSON java.io.FileNotFoundException: /usr/lib/unifi/data/backup/meta.json (No such file or directory)

But I guess the backup file was created anyway, because it downloaded just fine.
2nd and later attempts worked without any errors in the logs.

I've set an auto-backup to execute in ca 25 minutes, will report back what happened.

I'd be happy to do a short writeup on the steps/commands needed to solve this.
In case others run into it too and want a quick'n'easy solution.

Where would that best go? In a markdown file in the examples folder?
Or in the main README.md?

Maybe a reference from the readme to the more exhaustive description in the examples folder?

jacobalberty · 2022-01-20T17:39:58Z

Maybe a reference from the readme to the more exhaustive description in the examples folder?

That sounds good to me

mountaindude · 2022-01-20T18:07:20Z

The scheduled auto-backup that just ran resulted in an error in the log files:

[2022-01-20T19:00:00,075] <schedule-backup> ERROR system - Failed to load file[/usr/lib/unifi/data/backup/autobackup/autobackup_meta.json] as JSON java.io.FileNotFoundException: /usr/lib/unifi/data/backup/autobackup/autobackup_meta.json (No such file or directory)

But the backup was created anyway, so I suspect this happens just the first time when that file doesn't exist.
Coming backups will probably happen without log errors. I have another one scheduled in 55 minutes, will verify then.

Will include the above in the howto/writeup too.

jacobalberty · 2022-02-19T19:48:03Z

This is quite likely fixed in latest. I have not merged it into the v6.5 stuff yet but it should be working in latest as of right now. and it will definitely be in v7.0.x

github-actions · 2022-03-22T02:41:30Z

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

Jeb-InnoVision added the bug label Jan 10, 2022

Jeb-InnoVision assigned jacobalberty Jan 10, 2022

mountaindude mentioned this issue Jan 20, 2022

[DOC] Add tutorial on how to fix permissions issues that cause backup… #513

Merged

3 tasks

github-actions bot added the no-issue-activity label Mar 22, 2022

github-actions bot closed this as completed Mar 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unifi Controller does not Produce Backups #512

Unifi Controller does not Produce Backups #512

Jeb-InnoVision commented Jan 10, 2022

mountaindude commented Jan 20, 2022

jacobalberty commented Jan 20, 2022

mountaindude commented Jan 20, 2022

mountaindude commented Jan 20, 2022 •

edited

jacobalberty commented Jan 20, 2022

mountaindude commented Jan 20, 2022

mountaindude commented Jan 20, 2022 •

edited

jacobalberty commented Jan 20, 2022

mountaindude commented Jan 20, 2022

jacobalberty commented Feb 19, 2022

github-actions bot commented Mar 22, 2022

Unifi Controller does not Produce Backups #512

Unifi Controller does not Produce Backups #512

Comments

Jeb-InnoVision commented Jan 10, 2022

Operating system

UniFi Tag

Docker run

Bug description

Steps to reproduce

Relevant log output

mountaindude commented Jan 20, 2022

jacobalberty commented Jan 20, 2022

mountaindude commented Jan 20, 2022

mountaindude commented Jan 20, 2022 • edited

jacobalberty commented Jan 20, 2022

mountaindude commented Jan 20, 2022

mountaindude commented Jan 20, 2022 • edited

jacobalberty commented Jan 20, 2022

mountaindude commented Jan 20, 2022

jacobalberty commented Feb 19, 2022

github-actions bot commented Mar 22, 2022

mountaindude commented Jan 20, 2022 •

edited

mountaindude commented Jan 20, 2022 •

edited