Retention again :-(

[sonor3000]

Sorry to come up again with this topic, but I am not able to get retention to work.

In my running container, I am using the :latest tag, the mysql-backup commandline looks like this if I list the running processes:

/mysql-backup dump --target /db --server mariadb --user root --pass secret --retention 2c --frequency 1

Every minute a backup is created but no old backups are deleted.

What am I doing wrong?

[deitch]

I am using the :latest tag

The tag :latest? That one does not support retention. It only was added in the 1.x series (which currently is release candidate).

[PaBox]

Even with the following docker configuration, it is not working:

database-server:
    image: mariadb
    container_name: database-server
    command: [ "--max_connections=1024", "--verbose"]
    # tty: true
    # stdin_open: true
    user: 1599:1600
    environment:
      TZ: Europe/Berlin
      MYSQL_ROOT_PASSWORD: /run/secrets/secret_pw_root_mysql
      MYSQL_DATABASE: mc_db
      MYSQL_USER: mc
      MYSQL_PASSWORD: /run/secrets/secret_pw_user_mysql
    secrets:
      - secret_pw_root_mysql
      - secret_pw_user_mysql
    restart: on-failure:5
    volumes:
      - db-server-volume:/var/lib/mysql
    healthcheck:
      test: [ "CMD", "healthcheck.sh", "--connect", "--innodb_initialized" ]
      start_period: 0m
      interval: 10s
      retries: 25
    networks:
      - mc-backup-network
      - mc-proxy-network
  db-backup-server:
    image: databack/mysql-backup:1.0.0 # check appropriate image tags
    container_name: db-backup-server
    user: 1599:1600
    environment:
      TZ: Europe/Berlin
      DB_DUMP_DEBUG: true
      # DB_DUMP_SAFECHARS: true
      DB_SERVER: database-server
      DB_USER: mc
      DB_PASS: /run/secrets/secret_pw_user_mysql
      DB_DUMP_TARGET: /db
      DB_DUMP_BEGIN: "+0"
      DB_DUMP_FREQUENCY: 1
      RETENTION: "2c"
    command: dump
    secrets:
      - secret_pw_user_mysql
    restart: always
    depends_on:
      database-server:
        condition: service_healthy
    volumes:
      - db-backup-volume:/db
    networks:
      - mc-backup-network

So even with 1.0.0, the retention is not working correctly (in my case).

Also there are no errors to report, even with debug output enabled. Really weird. Shouldn't it be at least reporting an issue if retention != "" ?

P.S.: I noticed getting the following output now when supplying "command: dump --retention "2c"" instead of "command: dump";
level=info msg="beginning prune" run=00000000-0000-0000-0000-000000000000
but nothing is actually getting pruned. I can ALSO achieve this by changing "RETENTION" to "DB_DUMP_RETENTION" - is the documentation wrong there and it is also a different variable? It seems like there are problems launching the process and I verified it by running it all as root or user: "0".

[deitch]

Yes, v1.0.0 should support retention, so if something is not working correctly, this is a valid issue.

is the documentation wrong there and it is also a different variable?

Yes, correct. It should be DB_DUMP_RETENTION, because it sets the prefix. This should be a separate issue, but don't bother, we can fix docs in minutes, see #376.

The more important issue is the retention not working. There are various tests for retention, so it should have caught issues, but something always can slip through. Can you run with debugging enabled? --v=1 (or if you want a lot, --v=2)?

The logging output should filter out sensitive information, but please check before posting nonetheless.

[PaBox]

Yes, v1.0.0 should support retention, so if something is not working correctly, this is a valid issue.

is the documentation wrong there and it is also a different variable?

Yes, correct. It should be DB_DUMP_RETENTION, because it sets the prefix. This should be a separate issue, but don't bother, we can fix docs in minutes, see #376.

The more important issue is the retention not working. There are various tests for retention, so it should have caught issues, but something always can slip through. Can you run with debugging enabled? --v=1 (or if you want a lot, --v=2)?

The logging output should filter out sensitive information, but please check before posting nonetheless.

As you can see, Debugging is enabled via the environment file. Even though, I added --verbose=2 and got the following output:

db-backup-server  | time="2024-11-21T13:36:43Z" level=info msg="beginning dump 2024-11-21T13:36:43Z" run=56220c07-f16e-48b1-a5e5-01eca09ca039
db-backup-server  | time="2024-11-21T13:36:43Z" level=debug msg="uploading via protocol file from db_backup_2024-11-21T13:36:43Z.tgz to db_backup_2024-11-21T13:36:43Z.tgz" run=56220c07-f16e-48b1-a5e5-01eca09ca039
db-backup-server  | time="2024-11-21T13:36:43Z" level=debug msg="completed copying 629 bytes" run=56220c07-f16e-48b1-a5e5-01eca09ca039
db-backup-server  | time="2024-11-21T13:36:43Z" level=info msg="beginning prune" run=00000000-0000-0000-0000-000000000000
db-backup-server  | time="2024-11-21T13:36:43Z" level=debug msg="pruning target &{{file  %!s(*url.Userinfo=<nil>)  /db  %!s(bool=false) %!s(bool=false)   } /db}" run=00000000-0000-0000-0000-000000000000
db-backup-server  | time="2024-11-21T13:36:43Z" level=debug msg="checking filename that is standard backup pattern: db_backup_2024-11-21T00:05:53Z.tgz" run=00000000-0000-0000-0000-000000000000
db-backup-server  | time="2024-11-21T13:36:43Z" level=debug msg="checking filename that is standard backup pattern: db_backup_2024-11-21T02:21:12Z.tgz" run=00000000-0000-0000-0000-000000000000
db-backup-server  | time="2024-11-21T13:36:43Z" level=debug msg="checking filename that is standard backup pattern: db_backup_2024-11-21T13:35:43Z.tgz" run=00000000-0000-0000-0000-000000000000
db-backup-server  | time="2024-11-21T13:36:43Z" level=debug msg="checking filename that is standard backup pattern: db_backup_2024-11-21T13:36:43Z.tgz" run=00000000-0000-0000-0000-000000000000
db-backup-server  | time="2024-11-21T13:36:43Z" level=debug msg="pruning 0 files from target &{{file  %!s(*url.Userinfo=<nil>)  /db  %!s(bool=false) %!s(bool=false)   } /db}" run=00000000-0000-0000-0000-000000000000

This could indicate, that retention is not recognizing the new file format(I don't know since when its used though). Going by the documentation the default file naming scheme has changed in the past and as retention is not going further than checking the backup pattern, It might be it.

Also I noticed it starting to work now with setting a time versus "2c" - it now checks the age correctly and deletes them after "1h" as it should be. Really weird.

[deitch]

Small things first. The run ID was not being passed to pruning when called from Dump, and you can see the target is messed up in the log message, so fixing that. Thank you for catching it.

[deitch]

checking filename that is standard backup pattern

This means that the filename matched the pattern. Otherwise, it would have said: "ignoring filename that is not standard backup pattern". So it is ok with the filename pattern.

Something about the retain count logic is off. It looks right, and passes test, and yet it doesn't prune yours. And as you said, it works correctly with time (which is a different logic) than count.

[deitch]

Phew. #378 should fix it. Just merged in. When this job is done, you should have a new master image you can test. Please comment here?

[PaBox]

Is 1.0.0 the same as master? Or may there be unexpected consequences when switching - Just because the documentation and past comments suggested the old version is still on latest and master (but I am probably wrong about master).

I already found another problem when restoring via compose (SIGSEGV) so I will make another issue in a 2 hours or so anyway and start testing again :)

ADDED: Seems like I didn't have enough time today - I had to rework our bash script by a lot (different reasons like a UI) so I will test it tomorrow. Sorry for the delay. Another Issue will also be on the way.

[deitch]

Is 1.0.0 the same as master?

Yeah master is on the v1 stream, so no functional difference. Assuming these fix the issue, I plan to cut a v1.0.1.

[iamriajul]

Prune is not working with S3 Storage

[deitch]

@iamriajul welcome. What version are you using? And container image or binary? What debug messages do you see?

[iamriajul]

@deitch I'm using the container version. Use version 1.0.0
My docker service:

[iamriajul]

I tried running the prune command manually, but it says pruning 0 files similar to PaBox's comment

[deitch]

#378 went in post-v1.0.0. We do need to get a new release out there. In the meantime, try running it against the latest master branch, i.e. image: databack/mysql-backup:master (don't do that regularly or you always will get the most recent, including unstable). If that doesn't work, enabel debug logging and open a new issue.

In the meantime, I will close this one.

[iamriajul]

But I tried hourly retention too, that didn't work