backblaze deleting older than N days broken (PRO/7.7)

[maxslug]

I'm setting up backblaze-based backup and it seems the remove old backups option isn't working right. Might be a first-time thing, or not, not sure.

First Full backup run:

Starting backup of 1 domains to Backblaze bucket server_name-%d-%m-%Y ..

Testing backup destination Backblaze bucket server_name-13-09-2023 ..
.. done

Creating backup for virtual server server.com ..

    Copying virtual server configuration ..
    .. done
    Saving mail aliases ..
    .. done
    Saving mail and FTP users ..
    .. done
    Backing up mail and FTP user Cron jobs ..
    .. none to backup
    Copying Apache virtual host configuration ..
    .. done
    Copying SSL Apache virtual host configuration and certificate ..
    .. done
    Copying Logrotate configuration ..
    .. done
    Copying Procmail and SpamAssassin configuration files ..
    .. done
    Backing up AWStats configuration file ..
    .. done
    Creating TAR file of home directory ..
    .. done
    Uploading archive to Backblaze ..
    .. done

.. completed in 10 seconds


    Saving Virtualmin configuration ..
    .. done
    Saving templates and plans ..
    .. done
    Saving resellers ..
    .. done
    Saving email templates ..
    .. done
    Saving custom fields, links, categories and shells ..
    .. done
    Saving custom script installers ..
    .. done
    Saving scheduled backups and keys ..
    .. done
    Saving DKIM settings ..
    .. done
    Saving greylisting settings ..
    .. done
    Save mail rate limiting configuration ..
    .. done
    Saving mail server configuration ..
    .. done

.. done

Uploading archive to Backblaze ..
.. done

1 servers backed up successfully, 0 had errors. 8 Virtualmin configuration settings backed up successfully.

Backup is complete. Final size was 308.02 KiB.

Deleting backups from Backblaze bucket server_name-%d-%m-%Y matching server_name-.*-.*-.* older than 180 days ..

    .. failed to list Dropbox files : b2 ls: error: the following arguments are required: bucketName b2 ls [-h] [--profile PROFILE] [--long] [--json] [--replication] [--versions] [--recursive] [--withWildcard] bucketName [folderName] Using the file naming convention that / separates folder names from their contents, returns a list of the files and folders in a given folder. If no folder name is given, lists all files at the top level. The --long option produces very wide multi-column output showing the upload date/time, file size, file id, whether it is an uploaded file or the hiding of a file, and the file name. Folders don't really exist in B2, so folders are shown with - in each of the fields other than the name. The --json option produces machine-readable output similar to the server api response format. The --replication option adds replication status The --versions option selects all versions of each file, not just the most recent. The --recursive option will descend into folders, and will select only files, not folders. The --withWildcard option will allow using *, ? and `[]` characters in folderName as a greedy wildcard, single character wildcard and range of characters. It requires the --recursive option. Remember to quote folderName to avoid shell expansion. Examples �[1mNote�[0m Note the use of quotes, to ensure that special characters are not expanded by the shell. List csv and tsv files (in any directory, in the whole bucket): b2 ls --recursive --withWildcard bucketName "*.[ct]sv" List all info.txt files from buckets bX, where X is any character: b2 ls --recursive --withWildcard bucketName "b?/info.txt" List all pdf files from buckets b0 to b9 (including sub-directories): b2 ls --recursive --withWildcard bucketName "b[0-9]/*.pdf" Requires capability: • �[1mlistFiles�[0m positional arguments: bucketName folderName options: -h, --help show this help message and exit --profile PROFILE --long --json --replication --versions --recursive --withWildcard 

[jcameron]

Oh I see the cause of this .... it will be fixed in the next Virtualmin release (7.9 or higher)

[iliajie]

Oh I see the cause of this .... it will be fixed in the next Virtualmin release (7.9 or higher)

We can always do 7.8.3 or 7.8.4 .. as those are bug-fixes and this is what bug releases are for .. we shouldn't postpone it to 7.9! 7.9 is more about new features, while 7.8.x should be more about bug-fixes.

[maxslug]

I flagged a potential problem in the commit above in comments.

[maxslug]

And for anyone who needs to cleanup old buckets manually...

ID=000adcdef00000000000
KEY=12345
BUCKET=abcd

buckets:
	b2 list-buckets | tee buckets.txt

rm-all:
	for b in `cat rm.txt`; do echo $$b; b2 rm --versions --recursive $$b; done
	for b in `cat rm.txt`; do echo $$b; b2 delete-bucket $$b; done

rm:
	b2 rm --versions --recursive $(BUCKET)

auth:
	b2 authorize-account $(ID) $(KEY)

[maxslug]

I'm still seeing this issue with 7.9.0 Pro.

Backup failed! See the progress output above for the reason why. Total backup time was 03 minutes, 17 seconds.

Sent by Virtualmin at: https://server.com:10000/
...
11 servers backed up successfully, 0 had errors.

Deleting backups from local file /usr/share/webmin/virtual-server/bb: matching backup-name-.*-.*-.* older than 90 days ..
    .. failed to list Dropbox files : 
b2 ls: error: the following arguments are required: bucketName
b2 ls [-h] [--profile PROFILE] [--long] [--json] [--replication] [--versions]
      [--recursive] [--withWildcard]
      bucketName [folderName]

  Using the file naming convention that / separates folder names from their
  contents, returns a list of the files and folders in a given folder. If no
  folder name is given, lists all files at the top level.

  The --long option produces very wide multi-column output showing the upload
  date/time, file size, file id, whether it is an uploaded file or the hiding
  of a file, and the file name. Folders don't really exist in B2, so folders
  are shown with - in each of the fields other than the name.

  The --json option produces machine-readable output similar to the server api
  response format.

  The --replication option adds replication status

  The --versions option selects all versions of each file, not just the most
  recent.

  The --recursive option will descend into folders, and will select only files,
  not folders.

  The --withWildcard option will allow using *, ? and `[]` characters in
  folderName as a greedy wildcard, single character wildcard and range of
  characters. It requires the --recursive option. Remember to quote folderName
  to avoid shell expansion.

  Examples

  �[1mNote�[0m

    Note the use of quotes, to ensure that special characters are not expanded
    by the shell.

  List csv and tsv files (in any directory, in the whole bucket):

  b2 ls --recursive --withWildcard bucketName "*.[ct]sv"

  List all info.txt files from buckets bX, where X is any character:

  b2 ls --recursive --withWildcard bucketName "b?/info.txt"

  List all pdf files from buckets b0 to b9 (including sub-directories):

  b2 ls --recursive --withWildcard bucketName "b[0-9]/*.pdf"

  Requires capability:

    • �[1mlistFiles�[0m

positional arguments:
  bucketName
  folderName

options:
  -h, --help         show this help message and exit
  --profile PROFILE
  --long
  --json
  --replication
  --versions
  --recursive
  --withWildcard

[jcameron]

Can you post a screenshot of the destination and cleanup options section of the form for this backup?

[maxslug]

Can you post a screenshot of the destination and cleanup options section of the form for this backup?

Sure thing, PFA.

[jcameron]

Ok the issue here is that we don't support purging of date-based backblaze buckets properly, only of directories under buckets. But we'll fix this in the next release.

As a work-around, you could backup to a path like virtualminbackups/server-%Y-%m-%d