RHEL 6 & 7 utility tmpwatch deletes backups (e.g. in case of BACKUP=SSHFS)

[John-Leone]

Relax-and-Recover (ReaR) Issue Template

Relax-and-Recover 2.00 / Git

[root@xrearm2d rear]# cat /etc/rear/os.conf
OS_VENDOR=RedHatEnterpriseServer
OS_VERSION=6

# This file (local.conf) is intended for manual configuration. For configuration
# through packages and other automated means we recommend creating a new
# file named site.conf next to this file and to leave the local.conf as it is.
# Our packages will never ship with a site.conf.
#
export TMPDIR="/mnt/rear/"
OUTPUT=ISO
BACKUP=NETFS
OUTPUT_URL=sshfs://root@xxx.xxx.xxx.xxx/dbar/
BACKUP_URL=sshfs://root@xxx.xxx.xxx.xxx/dbar/
KEEP_OLD_OUTPUT_COPY=1
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/var/cache/yum')

x86

Hello,
This is informational only, we use SSHFS to run ReaR for backups. During a backup, SSHFS creates a working area mount point in /tmp. The mount is /tmp/rear.xxxxxxxx/outputfs.
We always run backups during the overnight hours, and noticed files were being deleted from our backup servers. After searching for three months on why our backups were deleted we found the issue.
Redhat 6 & 7 has utilities called tmpwatch and tmpfiles that run daily. These utilities delete files greater than 10 days old from /tmp and /var/tmp. So during the night when ReaR was backing up a server and if tmpwatch started at the same time, tmpwatch would traverse /tmp and delete backups greater that 10 days old on our backup server.
The resolution was to change the location of TMPDIR in the local.conf.

Thanks,
John

[gozora]

This is exact reason why I'm not a fan of automatic cleanup scripts. There is always that "little something" that should not pass the cleanup filter, but it does (for whatever reason).
BTW /var/tmp could be subject for cleanup as well, I'd even say that is is option # 2 for cleanup right after /tmp, so I'd not consider this final solution either, but that is just matter of taste.

V.

[jsmeix]

@John-Leone
many thanks for your information and in particular
for your explanatory description what goes on
so that even I - as non Red Hat user - can easily
understand what goes on on your system.

As far as I understand it it looks as if there is a severe bug
in that tmpfiles cleaning utility because it seems it does not limit
its work on the one filesystem where /tmp directly belongs to
(i.e. that cleaning utility is cossing filesystem boundaries).

As a test on a testing virtual machine where it does not matter
when it gets destroyed I would mount e.g. /usr/ at /tmp/usrmountpoint
and watch what that tmpfiles cleaning utility does with files in /usr
that are older than 10 days ...

[John-Leone]

I'm in agreement with you, we think this is a Redhat bug.
I did open a case with Redhat to report this problem.
Adding the TMPDIR variable will solve this problem for us but I know anyone using CentOS or Fedora can encounter this problem too.

[jsmeix]

Only FYI regarding /tmp and /var/tmp cleanup
see in general e.g.
https://unix.stackexchange.com/questions/30489/what-is-the-difference-between-tmp-and-var-tmp
and
https://en.opensuse.org/openSUSE:Tmp_on_tmpfs#.2Ftmp.2F_versus_.2Fvar.2Ftmp.2F
and
https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard
and in particular regarding FHS see
http://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s18.html

3.18. /tmp : Temporary files

3.18.1. Purpose
The /tmp directory must be made available for programs
that require temporary files.

Programs must not assume that any files or directories
in /tmp are preserved between invocations of the program.
Rationale

IEEE standard POSIX.1-2008 lists requirements similar
to the above section.

Although data stored in /tmp may be deleted
in a site-specific manner, it is recommended
that files and directories located in /tmp
be deleted whenever the system is booted.

FHS added this recommendation on the basis
of historical precedent and common practice,
but did not make it a requirement because
system administration is not within the scope of this standard.

http://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s15.html

5.15. /var/tmp : Temporary files preserved between system reboots

5.15.1. Purpose

The /var/tmp directory is made available for programs
that require temporary files or directories that are
preserved between system reboots.
Therefore, data stored in /var/tmp is more persistent
than data in /tmp.

Files and directories located in /var/tmp must not be deleted
when the system is booted. Although data stored in /var/tmp
is typically deleted in a site-specific manner, it is recommended
that deletions occur at a less frequent interval than /tmp.