test_basic_functionality fails in 2038

[bmwiedemann]

While working on reproducible builds for openSUSE, I found that our borgbackup 1.2.1 package fails to build in 2038 from a failed test.

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes. Found #2347

Is this a BUG / ISSUE report or a QUESTION?

BUG

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

1.2.1

Operating system (distribution) and version.

openSUSE Tumbleweed 20220826

Hardware / network configuration, and filesystems used.

x86_64, ext4

How much data is handled by borg?

NA

Full borg commandline that lead to the problem (leave away excludes and passwords)

build/test

Describe the problem you're observing.

test fails

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

on openSUSE or Debian do

osc co openSUSE:Factory/borgbackup && cd $_
osc build --vm-type=kvm --clean --noservice --build-opt=--vm-custom-opt="-rtc base=2038-01-25T00:00:00" --alternative-project home:bmwiedemann:reproducible openSUSE_Tumbleweed

Include any warning/errors/backtraces from the system logs

=================================== FAILURES ===================================
__________________ ArchiverTestCase.test_basic_functionality ___________________

self = <borg.testsuite.archiver.ArchiverTestCase testMethod=test_basic_functionality>

    def test_basic_functionality(self):
        have_root = self.create_test_files()
        # fork required to test show-rc output
        output = self.cmd('init', '--encryption=repokey', '--show-version', '--show-rc', self.repository_location, fork=True)
        self.assert_in('borgbackup version', output)
        self.assert_in('terminating with success status, rc 0', output)
        self.cmd('create', '--exclude-nodump', self.repository_location + '::test', 'input')
        output = self.cmd('create', '--exclude-nodump', '--stats', self.repository_location + '::test.2', 'input')
        self.assert_in('Archive name: test.2', output)
        self.assert_in('This archive: ', output)
        with changedir('output'):
            self.cmd('extract', self.repository_location + '::test')
        list_output = self.cmd('list', '--short', self.repository_location)
        self.assert_in('test', list_output)
        self.assert_in('test.2', list_output)
        expected = [
            'input',
            'input/bdev',
            'input/cdev',
            'input/dir2',
            'input/dir2/file2',
            'input/empty',
            'input/file1',
            'input/flagfile',
        ]
        if are_fifos_supported():
            expected.append('input/fifo1')
        if are_symlinks_supported():
            expected.append('input/link1')
        if are_hardlinks_supported():
            expected.append('input/hardlink')
        if not have_root:
            # we could not create these device files without (fake)root
            expected.remove('input/bdev')
            expected.remove('input/cdev')
        if has_lchflags:
            # remove the file we did not backup, so input and output become equal
            expected.remove('input/flagfile')  # this file is UF_NODUMP
            os.remove(os.path.join('input', 'flagfile'))
        list_output = self.cmd('list', '--short', self.repository_location + '::test')
        for name in expected:
            self.assert_in(name, list_output)
>       self.assert_dirs_equal('input', 'output/input')

build/lib.linux-x86_64-cpython-310/borg/testsuite/archiver.py:434: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
build/lib.linux-x86_64-cpython-310/borg/testsuite/__init__.py:181: in assert_dirs_equal
    self._assert_dirs_equal_cmp(diff, **kwargs)
build/lib.linux-x86_64-cpython-310/borg/testsuite/__init__.py:227: in _assert_dirs_equal_cmp
    self.assert_equal(d1, d2)
E   AssertionError: Lists differ: ['dir2', '0o40700', 399, 399, None, 2, 0, 2168741486428000000, {}] != ['dir2', '0o40700', 399, 399, None, 2, 0, 2147310847000000000, {}]
E   
E   First differing element 7:
E   2168741486428000000
E   2147310847000000000
E   
E   - ['dir2', '0o40700', 399, 399, None, 2, 0, 2168741486428000000, {}]
E   ?                                             ^  ^^^^^^^^
E   
E   + ['dir2', '0o40700', 399, 399, None, 2, 0, 2147310847000000000, {}]
E   ?                                             ^^^^^ + ^^^
=========================== short test summary info ============================
FAILED build/lib.linux-x86_64-cpython-310/borg/testsuite/archiver.py::ArchiverTestCase::test_basic_functionality
!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
============ 1 failed, 42 passed, 1 skipped, 54 deselected in 9.99s ============

Background:
As part of my work on reproducible builds for openSUSE, I check that software still gives identical build results in the future.
The usual offset is +16 years, because that is how long I expect some software will be used in some places.
This showed up failing tests in our package build.

[ThomasWaldmann]

We currently try to still support 32bit systems with borg, so guess this is the clamping to a 32bit safe range to avoid overflows.

So I'ld suggest you:

• try to reproduce the failure - does it always happen?
• go back to 2037 - if that makes the test succeed, it is likely the intentional clamping.

Currently, such timestamps are simply file system errors and misbehaved software. The timestamp is obviously untrue from a current perspective, so nothing of worth is lost if we clamp it to a non-overflowing range.

As time goes by, we will at some time drop support for 32bit timestamps. Likely not this year, but maybe we can already do it in a few years, when even the smallest systems which run borg will all run 64bit OSes.

[ThomasWaldmann]

https://github.com/borgbackup/borg/blob/1.2.2/src/borg/helpers/time.py#L42 see there.

[ThomasWaldmann]

MAX_S == 2147310847
MAX_NS == 2147310847000000000

So what you see there in the failing test is obviously a value clamped to MAX_NS.

[ThomasWaldmann]

So, I am closing this as "works as designed".

[ThomasWaldmann]

For your tests, if your intention is not to find y2k38 issues, I'ld suggest you do not go beyond the 32bit time limits.

If you do want to find such issues, well you have found one. But as you see in the file linked above, the solution for borg will be just to switch off 32bit timestamp support in the not so far future.

[bmwiedemann]

Yes, I was trying to find y2k38 issues.
What would happen, if we set SUPPORT_32BIT_PLATFORMS = False in our 64-bit builds, but kept it True in our i586 and armv7 builds?

[bmwiedemann]

https://build.opensuse.org/request/show/999784 seems like a good solution. Thanks for your input.

[ThomasWaldmann]

As long as all tested platforms are 64bit, yes. Not sure if our tests produce an overflow condition for the 32bit platforms.

[frispete]

Before accepting this modification, @ThomasWaldmann can you give me an appraisal, how this change will affect existing borg archives, especially if generated in 32bit OS environments, or written to in mixed installations of borg with differing values for that switch? In short, does it impact the archival format in any way?

[ThomasWaldmann]

IIRC, as long as timestamps (e.g. the ones seen in the filesystem) are in the valid range for 32bits (1970-2038), it won't make a difference in the repos/archives.

If a timestamp is out of the 32bit (now, or later: 64bit) range, it gets clamped to that range and nothing of value is lost, because negative timestamps or timestamps in the far future should not exist anyway.

[anarcat]

32 bit systems are getting really rare. I don't think we should support them anymore: Debian is about to drop i386 from its supported architectures, and so is the linux kernel. ARM also have very few 32 bit systems supported as well.

2038 issues are important to fix now, sooner than later. At the very least this check should be automatically tweaked depending on the architecture, and not have a "fail in 15 years" default. Architectures and packages that do need 32 bit support can either patch this stuff out or will just work automatically if there's some detection of this problem.

Would you consider a PR making this setting more dynamic?

Thanks

[leahneukirchen]

Clamping should happen on extraction ("best-effort"), not on storage.

[Jookia]

32 bit systems are getting really rare. I don't think we should support them anymore: Debian is about to drop i386 from its supported architectures, and so is the linux kernel. ARM also have very few 32 bit systems supported as well.

Could you elaborate on this, perhaps with some links? I can't find anything about Linux dropping 32-bit x86 support and Debian seems to be pumping i686 as usual for 12.

As for ARM having very few 32-bit systems: There are new 32-bit SoCs being made every year that support Linux. Look at the stm32mp13 which is due to be announced soon (probably within a year?). The same is happening with RISC-V with chips like Allwinner's D1. 32-bit systems hit that niche of 'I need something better than a microcontroller but smaller than compute' and they aren't going away in the forseeable future.

As to whether borg supports these, it would be nice. Why not use 64-bit timestamps on 32-bit systems?

[anarcat]

2 bit systems are getting really rare. I don't think we should support them anymore: Debian is about to drop i386 from its supported architectures, and so is the linux kernel. ARM also have very few 32 bit systems supported as well.

Could you elaborate on this, perhaps with some links? I can't find anything about Linux dropping 32-bit x86 support and Debian seems to be pumping i686 as usual for 12.

well there's nothing really formal yet. linux will probably support 32bit for a while longer still, because of embedded systems, but it's certainly something that's talked about:

https://lwn.net/Articles/838807/

32bit is also not getting speculative mitigation:

https://www.phoronix.com/news/Linux-x86-Retbleed

for debian, it's not clear yet, but it's certainly suffering, according to the release team:

https://release.debian.org/testing/arch_qualify.html

... look at the i386 column, not looking great...

As for ARM having very few 32-bit systems: There are new 32-bit SoCs being made every year that support Linux. Look at the stm32mp13 which is due to be announced soon (probably within a year?). The same is happening with RISC-V with chips like Allwinner's D1. 32-bit systems hit that niche of 'I need something better than a microcontroller but smaller than compute' and they aren't going away in the forseeable future.

true, if you count embedded systems, those matter quite a bit, and for a while.

As to whether borg supports these, it would be nice. Why not use 64-bit timestamps on 32-bit systems?

I think because integers are 32 bit on 32 bit systems, but maybe i'm missing something...

[Jookia]

well there's nothing really formal yet. linux will probably support 32bit for a while longer still, because of embedded systems, but it's certainly something that's talked about:

https://lwn.net/Articles/838807/

32bit is also not getting speculative mitigation:

https://www.phoronix.com/news/Linux-x86-Retbleed

for debian, it's not clear yet, but it's certainly suffering, according to the release team:

https://release.debian.org/testing/arch_qualify.html

... look at the i386 column, not looking great...

That's a lot further than saying it's about to drop support. Edit: and only focused on x86. ARMv7 has speculative mitigation for example.

As to whether borg supports these, it would be nice. Why not use 64-bit timestamps on 32-bit systems?

I think because integers are 32 bit on 32 bit systems, but maybe i'm missing something...

You don't have to only use integers in programs. There's other data types, such as long long which is 64-bit. There's also time structures like time_t provided by libc. On Linux time_t is 64-bit with 32-bit compatibility for libcs that haven't exported 64-bit time_t yet and old programs.

[anarcat]

On 2022-09-22 12:04:23, Jookia wrote: > for debian, it's not clear yet, but it's certainly suffering, according to the release team: > > https://release.debian.org/testing/arch_qualify.html > > ... look at the i386 column, not looking great... That's a lot further than saying it's about to drop support.

Well, there's been other conversations about this. :)

> > As to whether borg supports these, it would be nice. Why not use 64-bit timestamps on 32-bit systems? > > I think because integers are 32 bit on 32 bit systems, but maybe i'm missing something... You don't have to only use integers in programs. There's other data types, such as long long which is 64-bit. There's also time structures like time_t provided by libc. On Linux time_t is 64-bit with 32-bit compatibility for libcs that haven't exported 64-bit time_t yet and old programs.

I guess so, I'm just assuming there's a reason why this is clamped in borg, and that the reason is more complicated than "just use long long". This is not (entirely) C after all: the affected code looks like Python... But maybe it's all in the C code too, I haven't checked.

…

-- My passionate sense of social justice and social responsibility has always contrasted oddly with my pronounced lack of need for direct contact with other human beings and communities. I am truly a "lone traveler" and have never belonged to my country, my home, my friends, or even my immediate family, with my whole heart; in the face of all these ties, I have never lost a sense of distance and a need for solitude. - Albert Einstein