make test issues with changed Python3 regex behavior

[jbglaw]

Hi!

As I found the retro-fuse binaries useful, I wanted to add them to my CI builds setup (see http://toolchain.lug-owl.de/). Building was just a breeze, but I see unexpected errors during make test. In my setup, everything is built within a Docker container, but even running make test as root gives me some issues:

root@lili:/var/cache/git/retrofuse [master] # python3 --version
Python 3.11.2

root@lili:/var/cache/git/retrofuse [master] # make test                                                                                                                                        
./test/retro-fuse-test.py v6fs                                                                                                                                                                 
Create File ... PASS                                                                                                                                                                           
Delete File ... PASS
Rename File ... PASS
Create File Link ... PASS
Delete File Link ... PASS
Change File Mode ... PASS
Change File Times ... PASS
Change File Group ... skipped 'Suitable group not found'
Read/Write File ... PASS
Overwrite File ... PASS
Extend File ... PASS
Truncate File ... PASS
Read/Write Sparse File ... PASS
Create Directory ... PASS
Remove Directory ... PASS
Rename Directory ... PASS
Create Nested Directories ... PASS
Move Directory ... PASS
Change Directory Mode ... PASS
Filesystem Check ... ERROR

ERROR: Filesystem Check
Traceback (most recent call last):
  File "/var/cache/git/retrofuse/test/V6Tests.py", line 88, in test_99_FilesystemCheck
    v6FileList = v6.enumFiles('/mnt')
                 ^^^^^^^^^^^^^^^^^^^^
  File "/var/cache/git/retrofuse/test/SimhDrivers.py", line 166, in enumFiles
    return FileList.parse(fileListStr, re.compile(self._fileListPattern))
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/re/__init__.py", line 227, in compile
    return _compile(pattern, flags)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/re/__init__.py", line 294, in _compile
    p = _compiler.compile(pattern, flags)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/re/_compiler.py", line 743, in compile
    p = _parser.parse(p, flags)
        ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/re/_parser.py", line 980, in parse
    p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/re/_parser.py", line 455, in _parse_sub
    itemsappend(_parse(source, state, verbose, nested + 1,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/re/_parser.py", line 841, in _parse
    raise source.error('global flags not at the start '
re.error: global flags not at the start of the expression at position 25 (line 2, column 25)

Ran 20 tests in 41.817s

FAILED (errors=1, skipped=1)
make: *** [Makefile:219: test-v6fs] Error 1

With this patch, things go further:

diff --git a/test/SimhDrivers.py b/test/SimhDrivers.py
index 7fbcad1..3ee7ba1 100644
--- a/test/SimhDrivers.py
+++ b/test/SimhDrivers.py
@@ -205,8 +205,7 @@ attach rk0 system.dsk
     _fileListCmd = '/usr/bin/find %s -exec /bin/ls -ild {} \\; -o \\( -type -f -a -exec /usr/bin/cksum {} \\; \\) -o \\( \\! -type -f -a -exec echo - \\; \\)'
 
     # Regex pattern to parse output from the above command.
-    _fileListPattern = r'''
-                        (?mx)
+    _fileListPattern = r'''(?mx)
                         ^
                         \s* (?P<inode> \d+)
                         \s+ (?P<type> [bcd-]) (?P<mode> [rwsStTx-]{9}t?)
@@ -344,8 +343,7 @@ attach rp0 system.dsk
     _fileListCmd = '/bin/find %s -exec /bin/ls -ild {} \\; -a \\( -type f -a -exec /bin/cksum {} \\; \\) -o \\( \\! -type f -a -exec echo - \\; \\)'
 
     # Regex pattern to parse output from the above command.
-    _fileListPattern = r'''
-                        (?mx)
+    _fileListPattern = r'''(?mx)
                         ^
                         \s* (?P<inode> \d+)
                         \s+ (?P<type> [bcd-]) (?P<mode> [rwsStTx-]{9}t?)
@@ -492,9 +490,8 @@ attach rl1 swap.dsk
     _fileListCmd = '/bin/find %s -exec /usr/ucb/ls -ildn {} \\; \\( -type f -exec /bin/cksum {} \\; \\) -o -exec echo - \\;'
 
     # Regex pattern to parse output from the above command.
-    _fileListPattern = r'''
+    _fileListPattern = r'''(?mx)
                         ^
-                        (?mx)
                         \s* (?P<inode> \d+)
                         \s+ (?P<type> [bcd-]) (?P<mode> [rwsStTx-]{9})
                         \s* (?P<linkCount> \d+)
@@ -646,9 +643,8 @@ attach rq0 system.dsk
     _fileListCmd = '/usr/bin/find %s -exec /bin/ls -ildg {} \\; \\( -type f -exec /bin/cksum {} \\; \\) -o -exec echo - \\;'
 
     # Regex pattern to parse output from the above command.
-    _fileListPattern = r'''
+    _fileListPattern = r'''(?mx)
                         ^
-                        (?mx)
                         \s* (?P<inode> \d+)
                         \s+ (?P<type> [bcd-]) (?P<mode> [rwsStTx-]{9})
                         \s* (?P<linkCount> \d+)

...and tests pass with current OpenSIMH:

root@lili:/var/cache/git/retrofuse [master] # pdp11

PDP-11 simulator Open SIMH V4.1-0 Current        git commit id: 348f5f29
sim> sh ver
PDP-11 simulator Open SIMH V4.1-0 Current
    Simulator Framework Capabilities:
        32b data
        32b addresses
        Threaded Ethernet Packet transports:PCAP:TAP:VDE:NAT:UDP
        Idle/Throttling support is available
        Virtual Hard Disk (VHD) support
        RAW disk and CD/DVD ROM support
        Asynchronous I/O support (Lock free asynchronous event queue)
        Asynchronous Clock support
        FrontPanel API Version 12
    Host Platform:
        Compiler: GCC 14.0.0 20230711 (experimental)
        Simulator Compiled as C arch: x64 (Release Build) on Aug 13 2023 at 16:19:51
        Build Tool: simh-makefile
        Memory Access: Little Endian
        Memory Pointer Size: 64 bits
        Large File (>2GB) support
        SDL Video support: SDL Version (Compiled: 2.28.0, Runtime: 2.26.2), PNG Version (Compiled: 1.6.40, Runtime: 1.6.39), zlib: (Compiled: 1.2.13, Runtime: 1.2.11)
        PCRE RegEx (Version 8.39 2016-06-14) support for EXPECT commands
        OS clock resolution: 1ms
        Time taken by msleep(1): 1ms
        Ethernet packet info: libpcap version 1.10.3 (with TPACKET_V3)
        OS: Linux lili 5.16.0-4-amd64 #1 SMP PREEMPT Debian 5.16.12-1 (2022-03-08) x86_64 GNU/Linux
        Processor Name: Intel(R) Xeon(R) CPU E5-2450L 0 @ 1.80GHz
        tar tool: tar (GNU tar) 1.34
        curl tool: curl 7.82.0 (x86_64-pc-linux-gnu) libcurl/7.82.0 OpenSSL/1.1.1m zlib/1.2.11 brotli/1.0.9 zstd/1.5.2 libidn2/2.3.3 libpsl/0.21.0 (+libidn2/2.3.0) libssh2/1.10.0 nghttp2/1.52.0 librtmp/2.3 OpenLDAP/2.4.59
        git commit id: 348f5f29
        git commit time: 2023-07-31T10:34:33-0700

[jaylogue]

@jbglaw Thank you for the bug report, and the fix! The master branch has been updated accordingly.

Also, I'm glad to see someone making use of retro-fuse. I hope it's working well for you.

[jbglaw]

Sure! This is a nice project, especially due to its direction of using original sources.

Just out of curiousity: Did you ever find a real bug in former upstream filesystem drivers?

[jaylogue]

Just out of curiousity: Did you ever find a real bug in former upstream filesystem drivers?

I did! I discovered a bug in v7 that leaked an inode whenever the fs ran out of space at just the right moment (fixed here). Interestingly, the bug also appeared in v6 and in 2.9bsd. It was finally fixed (in the bsd lineage at least) by the time 2.11bsd shipped.

[jbglaw]

Just another thought: Maybe add an install target with the usual $prefix/$DESTDIR stuff? For the CI builds (http://toolchain.lug-owl.de/laminar/jobs/retrofuse), I keep the build artifacts. (Ie. the SIMH pdp11 used for testing within retro-fuse is an extracted tarball of my OpenSIMH CI builds.) Right now, I'm guesstimating the four FUSE binary names. The Makefile though has them all (with $(ALL_PROGS)), so that would be a no-brainer to implement?