Tests: "expected" files vanish during test run

[oliverrahner]

While preparing test cases for #331, after running the testsuite using ./Build test on the current master, I noticed that some of the template files that the output files are checked against are deleted during the run.
As that makes the test cases not repeatable, I guess this is not the intended behavior?

The test summary:

Test Summary Report
-------------------
t/cfg-multi.t       (Wstat: 2048 Tests: 40 Failed: 8)
  Failed tests:  17-20, 33-36
  Non-zero exit status: 8
t/cfg-single.t      (Wstat: 1024 Tests: 52 Failed: 4)
  Failed tests:  17-20
  Non-zero exit status: 4
t/cfg-split.t       (Wstat: 6144 Tests: 48 Failed: 24)
  Failed tests:  17-36, 45-48
  Non-zero exit status: 24
Files=29, Tests=352, 91 wallclock secs ( 0.93 usr  0.19 sys + 78.09 cusr 19.62 csys = 98.83 CPU)
Result: FAIL
Failed 3/29 test programs. 36/352 subtests failed.

git status:

On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        deleted:    t/cfg/multiple-newstr/multiple.man.de.1
        deleted:    t/cfg/multiple-newstr/multiple.man.es.1
        deleted:    t/cfg/multiple-newstr/multiple.man.fr.1
        deleted:    t/cfg/multiple-newstr/multiple.man.it.1
        deleted:    t/cfg/multiple-uptodate/multiple.man.de.1
        deleted:    t/cfg/multiple-uptodate/multiple.man.es.1
        deleted:    t/cfg/multiple-uptodate/multiple.man.fr.1
        deleted:    t/cfg/multiple-uptodate/multiple.man.it.1
        deleted:    t/cfg/single-uptodate/single-uptodate.man.fr.1
        deleted:    t/cfg/split-fuzzied-noup/first.man.de
        deleted:    t/cfg/split-fuzzied-noup/first.man.fr
        deleted:    t/cfg/split-fuzzied-noup/second.man.de
        deleted:    t/cfg/split-fuzzied-noup/second.man.fr
        deleted:    t/cfg/split-fuzzied/first.man.de
        deleted:    t/cfg/split-fuzzied/first.man.fr
        deleted:    t/cfg/split-fuzzied/second.man.de
        deleted:    t/cfg/split-fuzzied/second.man.fr
        deleted:    t/cfg/split-fuzzy/first.man.de
        deleted:    t/cfg/split-fuzzy/first.man.fr
        deleted:    t/cfg/split-fuzzy/second.man.de
        deleted:    t/cfg/split-fuzzy/second.man.fr
        deleted:    t/cfg/split-multimaster/first.man.de
        deleted:    t/cfg/split-multimaster/first.man.fr
        deleted:    t/cfg/split-multimaster/second.man.de
        deleted:    t/cfg/split-multimaster/second.man.fr
        deleted:    t/cfg/split-multimaster/third.man.de
        deleted:    t/cfg/split-multimaster/third.man.fr
        deleted:    t/cfg/split-uptodate/first.man.de
        deleted:    t/cfg/split-uptodate/first.man.fr
        deleted:    t/cfg/split-uptodate/second.man.de
        deleted:    t/cfg/split-uptodate/second.man.fr
        deleted:    t/cfg/split-yaml/_index.vi.md
        deleted:    t/cfg/split-yaml/vi.yaml

no changes added to commit (use "git add" and/or "git commit -a")

[mquinson]

Hello,

I am really surprised about that, as I usually run several tests back to back on my local machine.

The only issue I've noticed is that if you interrupt the tests before their end, you'll have permission errors, as the test suite is running chmod 444 a lot of files and directories during the test to make them read-only. It even marks a bunch of directories as unusable with chmod 0. The intention here is exactly to test that what you describe does not happen. Tests should not modify files out of their target directory in any way, and should not even read where it's not supposed to read. This is why I would appreciate if you could dig further on your side to see if you can understand what's going on.

By the way, if the permission issue occures to you, you want to use the little fix_perms.sh script that is in charge of marking the files read/write again.

Thanks for reporting. I'm interested in the outcome of your exploration here.

[oliverrahner]

So, the first-level cause is that I was maybe running the tests as root... eh... hehe... yeah.
When I run it as a regular user, the previously failing test cases pass.

That still leaves us to guess what the root cause is, but I don't have more time to spend on this unfortunately.
To be honest, the whole stuff with reducing permissions during the test cases feels kinda hacky.

I will continue creating testcases for my PR.