macOS generates ._ files that cause unwanted checkbuild complaints

[masukomi]

given this in the resources section of META6.json

  "resources": [
    "data/hey/hey.db"
  ],

and this resources dir tree

❯ tree resources/
resources/
└── data
    └── hey
        └── hey.db

2 directories, 1 file

I get this error on upload

❯ fez upload
>>= Looking in "sdist/W6WQVrhA.tar.gz" for META6.json
>>= meta<provides> looks OK
=<< File "dist/resources/data/hey/._hey.db" in tar not found in meta<resources>
>>= Hey:ver<1.0.0-beta.6>:auth<zef:masukomi> could use some sprucing up
>>= Upload anyway (y/N)? N
=<< Ok, exiting

Note the ._ that prefaces hey.db That shouldn't be there.

note that there is no hidden file in that directory

❯ ls -lash resources/data/hey/
total 128
  0 drwxr-xr-x  3 masukomi  staff    96B Dec 17 15:27 ./
  0 drwxr-xr-x  3 masukomi  staff    96B Dec 17 15:27 ../
128 -rw-r--r--@ 1 masukomi  staff    64K Dec 17 15:27 hey.db

To test/replicate:
[edit deleted replication instructions because they wouldn't work for other people. see new instructions in comments below]

❯ fez --version
>>= fez version: 38

[masukomi]

the problematic file ._hey.db

is coming into existence here

#Pax.rakumod...
method ls($file) {
  my $p = run 'pax', '-c', '-z', '-f', $file, :out, :err;

specifically, it's present in the $p.out.lines

if i then run pax -c -z -f <generated .tar.gz file in sdist>

I also see the ._hey.db file.

❯ pax -c -z -f sdist/bggtbGM0.tar.gz
dist/META6.json
dist/LICENSE
dist/bin/hey
dist/Changes
dist/resources/data/hey/._hey.db
dist/resources/data/hey/hey.db
...

BUT
If i decompress the same file with tar -xzvf and then inspect that directory...

❯ ls -las dist/resources/data/hey/
total 128
  0 drwxr-xr-x  3 masukomi  staff     96 Dec 20 22:39 ./
  0 drwxr-xr-x  3 masukomi  staff     96 Dec 20 22:39 ../
128 -rw-r--r--@ 1 masukomi  staff  65536 Dec 17 15:27 hey.db

So. it looks like this is either a bug in pax or an aspect of its output that I don't understand.

EDIT: the problem is that tar and ls both lie on macOS. See the next comment on this ticket

Fez is correctly complaining about that file missing from the meta6. However, it's still a problem for fez in that that the chosen tool for inspecting the tar file isn't producing the expected results.

I believe the correct solution here is to use tar not pax for creating the tar and for inspecting it.

If i run tar -ztvf <generated tar.gz file> to list the contents of the tar without extracting it, I get the expected result with no phantom ._hey.db file listed.

Related bug in fez though:

If i run
fez checkbuild directly it says everything is fine. It's only when run via fez upload that there's a problem.

This means that the checkbuild isn't performing a "valid" test. In that it doesn't produce the same warnings as the checkbuild that's run prior to upload.

if no-one gets to it before me I guess i'll file that as a separate bug.

EDIT: new ticket for that problem here

---

note. the following discoveries were made against fez 37. The release is fez 38 but that doesn't seem to be in this repo. There are also no tags in this repo which is going to make it very difficult to find out when a bug was introduced in the future.

[masukomi]

News from the internet:

• It's a macOS thing.
• those files really exist
• tar is lying because apple has hacked their implementation of it to do so.
• ls is also lying, because apple has hacked their implementation of it to do so (presumed)

Very nice Stack Exchange answer here explaining what those files are.

With that being said... it's still a problem.

3 potential solutions:

1. switch to tar
2. exclude files starting with ._ when there is another file in the list with the same name but without the ._ preface.
3. potentially delete the files before compressing. I'm seeing conflicting info if this is possible. Obviously don't do it in the original directory.

Two sounds easy, but i think it's dangerous. It seems easy to screw up the handling of the edge case of a user actually having a file that starts with ._, or worse ._._, or even worse IS ._ with nothing after it.

Three is probably the best solution, but if it doesn't work I'm inclined to suggest we switch to tar and just add footnote to the docs indicating that on macOS there may be some ._ files that come along for the ride.

I'm very dissatisfied by this answer because I really don't like using tools that lie to me about directory contents but I'm not seeing a better practical solution.

Non solutions:

implementing tar in raku

Not a solution because including those files is actually "correct" behavior from the perspective of tar. The problem is that tar is lying when it claims they aren't there. The only thing we'd get is a version of tar that doesn't lie when examining the contents. We don't really need that because apparently pax doesn't lie.

[tony-o]

thank you for the detailed report, will find my macbook and fix this in the next few days

as a work around, does adding the following key:value json to your ~/.fez-config.json work? "bundlers": ["Fez::Util::Tar"] change your results at all? This should, effectively, alter fez to only using tar/gzip

[masukomi]

my replication instructions were flawed and wouldn't work for others. the following should work

first, add extended attributes to a file in resources

xattr -w feztest 1 resources/path/to/resource.foo

second, abuse ditto to convert them to an AppleDouble file (._ prefixed)

ditto -c -k resources/path/to/resource.foo archive.zip
unzip -o archive.zip # you should see the new ._resource.foo listed in the output
rm archive.zip

now you should have an AppleDouble file that "should" error out in the checkbuild process with pax. I say "should" because i can't actually test it because there's a new problem.... (see below)

as a work around, does adding the following key:value json to your ~/.fez-config.json work? "bundlers": ["Fez::Util::Tar"]

I can't say definitively. Since filing this ticket I've upgraded to Ventura from Monterrey and now I can't get it to EVER use pax. And now we have 2 new issues:

1. can't use pax AT ALL
2. tar is unreasonably slow. ("is fez broken?!?!" levels of slow)

the for @chandlers -> $h { loop in Bundle.rakumod is nexting before adding pax to to @handlers because ::("$h").able returns False

What i can say is that it is using tar and it is not causing an error (not surprising since it's just a symlink to bsdtar which makes everything think the AppleDouble files don't exist) HOWEVER IT IS HORRIBLY SLOW.

How slow?

________________________________________________________
Executed in  782.90 secs    fish           external
   usr time  746.75 secs    0.13 millis  746.75 secs
   sys time   35.31 secs    3.39 millis   35.31 secs

746 seconds == 12 minutes. ~99% of that is in Fez::Util::Tar.bundle

[tony-o]

I can't seem to reproduce with tar or pax:

~/o/xxx xattr -w feztest 1 resources/test.xxx
~/o/xxx ditto -c -k resources/test.xxx archive.zip
~/o/xxx unzip -o archive.zip
Archive:  archive.zip
  inflating: test.xxx
  inflating: ._test.xxx
~/o/xxx fez upload
>>= Looking in "sdist/Yti512YR.tar.gz" for META6.json
>>= meta<provides> looks OK
>>= meta<resources> looks OK
>>= XXX:ver<1.0.0>:auth<zef:tony-o> looks OK
=<< Key is expired, please login:
>>= Username: ^C
~/o/xxx vim sdist/Yti512YR.tar.gz
~/o/xxx cat META6.json
{
  "name":"XXX",
  "auth": "zef:tony-o",
  "version": "1.0.0",
  "api": 1,
  "provides":{
    "XXX": "lib/XXX.rakumod"
  },
  "resources":[
    "test.xxx"
  ]
}
~/o/xxx tar tvf sdist/Yti512YR.tar.gz
-rw-r--r--  0 tonyo  staff     162 Jan  4 19:32 xxx/META6.json
-rw-r--r--  0 tonyo  staff       9 Jan  4 19:32 xxx/resources/test.xxx
-rw-r--r--  0 tonyo  staff      53 Dec  8 07:50 xxx/lib/XXX.rakumod
-rw-r--r--  0 tonyo  staff       9 Jan  4 19:32 xxx/test.xxx
-rw-r--r--  0 tonyo  staff     372 Jan  4 19:33 xxx/archive.zip

In this case it seems to just ignore the the double file

[tony-o]

@masukomi are you able to provide more insight to this issue?

[sdondley]

I'm having an issue that may be related. It occurs when I try to release a module with mi6 which uses fez: https://stackoverflow.com/questions/75190418/no-single-root-directory-found-all-dists-must-extract-to-a-single-directory-e

Oddly, it worked fine on my m1 mac.

[2colours]

Just to clarify: we did try to nail that issue down and we did reach to the odd "._"-prefixed file returned by the pax command, so it's not a random guess in that sense.