gcc-aarch64 is not reproducible on Travis CI

[ollieparanoid]

It fails with:

(004383) [10:27:27] Compare: usr/bin/aarch64-alpine-linux-musl-c++
(004383) [10:27:27] CHALLENGE FAILED for usr/bin/aarch64-alpine-linux-musl-c++:Entry 'usr/bin/aarch64-alpine-linux-musl-c++' has a different type!
(004383) [10:27:27] Compare: usr/bin/aarch64-alpine-linux-musl-g++
(004383) [10:27:27] CHALLENGE FAILED for usr/bin/aarch64-alpine-linux-musl-g++:Entry 'usr/bin/aarch64-alpine-linux-musl-g++' has a different type!

Log:
https://api.travis-ci.org/jobs/261273294/log.txt?deansi=true

[ollieparanoid]

I've tried to add some more debug output in pmbootstrap, and rebuilt it.
Better than nothing I guess, but I need to figure out what 0 and 1 mean in this context.

Good news is, that I was able to reproduce the error with a fresh build.

(004331) [23:30:39] Compare: usr/bin/aarch64-alpine-linux-musl-c++
(004331) [23:30:39] NOTE: Type in /home/travis/.local/var/pmOS-verify-release-package/repo_staging/x86_64/g++-aarch64-6.4.0-r4.apk: b'1'
(004331) [23:30:39] NOTE: Type in /home/travis/.local/var/pmbootstrap/packages/x86_64/g++-aarch64-6.4.0-r4.apk: b'0'
(004331) [23:30:39] CHALLENGE FAILED for usr/bin/aarch64-alpine-linux-musl-c++:Entry 'usr/bin/aarch64-alpine-linux-musl-c++' has a different type!
(004331) [23:30:39] Compare: usr/bin/aarch64-alpine-linux-musl-g++
(004331) [23:30:39] NOTE: Type in /home/travis/.local/var/pmOS-verify-release-package/repo_staging/x86_64/g++-aarch64-6.4.0-r4.apk: b'0'
(004331) [23:30:39] NOTE: Type in /home/travis/.local/var/pmbootstrap/packages/x86_64/g++-aarch64-6.4.0-r4.apk: b'1'
(004331) [23:30:39] CHALLENGE FAILED for usr/bin/aarch64-alpine-linux-musl-g++:Entry 'usr/bin/aarch64-alpine-linux-musl-g++' has a different type!

Full log:
https://api.travis-ci.org/jobs/262865376/log.txt?deansi=true

[ollieparanoid]

Resolved mapping:

REGTYPE = b"0"                  # regular file
AREGTYPE = b"\0"                # regular file
LNKTYPE = b"1"                  # link (inside tarfile)
SYMTYPE = b"2"                  # symbolic link
CHRTYPE = b"3"                  # character special device
BLKTYPE = b"4"                  # block special device
DIRTYPE = b"5"                  # directory
FIFOTYPE = b"6"                 # fifo special device
CONTTYPE = b"7"                 # contiguous file

So we have a symlink "link (inside tarfile)" for that file in the version, that Travis builds, and a regular file in the version, that the pmbuilder builds. What the hell.

[ollieparanoid]

More information with more debugging output and yet another rebuild.

004356) [00:28:36] Compare: usr/bin/aarch64-alpine-linux-musl-c++
(004356) [00:28:36] NOTE: usr/bin/aarch64-alpine-linux-musl-c++ in /home/travis/.local/var/pmOS-verify-release-package/repo_staging/x86_64/g++-aarch64-6.4.0-r4.apk:
?rwxr-xr-x root/root          0 2017-08-09 22:59:22 usr/bin/aarch64-alpine-linux-musl-c++ link to usr/bin/aarch64-alpine-linux-musl-g++ 
(004356) [00:28:36] NOTE: usr/bin/aarch64-alpine-linux-musl-c++ in /home/travis/.local/var/pmbootstrap/packages/x86_64/g++-aarch64-6.4.0-r4.apk:
?rwxr-xr-x root/root    5826904 2017-08-10 00:26:32 usr/bin/aarch64-alpine-linux-musl-c++ 
(004356) [00:28:36] CHALLENGE FAILED for usr/bin/aarch64-alpine-linux-musl-c++:Entry 'usr/bin/aarch64-alpine-linux-musl-c++' has a different type!
(004356) [00:28:36] Compare: usr/bin/aarch64-alpine-linux-musl-g++
(004356) [00:28:36] NOTE: usr/bin/aarch64-alpine-linux-musl-g++ in /home/travis/.local/var/pmOS-verify-release-package/repo_staging/x86_64/g++-aarch64-6.4.0-r4.apk:
?rwxr-xr-x root/root    5826904 2017-08-09 22:59:22 usr/bin/aarch64-alpine-linux-musl-g++ 
(004356) [00:28:36] NOTE: usr/bin/aarch64-alpine-linux-musl-g++ in /home/travis/.local/var/pmbootstrap/packages/x86_64/g++-aarch64-6.4.0-r4.apk:
?rwxr-xr-x root/root          0 2017-08-10 00:26:32 usr/bin/aarch64-alpine-linux-musl-g++ link to usr/bin/aarch64-alpine-linux-musl-c++ 
(004356) [00:28:36] CHALLENGE FAILED for usr/bin/aarch64-alpine-linux-musl-g++:Entry 'usr/bin/aarch64-alpine-linux-musl-g++' has a different type!

Full log

This means:

• in staging build:
  • aarch64-alpine-linux-musl-c++ links to aarch64-alpine-linux-musl-g++
  • aarch64-alpine-linux-musl-g++ is the real binary
• in Travis' build:
  • aarch64-alpine-linux-musl-g++ links to aarch64-alpine-linux-musl-c++
  • aarch64-alpine-linux-musl-c++ is the real binary

Apparently this is not directly an issue with GCC's build system, but with how tar packages hard-linked files. And easy workaround I can think of, is creating symbolic links instead of hardlinks - I'll try that.
Ideas for fixes:

• make patch, that changes the hardlink into a symlink couldn't find where to do that in GCC's buildsystem
• in package(), replace the hardlink with a symlink I would need to do that at the end of package(), which makes the aportgen code unnecessary complicated
• try export LN="ln -s", because apparently they use $(LN) all the time instead of directly calling ln. So I would force all links to become symlinks. didn't work, at least the way I've tried it.

[ollieparanoid]

I've come to the conclusion, that this could be a bug in busybox' tar. Next up: try the regular tar!
https://github.com/postmarketOS/pmbootstrap/commit/174b6a6bcf2f42b502b0c1171599919643fb0b10

[ollieparanoid]

Nope, abuild depends on tar! so it gets always installed, when building a package.

[ollieparanoid]

Finally some good news on this ticket. This is what I'm doing now:

• rename package() to _package()
• create my own package() function, that calls _package and replaces the hardlink with the symlink after the other package function went through

...and while I thought nothing changed at first, a closer look reveals in the latest log:

(004284) [19:54:03] Compare: usr/bin/aarch64-alpine-linux-musl-gcc
(004284) [19:54:03] NOTE: usr/bin/aarch64-alpine-linux-musl-gcc in /home/travis/.local/var/pmOS-verify-release-package/repo_staging/x86_64/gcc-aarch64-6.4.0-r4.apk:
?rwxr-xr-x root/root          0 2017-08-11 18:44:30 usr/bin/aarch64-alpine-linux-musl-gcc link to usr/bin/aarch64-alpine-linux-musl-gcc-6.4.0 
(004284) [19:54:03] NOTE: usr/bin/aarch64-alpine-linux-musl-gcc in /home/travis/.local/var/pmbootstrap/packages/x86_64/gcc-aarch64-6.4.0-r4.apk:
?rwxr-xr-x root/root     811272 2017-08-11 19:51:54 usr/bin/aarch64-alpine-linux-musl-gcc 
(004284) [19:54:03] CHALLENGE FAILED for usr/bin/aarch64-alpine-linux-musl-gcc:Entry 'usr/bin/aarch64-alpine-linux-musl-gcc' has a different type!

and

(004284) [19:54:03] Compare: usr/bin/aarch64-alpine-linux-musl-gcc-6.4.0
(004284) [19:54:03] NOTE: usr/bin/aarch64-alpine-linux-musl-gcc-6.4.0 in /home/travis/.local/var/pmOS-verify-release-package/repo_staging/x86_64/gcc-aarch64-6.4.0-r4.apk:
?rwxr-xr-x root/root     811272 2017-08-11 18:44:30 usr/bin/aarch64-alpine-linux-musl-gcc-6.4.0 
(004284) [19:54:03] NOTE: usr/bin/aarch64-alpine-linux-musl-gcc-6.4.0 in /home/travis/.local/var/pmbootstrap/packages/x86_64/gcc-aarch64-6.4.0-r4.apk:
?rwxr-xr-x root/root          0 2017-08-11 19:51:54 usr/bin/aarch64-alpine-linux-musl-gcc-6.4.0 link to usr/bin/aarch64-alpine-linux-musl-gcc 
(004284) [19:54:03] CHALLENGE FAILED for usr/bin/aarch64-alpine-linux-musl-gcc-6.4.0:Entry 'usr/bin/aarch64-alpine-linux-musl-gcc-6.4.0' has a different type!

So actually we have a different hardlink now, and replacing the hardlink does work after all!
To find all hardlinks, I've written a simple tar_find_hardlinks.py script. It turns out, that the gcc-6.4.0 hardlink is the last one.

Close to actually fixing this now!

[ollieparanoid]

It finally built! \o/
Now I only need to make the code pretty, and get this into pmbootstrap/master (with a PR).