Skip to content
This repository has been archived by the owner on Nov 19, 2017. It is now read-only.

gcc-aarch64 is not reproducible on Travis CI #1

Closed
ollieparanoid opened this issue Aug 5, 2017 · 7 comments
Closed

gcc-aarch64 is not reproducible on Travis CI #1

ollieparanoid opened this issue Aug 5, 2017 · 7 comments

Comments

@ollieparanoid
Copy link
Owner

It fails with:

(004383) [10:27:27] Compare: usr/bin/aarch64-alpine-linux-musl-c++
(004383) [10:27:27] CHALLENGE FAILED for usr/bin/aarch64-alpine-linux-musl-c++:Entry 'usr/bin/aarch64-alpine-linux-musl-c++' has a different type!
(004383) [10:27:27] Compare: usr/bin/aarch64-alpine-linux-musl-g++
(004383) [10:27:27] CHALLENGE FAILED for usr/bin/aarch64-alpine-linux-musl-g++:Entry 'usr/bin/aarch64-alpine-linux-musl-g++' has a different type!

Log:
https://api.travis-ci.org/jobs/261273294/log.txt?deansi=true

@ollieparanoid
Copy link
Owner Author

I've tried to add some more debug output in pmbootstrap, and rebuilt it.
Better than nothing I guess, but I need to figure out what 0 and 1 mean in this context.

Good news is, that I was able to reproduce the error with a fresh build.

(004331) [23:30:39] Compare: usr/bin/aarch64-alpine-linux-musl-c++
(004331) [23:30:39] NOTE: Type in /home/travis/.local/var/pmOS-verify-release-package/repo_staging/x86_64/g++-aarch64-6.4.0-r4.apk: b'1'
(004331) [23:30:39] NOTE: Type in /home/travis/.local/var/pmbootstrap/packages/x86_64/g++-aarch64-6.4.0-r4.apk: b'0'
(004331) [23:30:39] CHALLENGE FAILED for usr/bin/aarch64-alpine-linux-musl-c++:Entry 'usr/bin/aarch64-alpine-linux-musl-c++' has a different type!
(004331) [23:30:39] Compare: usr/bin/aarch64-alpine-linux-musl-g++
(004331) [23:30:39] NOTE: Type in /home/travis/.local/var/pmOS-verify-release-package/repo_staging/x86_64/g++-aarch64-6.4.0-r4.apk: b'0'
(004331) [23:30:39] NOTE: Type in /home/travis/.local/var/pmbootstrap/packages/x86_64/g++-aarch64-6.4.0-r4.apk: b'1'
(004331) [23:30:39] CHALLENGE FAILED for usr/bin/aarch64-alpine-linux-musl-g++:Entry 'usr/bin/aarch64-alpine-linux-musl-g++' has a different type!

Full log:
https://api.travis-ci.org/jobs/262865376/log.txt?deansi=true

@ollieparanoid ollieparanoid self-assigned this Aug 9, 2017
@ollieparanoid
Copy link
Owner Author

ollieparanoid commented Aug 9, 2017

Resolved mapping:

REGTYPE = b"0"                  # regular file
AREGTYPE = b"\0"                # regular file
LNKTYPE = b"1"                  # link (inside tarfile)
SYMTYPE = b"2"                  # symbolic link
CHRTYPE = b"3"                  # character special device
BLKTYPE = b"4"                  # block special device
DIRTYPE = b"5"                  # directory
FIFOTYPE = b"6"                 # fifo special device
CONTTYPE = b"7"                 # contiguous file

So we have a symlink "link (inside tarfile)" for that file in the version, that Travis builds, and a regular file in the version, that the pmbuilder builds. What the hell.

@ollieparanoid
Copy link
Owner Author

ollieparanoid commented Aug 10, 2017

More information with more debugging output and yet another rebuild.

004356) [00:28:36] Compare: usr/bin/aarch64-alpine-linux-musl-c++
(004356) [00:28:36] NOTE: usr/bin/aarch64-alpine-linux-musl-c++ in /home/travis/.local/var/pmOS-verify-release-package/repo_staging/x86_64/g++-aarch64-6.4.0-r4.apk:
?rwxr-xr-x root/root          0 2017-08-09 22:59:22 usr/bin/aarch64-alpine-linux-musl-c++ link to usr/bin/aarch64-alpine-linux-musl-g++ 
(004356) [00:28:36] NOTE: usr/bin/aarch64-alpine-linux-musl-c++ in /home/travis/.local/var/pmbootstrap/packages/x86_64/g++-aarch64-6.4.0-r4.apk:
?rwxr-xr-x root/root    5826904 2017-08-10 00:26:32 usr/bin/aarch64-alpine-linux-musl-c++ 
(004356) [00:28:36] CHALLENGE FAILED for usr/bin/aarch64-alpine-linux-musl-c++:Entry 'usr/bin/aarch64-alpine-linux-musl-c++' has a different type!
(004356) [00:28:36] Compare: usr/bin/aarch64-alpine-linux-musl-g++
(004356) [00:28:36] NOTE: usr/bin/aarch64-alpine-linux-musl-g++ in /home/travis/.local/var/pmOS-verify-release-package/repo_staging/x86_64/g++-aarch64-6.4.0-r4.apk:
?rwxr-xr-x root/root    5826904 2017-08-09 22:59:22 usr/bin/aarch64-alpine-linux-musl-g++ 
(004356) [00:28:36] NOTE: usr/bin/aarch64-alpine-linux-musl-g++ in /home/travis/.local/var/pmbootstrap/packages/x86_64/g++-aarch64-6.4.0-r4.apk:
?rwxr-xr-x root/root          0 2017-08-10 00:26:32 usr/bin/aarch64-alpine-linux-musl-g++ link to usr/bin/aarch64-alpine-linux-musl-c++ 
(004356) [00:28:36] CHALLENGE FAILED for usr/bin/aarch64-alpine-linux-musl-g++:Entry 'usr/bin/aarch64-alpine-linux-musl-g++' has a different type!

Full log

This means:

  • in staging build:
    • aarch64-alpine-linux-musl-c++ links to aarch64-alpine-linux-musl-g++
    • aarch64-alpine-linux-musl-g++ is the real binary
  • in Travis' build:
    • aarch64-alpine-linux-musl-g++ links to aarch64-alpine-linux-musl-c++
    • aarch64-alpine-linux-musl-c++ is the real binary

Apparently this is not directly an issue with GCC's build system, but with how tar packages hard-linked files. And easy workaround I can think of, is creating symbolic links instead of hardlinks - I'll try that.
Ideas for fixes:

  • make patch, that changes the hardlink into a symlink couldn't find where to do that in GCC's buildsystem
  • in package(), replace the hardlink with a symlink I would need to do that at the end of package(), which makes the aportgen code unnecessary complicated
  • try export LN="ln -s", because apparently they use $(LN) all the time instead of directly calling ln. So I would force all links to become symlinks. didn't work, at least the way I've tried it.

@ollieparanoid
Copy link
Owner Author

ollieparanoid commented Aug 11, 2017

I've come to the conclusion, that this could be a bug in busybox' tar. Next up: try the regular tar!
https://github.com/postmarketOS/pmbootstrap/commit/174b6a6bcf2f42b502b0c1171599919643fb0b10

@ollieparanoid
Copy link
Owner Author

Nope, abuild depends on tar! so it gets always installed, when building a package.

@ollieparanoid
Copy link
Owner Author

Finally some good news on this ticket. This is what I'm doing now:

  • rename package() to _package()
  • create my own package() function, that calls _package and replaces the hardlink with the symlink after the other package function went through

...and while I thought nothing changed at first, a closer look reveals in the latest log:

(004284) [19:54:03] Compare: usr/bin/aarch64-alpine-linux-musl-gcc
(004284) [19:54:03] NOTE: usr/bin/aarch64-alpine-linux-musl-gcc in /home/travis/.local/var/pmOS-verify-release-package/repo_staging/x86_64/gcc-aarch64-6.4.0-r4.apk:
?rwxr-xr-x root/root          0 2017-08-11 18:44:30 usr/bin/aarch64-alpine-linux-musl-gcc link to usr/bin/aarch64-alpine-linux-musl-gcc-6.4.0 
(004284) [19:54:03] NOTE: usr/bin/aarch64-alpine-linux-musl-gcc in /home/travis/.local/var/pmbootstrap/packages/x86_64/gcc-aarch64-6.4.0-r4.apk:
?rwxr-xr-x root/root     811272 2017-08-11 19:51:54 usr/bin/aarch64-alpine-linux-musl-gcc 
(004284) [19:54:03] CHALLENGE FAILED for usr/bin/aarch64-alpine-linux-musl-gcc:Entry 'usr/bin/aarch64-alpine-linux-musl-gcc' has a different type!

and

(004284) [19:54:03] Compare: usr/bin/aarch64-alpine-linux-musl-gcc-6.4.0
(004284) [19:54:03] NOTE: usr/bin/aarch64-alpine-linux-musl-gcc-6.4.0 in /home/travis/.local/var/pmOS-verify-release-package/repo_staging/x86_64/gcc-aarch64-6.4.0-r4.apk:
?rwxr-xr-x root/root     811272 2017-08-11 18:44:30 usr/bin/aarch64-alpine-linux-musl-gcc-6.4.0 
(004284) [19:54:03] NOTE: usr/bin/aarch64-alpine-linux-musl-gcc-6.4.0 in /home/travis/.local/var/pmbootstrap/packages/x86_64/gcc-aarch64-6.4.0-r4.apk:
?rwxr-xr-x root/root          0 2017-08-11 19:51:54 usr/bin/aarch64-alpine-linux-musl-gcc-6.4.0 link to usr/bin/aarch64-alpine-linux-musl-gcc 
(004284) [19:54:03] CHALLENGE FAILED for usr/bin/aarch64-alpine-linux-musl-gcc-6.4.0:Entry 'usr/bin/aarch64-alpine-linux-musl-gcc-6.4.0' has a different type!

So actually we have a different hardlink now, and replacing the hardlink does work after all!
To find all hardlinks, I've written a simple tar_find_hardlinks.py script. It turns out, that the gcc-6.4.0 hardlink is the last one.

Close to actually fixing this now!

@ollieparanoid
Copy link
Owner Author

It finally built! \o/
Now I only need to make the code pretty, and get this into pmbootstrap/master (with a PR).

samueldr pushed a commit to samueldr/pmaports that referenced this issue Nov 17, 2018
This fixes ollieparanoid/binary-package-repo#1

GCC generates hardlinks between files `A` and `B` in its `make install` step. The problem is, that `tar` randomly packages `A` as full binary, and links `B` to `A`, or the other way around! I was able to reproduce this issue consistently when re-building `gcc-aarch64` on Travis CI (interestingly, this did not appear for `gcc-armhf`).

The fix is, to delete `B` and create a symlink `B` that points to `A` instead.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant