multiple related updates: gnulib, coreutils, gettext-full, elfutils #14802
Conversation
github-actions
bot
added
build/scripts/tools
|
All the test failures are occurring during the Anyone understand why download from an upstream source isn't tried? |
|
A few of the last CI build errors are with I didn't see this in my local builds but managed to reproduce it (when enabling |
guidosarducci
force-pushed
the
master-update-dwarves-elfutils-gnulib
branch
from
88bf02a
to
ac5bf56
Compare
CI Check SummaryGood news, most all tests pass! The 3 remaining build errors are:
and Thoughts
Is it possible that package
@nbd168 and @mpratt14 Did either of you see similar warnings when you first added The other MacOS failure is with building I also noticed @feckert ran into a similar issue in #12565 . Could you share the underlying problem you needed to fix? |
|
the libtool error is fake, ignore it for now... I don't have a Mac so I can only offer advice for coreutils, which I expect will be the next thing to fail in host tools build once you figure out elfutils, based on #13132 feel free to pull my patch in the comments if you want and if it's discovered that we would still need it |
|
Hmm, how is the libtool error fake? Looking at https://github.com/openwrt/openwrt/actions/runs/8172980671/job/22344385380?pr=14802#step:6:87 it seems a real failure to build, with the same message as you/Florian saw before: Could you clarify what you mean? Also, since I haven't touched coreutils, how do you believe it might be related? |
|
no, the libtool error is not related to coreutils, but coreutils might be broken for mac build because of updating gnulib i don't know why the libtool error shows, but I have experienced it appear before and then later disappear after the other program's error is fixed. in particular, notice that before the output from the script which prints the 10 lines of verbose logs in the CI log, that there is only an "ERROR" line for elfutils |
|
by the way, you should stop referring to the issue #12565 as there was nothing wrong (per last comment) |
It's possible, but MacOS CI builds also failed before for a PR updating only In any case, I'll try building with the
What do think that suggests? I might not be seeing it...
According to @feckert the problem disappeared after reverting a local patch, so I wanted to understand what that was and if it might be related to the problem here. |
|
Thanks @guidosarducci for this PR. I ended up as elfutils maintainer by accident. I'm overdue to drop my name from the package. There are probably some package/libs/elfutils/ and tools/elfutils differences that should not exist. Both mostly have a common goal of buildling elfutils. The tools/elfutils/patches/100-portability.patch gives me creeps. It touches different not directly related aspects, including overlaps with package/libs/elfutils patches and many mac specific changes. It should be logically divided into smaller units and upstreamed as much as possible. However, it is a job for another PR. |
|
The amount of random this "fixes MacOS patches are crazy" |
just to be clear, the elfutils issue still needs fixing, I'm just thinking ahead he never pulled the patch that I linked in that PR
that the libtool error is fake |
|
I'd like to start looking at the elfutils macOS problem even though I don't have one, but the 10 last lines of output isn't enough to see the actual problem, it's just the harmless warning at the end... |
guidosarducci
force-pushed
the
master-update-dwarves-elfutils-gnulib
branch
from
ac5bf56
to
c14bf6a
Compare
@robimarko Yes, I agree and am not pleased. The addition of
@luizluca Well, thanks for sticking around this far, and I hope you can continue as maintainer. Too many foundational packages (e.g.
For goodness sake, yes! I'd really like to see the "gnulibification" of |
Some updates...I included the Regarding the So we'll need to look at the detailed build logs, which I'll include here to avoid CI logs disappearing: Some odd behaviour...libbpf / bpftoolAs I noted in #14802 (comment), the Could someone double check? Here are the logs for Is it possible that MacOSLooking at logs for Help?Could one of the members familiar with CI operations please double-check my findings and if true help understand why some |
|
@guidosarducci I was able to compile for both WRT3200ACM and TL-WR1043N/ND v4. I haven't tested either yet.
I tried building the Presently we need to fix the following components: fails with Xcode 15.3, workaround: revert to Xcode 15.2 version 5.38.2 fails to build, workaround: revert to 5.28.1 version 0.9.5 failed to build for WRT3200ACM, workaround: revert to 0.9.3 The rest builds correctly https://httpstorm.com/share/.openwrt/test/2024-03-12_xcode-15.3/1043nd.2024-03-12.04-world.txt |
so 15.3 is hopeless? all I see are headers that fail to include, no fixing that unless we store our own copy (thanks apple for making a closed source sdk) |
Finding solutions is much easier when you have a working environment as close to the broken one as possible.
Compatibility had been surprisingly good in the past few years. Though it took a lot of effort to reach that point, and yes they break things every few years. That Apple is highly poisonous. |
|
sure a patch fix is possible, the real problem is doing it ourselves is "unauthorized" just like how if I somehow downloaded it to my Linux machine that would be an "unauthorized" download we don't have a build target for xcode, it's just a random prerequisite like the fact that you need however for a local fix, you can try hunting down the bad header my main point is that the version release is hopeless not that the entire situation is :P |
[...SNIP...] @httpstorm Thanks Georgi for the offer to test further and sorry myself for the delayed response. I've been trying to build a MacOS container myself on a local build server (using Docker-OSX and your instructions) to recreate the tests but it's been very slow going and honestly quite frustrating. I did find a comment about reusing OpenWrt CI images from @ynezz , but no clue about a pre-built Macos image we can use. I suggest OpenWrt explain how or make one available for just this kind of troubleshooting. After looking at the CI check logs, I found this description of our MacOS CI image used, which is MacOS 12.7.3 (Monterrey) with Xcode 14.2. Are you able to set up this environment and try building this PR? In any case, from looking at the CI test logs it's clear there are bugs in our CI processes and/or our tools/toolchain Makefiles, which also affect other PRs. All the 3 CI check failures are either questionable or just wrong:
I think @nbd168 and @mpratt14 are familiar with Hopefully the above can be sorted quickly, then the only remaining hard item blocking this PR is getting openwrt/procd#6 merged. Thanks again everyone! |
|
if you're looking at GitHub logs then I'm not seeing what you're seeing, can you show what logs exactly? or are you seeing that it's still the old gnulib version when it runs? |
|
I already uploaded the full logs and specific examples earlier in #14802 (comment). To download for yourself, the MacOS tools build info is under https://github.com/openwrt/openwrt/actions/runs/8196599103/job/22417156024?pr=14802. Just expand the |
|
In case it is strictly necessary I can try installing macOS 12.7.3, but I'd rather avoid this if possible. It's my only computer. Current state of branch main 448b48c
More logs: same link as before. WRT3200ACM 448b48c (main) make toolchain/gcc/initial/{clean,compile} -j 16
make toolchain/gcc/final/{clean,compile} -j 16wrt3200acm.2024-03-17.01-world.txt x64 448b48c (main)
make toolchain/gcc/initial/{clean,compile} -j 1 V=s
make toolchain/gcc/final/{clean,compile} -j 1 V=s
make target/linux/{clean,compile} -j 1 V=sx64 @guidosarducci master-update-dwarves-elfutils-gnulib make tools/elfutils/{clean,compile} -j 1 V=s
make tools/coreutils/{clean,compile} -j 1 V=sx64.2024-03-16.01-elfutils-0.191.txt I tried reverting the patches related to these packages, but the old versions which build on main, fail on your branch. I would assume some of the other patches to be the cause. x64 448b48c (main) comparison: x64 448b48c (main) comparison:
cd /Volumes/x64/tools
tar xvf /Volumes/x64/openwrt/dl/coreutils-9.4.tar.xz
cd coreutils-9.4
./configure
make -j 16x64.2024-03-17.05-coreutils-9.3-host.txt |
robimarko
force-pushed
the
master-update-dwarves-elfutils-gnulib
branch
from
2d17547
to
c514051
Compare
|
GH changed their images to macOS 14 and ARM64, so lets see if our CI fails |
A funny bug was discovered where if the buildroot's path has the name of the build target within it, it will also be substituted along with the stampfile's name for each program, causing an attempt to touch a file in a directory that doesn't exist. ... ... touch: cannot touch '/Volumes/touch/openwrt/staging_dir/host/stamp/.touch_installed': No such file or directory touch: cannot touch '/Volumes/ln/openwrt/staging_dir/host/stamp/.ln_installed': No such file or directory touch: cannot touch '/Volumes/chown/openwrt/staging_dir/host/stamp/.chown_installed': No such file or directory make[2]: *** [Makefile:50: /Volumes/coreutils/openwrt/staging_dir/host/stamp/.coreutils_installed] Error 1 ... ... Split up the path with $(dir) and $(notdir) before substitution to fix the syntax. Reported-by: Georgi Valkov <gvalkov@gmail.com> Tested-by: Georgi Valkov <gvalkov@gmail.com> # MacOS Signed-off-by: Michael Pratt <mcpratt@pm.me> Signed-off-by: Tony Ambardar <itugrok@yahoo.com>
There are other wrapper scripts released with makeinfo like texi2pdf which are required by the build prerequisites of some tools, and have a similar purpose and usage. Let the makeinfo perl script handle all of these cases. It's worth mentioning that "texi2any" is the actual program and "makeinfo" is one of it's aliases. From upstream GNU: makeinfo: texi2any rm -f $@ -$(LN_S) texi2any $@ Signed-off-by: Michael Pratt <mcpratt@pm.me> Signed-off-by: Tony Ambardar <itugrok@yahoo.com>
Patches refreshed automatically. Tested-by: Georgi Valkov <gvalkov@gmail.com> # MacOS Signed-off-by: Tony Ambardar <itugrok@yahoo.com>
For modules that depend on the reallocarray module, like ialloc, xalloc, and safe-alloc, it was not possible to skip importing the reallocarray module as they all contained at least one function that called reallocarray() and would cause build failure if the host system didn't declare it. This upstreamable patch adds macros that toggle whether to define functions that depend on reallocarray() based on whether the reallocarray module is being imported. Tested-by: Georgi Valkov <gvalkov@gmail.com> # MacOS Signed-off-by: Michael Pratt <mcpratt@pm.me> Signed-off-by: Tony Ambardar <itugrok@yahoo.com>
The tdestroy() function, which is a GNU extension to the standard C library, is defined in gnulib in tsearch.c but is missing it's corresponding declaration in search.in.h by being completely missing... This patch is large but upstreamable, including all of the macros and conditionals and configure checks that upstream GNU would expect for portable support, like using the @@ placeholder/substitution method to determine whether or not to have declarations based on whether or not tdestroy() is already declared within the standard headers of the default include paths. There were also some typedefs and aliases missing, along with the warnings and preprocessor exceptions that need to be added for consistency with the usage of the rest of the functions in the files. Tested-by: Georgi Valkov <gvalkov@gmail.com> # MacOS Signed-off-by: Michael Pratt <mcpratt@pm.me> Signed-off-by: Tony Ambardar <itugrok@yahoo.com>
Release Announcement: https://savannah.gnu.org/news/?group_id=425 Refresh: - 200-libunistring-missing-link.patch Tested-by: Georgi Valkov <gvalkov@gmail.com> # MacOS Signed-off-by: Tony Ambardar <itugrok@yahoo.com>
Force bison to ignore the M4 environment variable and hardcode it to the locally built m4 during build operations using the relocatable path variable STAGING_DIR_HOST. This allows bison to continue to function while we are forcefully avoiding autoreconf and other autoconf and automake-like operations by giving a fake path to m4 with the M4 environment variable. The specific path can still be overridden independently from the environment within the line of invocation that runs bison by setting STAGING_DIR_HOST within the command, so document this in the help printout. Tested-by: Georgi Valkov <gvalkov@gmail.com> # MacOS Signed-off-by: Michael Pratt <mcpratt@pm.me> Signed-off-by: Tony Ambardar <itugrok@yahoo.com>
Update to latest stable release. Add configure option to disable support for the Year 2038 problem. (for now, as some versions of GCC do not yet support it) Syncing bootstrap script fails, backport an upstream patch which can be removed at next coreutils update. Several headers from the stable gnulib branch cause build failure because the changes in the imported versions are incompatible with the Makefile that gets generated for coreutils. This version of coreutils was released after being bootstrapped and autoreconf'ed with a significantly different version of gnulib compared to our local gnulib, so skip importing them (and restore the backup). While at it, organize restoring the originally shipped version of files into a Make foreach function. Refresh patch: - 000-bootstrap.patch New patch: - 001-bootstrap-sync.patch Link: https://lists.gnu.org/archive/html/coreutils/2023-08/msg00099.html Tested-by: Georgi Valkov <gvalkov@gmail.com> # MacOS Signed-off-by: Michael Pratt <mcpratt@pm.me> Signed-off-by: Tony Ambardar <itugrok@yahoo.com>
Update to latest stable release. The following commits in gnulib caused a conflict in locally bootstrapped coreutils with stable gnulib: 8f4b4e52c991de2233b471f8e35a068866b31f01 2749234203959df8d72cd8638d4e00a9fff450db A module (strftime) was marked deprecated and replaced by another module (nstrftime) in the version of gnulib that coreutils was released with compared to the stable branch that we use for importing. Conflicts from the previous version of coreutils are now gone, so other imported headers are now good. Refresh patch: - 000-bootstrap.patch Remove upstreamed patch: - 001-bootstrap-sync.patch Link: https://lists.gnu.org/archive/html/coreutils/2024-03/msg00132.html Tested-by: Georgi Valkov <gvalkov@gmail.com> # MacOS Signed-off-by: Michael Pratt <mcpratt@pm.me> Signed-off-by: Tony Ambardar <itugrok@yahoo.com>
Release Notes: https://sourceware.org/pipermail/elfutils-devel/2024q1/006876.html Manually refresh: - 100-portability.patch Change: - replace libgen.h with gnulib "dirname" module for compilation errors: In file included from ./../libdw/libdwP.h:38, from eblobjnote.c:42: /usr/include/libgen.h:35:9: error: attempt to use poisoned "basename" 35 | #define basename __xpg_basename | ^ Tested-by: Georgi Valkov <gvalkov@gmail.com> # MacOS Co-Developed-by: Nick Hainke <vincent@systemli.org> Signed-off-by: Nick Hainke <vincent@systemli.org> Signed-off-by: Tony Ambardar <itugrok@yahoo.com>
Instead of editing the SUBDIRS variable with a patch, it can be overriden at the end of the command line when invoking Make. This tool has a series of recursive Makefiles in each subdirectory, therefore SUBDIRS is set to a pattern of Make functions so that the result is variable depending on the current subdirectory that Make is being invoked in. It's not necessary to have gnulib-cache.m4 in EXTRA_DIST since we don't need to re-import after packaging this in the SDK, so get rid of the entire patch hunk for ./Makefile.am Tested-by: Georgi Valkov <gvalkov@gmail.com> # MacOS Signed-off-by: Michael Pratt <mcpratt@pm.me> Signed-off-by: Tony Ambardar <itugrok@yahoo.com>
Organize the Makefile lines involved in gnulib importing and its workarounds. It improves readability and keeps git history organized. Tested-by: Georgi Valkov <gvalkov@gmail.com> # MacOS Signed-off-by: Michael Pratt <mcpratt@pm.me> Signed-off-by: Tony Ambardar <itugrok@yahoo.com>
On macOS, stdlib.h in the standard include paths does not provide reallocarray() while both elfutils and gnulib do, however they are declared differently, leading to an error: ./system.h:101:1: error: static declaration of 'reallocarray' follows non-static declaration reallocarray (void *ptr, size_t nmemb, size_t size) A normal "configure && make" build cycle results in both declarations being enabled as a result of both elfutils and gnulib having completely separate configure checks where gnulib uses an internal placeholder symbol HAVE_REALLOCARRAY, and elfutils uses a standard autoconf macro HAVE_DECL_REALLOCARRAY. Fix this by excluding the import of the reallocarray module which causes gnulib checks in the configure stage to not even consider whether to declare reallocarray later on, so the decision is only between the standard include stdlib.h and the elfutils header. Tested-by: Georgi Valkov <gvalkov@gmail.com> # MacOS Signed-off-by: Michael Pratt <mcpratt@pm.me> Signed-off-by: Tony Ambardar <itugrok@yahoo.com>
A false tdestroy() function was added in order to make elfutils build on macOS again. A previous commit added declarations for a real version of tdestroy() into gnulib, which is already imported, as well as the preprocessor flags and the triggers for the Makefile.am conditional in order to include the source to be built. Tested-by: Georgi Valkov <gvalkov@gmail.com> # MacOS Signed-off-by: Michael Pratt <mcpratt@pm.me> Signed-off-by: Tony Ambardar <itugrok@yahoo.com>
Quilt refresh combined two sets of changes to the same file. The switch from using libgen.h to dirname.h because of function poisoning from gnulib's import of basename() was added as a new patch hunk instead of an edit to the original one. The original patch hunk was to fix build errors on an earlier version of elfutils before the "dirname" module was being imported to fix further build errors with the 0.191 version. Tested-by: Georgi Valkov <gvalkov@gmail.com> # MacOS Signed-off-by: Michael Pratt <mcpratt@pm.me> Signed-off-by: Tony Ambardar <itugrok@yahoo.com>
robimarko
force-pushed
the
master-update-dwarves-elfutils-gnulib
branch
from
c514051
to
1991bfb
Compare
|
I am counting on follow-up patches if required :) |
|
@robimarko Thanks for tackling this; we'll keep fingers crossed but hopefully not necessary after the time spent working on it. Could I trouble you to check on #15215 as well? That 6.6 update has been stalled for a month... |
|
@guidosarducci The discussion in it is still active |
|
@robimarko he's referring to the initial PR for that being stale which made him make this new one (referenced in the initial comment) |
|
@mpratt14 Well, that wasn't really clear to me. It is probably just a lack of people interested in that target |
|
Getting : When trying to compiling net/frr, does anyone have a clue?. Never happened before this elfutils upgrade. Have reported in here . |
|
Did you rebuild the tools and toolchain? |
|
Yes, twice, the last one is deleting all of the OpenWrt related directories and redo it again. The temporary ugly and really bad solution for now is adding "-Wl,--allow-multiple-definition" into frr cflags. |
|
I'm going to try to handle this today if I have the time and by the way, this is actually a side effect of some of the hacky things Felix did in the Makefile.am files for elfutils, and not actually gnulib's fault for once |
|
@mpratt14 We have been bisecting it, so please join us: |
Description
This series began as an update of
dwarvesand its related dependencies, several of which are out of date and just recently updated. It includestoolchain/musl 1.25, the latest stabletools/gnulib, bothtools/elfutilsandelfutilssynced to version 0.191, and adwarvesmaintainer updated missed when I created the package.Testing
Build and run tested using QEMU with malta/le64 and arm/32.
Reviewer Requests
@mpratt14 added host
gnuliband @nbd168 modified hostelfutilswithgnulibto enable MacOS support.@luizluca maintains package
elfutilsand @PolynomialDivision worked on anelfutils 0.190update in #13871.I only noticed writing this PR that @hnyman also opened a
muslupdate PR #14765, and ran into some of the same issues as me. However, in this PR I've worked to fix the issues related to the<string.h>vs<libgen.h>change rather than revert it.Related
Background on the
muslbasename()changes: https://git.musl-libc.org/cgit/musl/log/?qt=grep&q=basename.After reviewing our packages and feeds, there are only a small number of simple fixes needed:
procdis fixed by trace: use standard POSIX header for basename() procd#6rpcd-mod-luciis fixed by rpcd-mod-luci: use standard POSIX header for basename() luci#6968.zabbixis fixed by zabbix: zabbix_helper_mac80211.c: add <libgen.h> header packages#23599.