ksh cannot be successfully built with macOS Command Line Tools 14.2 #596
Comments
|
Something is definitely weird here, I can't build @4ec93745 either, but that's the commit that's RUNNING right now... |
|
Ok, is #567 related? |
|
The compiler works, I can build other non-trivial things, like |
|
Cannot reproduce. Everything builds correctly on my end. I have no idea what Do you have Homebrew installed by any chance? |
|
Homebrew? Ugh. No. Definitely not. I do have MacPorts, but I've always had MacPorts. |
|
I had not tried to build u+m on 12.6.2 before opening this ticket. The last time I built u+m was 10/31, which would have been on 12.6.1 or maybe 12.6, I don't remember exactly when I installed 12.6.1. There's been some churn on the M1 machine, but other than installing OS updates and re-running |
|
The
|
|
When compiling on Debian Linux, the I can spin up an Intel macOS VM, but I don't have an easy way to spin up an arm64 VM for testing. |
|
The How does one get |
EDIT: BSD message digest functions, is that why macOS thinks it's there? |
|
FYI, I successfully compiled and tested current HEAD on all of my ARM based Unix boxes without any errors which include Debian, Ubuntu, and FreeBSD. I also tested my macOS 10.13.6 x86_64 laptop as well with success. Sorry, I just do not have an Apple M1 to test with. |
|
@hyenias If you would, could you check a representative sampling of them and see how many of them (if any of them) have an affirmative detection for Thanks. EDIT: You may have to set IFFEFLAGS to -d1 or -d2 in order to see the test. |
|
Since your referenced library libmd is for BSD systems, I checked my two BSD type systems. I did not have to include macOS Intel 10.13.6: FreeBSD 13.1 arm64: |
I see that one of yours tests for it and apparently doesn't find it, and the other tests for it and finds it. On macOS here, it tests for it twice, and apparently finds it twice despite it not being there, which causes the failure when it tries to link against it. I didn't think If you use |
|
That was a harder file to read. Here you go for what it is worth because it does not make any sense to me. ...
|
|
On a brand new macOS 12.4 VM with no other software than the CLT installed: CLT 13.4.0.0.1.1651278267 builds current HEAD successfully. Compiling on the same VM, CLT 13.4 correctly determines that CLT 13.4 fails the link step in the CLT 14.2 succeeds in the link step when including The @pghvlaans Do you know what compiler was producing the error in #567 ? Edit... all tests performed on M1Pro MacBookPro. Edit2... CLT 13.4 doesn't actually perform the second test at |
|
I removed CLT 14.2 from both of my production Mac and reinstalled 13.4, and HEAD builds without error on both without any other changes. |
posguy99
changed the title
|
@McDutchie Here are regular |
|
Thanks. I'll try to get to this as soon as I can. Life has been getting in the way. :/ |
|
On my M1 with macOS 13 and CTL 14.2, I have I don't have CLT 13.4 handy to test, but was this file added with CLT 14? I still have my 2015 MacBook with 12.6 that I can pull out and see if I can reproduce it there. |
|
CLT 13.x does not ship the macOS 13 SDK, it was added in CLT 14. So in macOS 13.x, libmd must be in the dylib cache. I don't have a macOS 13 machine to be able to look. You can build with CLT 14.2 on macOS 13? |
|
If you want to look, supposedly https://github.com/keith/dyld-shared-cache-extractor is the tool to use. |
|
@tvalenta For some reason, I can't create a macOS13.x VM. On my M1Pro, macOS 12 wants to download an update in order for UTM to create a macOS 13 VM. I think it wants to upgrade to 13.x, and I’m not about to do that. |
|
On my 2015 MacBook (with the way nicer keyboard), I'm running macOS 12.6.3. https://github.com/apple-oss-distributions/libmd shows activity for libmd that coincides with macOS 13.0. With installation of XCode CLT 14.2, the SDK defaults to SDK version 13.1, which includes the file On my M1 MacBook running 13.x, I have no trouble building ksh93 with CLT 14.2, using the MacOS 13.1 SDK. I haven't dumped the dylib cache, but minimal tests indicate that libmd.dylib is indeed in the OS. I don't have an Intel MacBook that can run Ventura handy, but I can get my hands on one and will see if I can find the library there, too. On my Intel MacBook with 12.6.3, I have some of the older SDKs (11.3 and 12.3). If I run Running Based on https://firefox-source-docs.mozilla.org/widget/cocoa/sdks.html, I tried Could it be entirely serendipitous that ksh compiles at all on macOS 13.0? |
|
OK, while I was not able to create a macOS 13.x VM cold, I created a 12.4 VM and upgraded it to 13.2, which worked. And sure enough, on macOS 13.2, using CLT 14.2, I can confirm as well that ksh compiles without error. |
|
This makes me suspect that, for at least the past decade, macOS builds haven't uhh included libcmd and libsum, so would anyone notice if it was entirely excluded in a build?
I could dig deeper but not tonight.
…On February 4, 2023 7:20:10 PM PST, Marc Wilson ***@***.***> wrote:
OK, while I was not able to create a macOS 13.x VM cold, I created a 12.4 VM and upgraded it to 13.2, which worked.
And sure enough, on macOS 13.2, using CLT 14.2, I can confirm as well that ksh compiles without error.
--
Reply to this email directly or view it on GitHub:
#596 (comment)
You are receiving this because you were mentioned.
Message ID: ***@***.***>
|
|
@McDutchie Are the two Mamfile patches from here https://github.com/apple-oss-distributions/ksh/tree/ksh-30/patches relevant? I will have to spin a new 12.x VM with CLT 14.2, but I figured Apple had to get around this problem somehow since macOS 13 still distributes Removing |
|
Ok, yes, applying both src__lib__libcmd__Mamfile.diff and src__lib__libsum__Mamfile.diff allows So that's how Apple must have gotten around the build failure. |
|
Here are the two patches. |
|
Thanks for finding that. Now I just need to test that those patches don't break the build on any other system. |
|
@hyenias You have at least one system that actually ships |
|
Applying the patch on FreeBSD doesn't break the build, but also produces a ksh binary that is not linked against libmd. I don't know what the ramifications of that are. |
|
I don't claim to know what |
|
@McDutchie Random question... why does the |
|
So, I don't understand: I have, in fact, been building on macOS 12.6.3 with Command Line Tools 14.2, without the patch, and it's always been building fine. I have no idea how to reproduce your failure. I really don't want to blindly apply a patch without having a clue what's going on. Perhaps seeing the exact compiler commands will provide some sort of clue. I've pushed a commit (8f37528) that makes iffe show these with IFFEFLAGS=-d1. Could you reproduce the failure with that commit and paste the debug output with the compiler command lines here? |
|
That's CLT 13.4 and CLT 14.2 building on macOS 12.4. Would the patch provide any more info than that? Do you want me to do both a build with 13.4 and with 14.2? Edit... to be clear, while it happens with the production machines, every time I test, I re-spin the VM(s), and the only thing in the VM(s) is the CLT. No other software is installed. And @tvalenta gets the same results. |
|
After installing the incredible bloatware that is the XCode graphical interface, I can now reproduce the build error. |
|
After installing XCode.app, I can reproduce it even with a simple test program: So, this is not our bug. XCode breaks the system. |
Which is weird since I have never had XCode installed on my M1, and never had it installed in any of the VMs I've been testing in. |
Agreed. As @tvalenta pointed out, the stub that the macOS 13 SDK installs fools the lazy linker into thinking the library is there, but it isn't in the dylib cache on macOS 12, so at runtime it goes looking for it and fails. Can it be worked around, though? Can |
|
The workaround should be in the same Mamfiles that Apple patched, but I'm not happy with simply deleting the test for libmd. It should do what you say -- actually attempt to execute the test programs, instead of being satisfied with just being able to compile them. The patch below should accomplish this (by adding Patch v1diff --git a/src/lib/libcmd/Mamfile b/src/lib/libcmd/Mamfile
index 5aa4e2478..540dbab63 100644
--- a/src/lib/libcmd/Mamfile
+++ b/src/lib/libcmd/Mamfile
@@ -39,8 +39,8 @@ make install
exec - esac
exec - continue
exec - elif test ! -f ${INSTALLROOT}/lib/lib$i.a
- exec - then case `{ ${CC} ${mam_cc_FLAGS} ${CCFLAGS} -L${INSTALLROOT}/lib ${LDFLAGS} -o 1.${COTEMP}.x 1.${COTEMP}.o -l$i 2>&1 || echo '' "$x" ;} | sed -e 's/[][()+@?]/#/g' || :` in
- exec - *$x*) case `{ ${CC} ${mam_cc_FLAGS} ${CCFLAGS} ${LDFLAGS} -o 1.${COTEMP}.x 1.${COTEMP}.o -l$i 2>&1 || echo '' "$x" ;} | sed -e 's/[][()+@?]/#/g' || :` in
+ exec - then case `{ ${CC} ${mam_cc_FLAGS} ${CCFLAGS} -L${INSTALLROOT}/lib ${LDFLAGS} -o 1.${COTEMP}.x 1.${COTEMP}.o -l$i 2>&1 && ./1.${COTEMP}.x || echo '' "$x" ;} | sed -e 's/[][()+@?]/#/g' || :` in
+ exec - *$x*) case `{ ${CC} ${mam_cc_FLAGS} ${CCFLAGS} ${LDFLAGS} -o 1.${COTEMP}.x 1.${COTEMP}.o -l$i 2>&1 && ./1.${COTEMP}.x || echo '' "$x" ;} | sed -e 's/[][()+@?]/#/g' || :` in
exec - *$x*) continue ;;
exec - esac
exec - ;;
diff --git a/src/lib/libsum/Mamfile b/src/lib/libsum/Mamfile
index ed011c588..27048ef66 100644
--- a/src/lib/libsum/Mamfile
+++ b/src/lib/libsum/Mamfile
@@ -39,8 +39,8 @@ make install
exec - esac
exec - continue
exec - elif test ! -f ${INSTALLROOT}/lib/lib$i.a
- exec - then case `{ ${CC} ${mam_cc_FLAGS} ${CCFLAGS} -L${INSTALLROOT}/lib ${LDFLAGS} -o 1.${COTEMP}.x 1.${COTEMP}.o -l$i 2>&1 || echo '' "$x" ;} | sed -e 's/[][()+@?]/#/g' || :` in
- exec - *$x*) case `{ ${CC} ${mam_cc_FLAGS} ${CCFLAGS} ${LDFLAGS} -o 1.${COTEMP}.x 1.${COTEMP}.o -l$i 2>&1 || echo '' "$x" ;} | sed -e 's/[][()+@?]/#/g' || :` in
+ exec - then case `{ ${CC} ${mam_cc_FLAGS} ${CCFLAGS} -L${INSTALLROOT}/lib ${LDFLAGS} -o 1.${COTEMP}.x 1.${COTEMP}.o -l$i 2>&1 && ./1.${COTEMP}.x || echo '' "$x" ;} | sed -e 's/[][()+@?]/#/g' || :` in
+ exec - *$x*) case `{ ${CC} ${mam_cc_FLAGS} ${CCFLAGS} ${LDFLAGS} -o 1.${COTEMP}.x 1.${COTEMP}.o -l$i 2>&1 && ./1.${COTEMP}.x || echo '' "$x" ;} | sed -e 's/[][()+@?]/#/g' || :` in
exec - *$x*) continue ;;
exec - esac
exec - ;; |
Yes – it is the same issue. |
They are statically linked in the stock 93u+ build. E.g., |
The macOS 13 SDK that comes with Xcode.app includes a libmd stub
that fools the linker into thinking the md (message digest) library
is actually installed, though this is not the case. Consequently,
after installing Xcode.app, compiling programs with -lmd
incorrectly succeeds, but executing them fails:
$ echo 'int main(void) { return 0; }' >test.c
$ cc test.c -lmd
$ ./a.out
dyld[63517]: Library not loaded: '/usr/lib/libmd.dylib'
Referenced from: '/usr/local/src/ksh93/ksh/a.out'
Reason: tried: '/usr/lib/libmd.dylib' (no such file), '/usr/local/lib/libmd.dylib' (no such file)
Abort
This fools the probes for libmd in lib{cmd,sub}/Mamfile as they
only try to compile the program, not run it. A false positive
result for libmd results, and the build fails at a later (seemingly
unrelated) stage.
Clearly, this is Apple's bug and not ours. As a workaround for
their 93u+ 2012-08-01 build, Apple patched lib{cmd,sum}/Mamfile to
simply remove probing for libmd. But that is not an acceptable
multi-platform solution. A better workaround is almost as simple:
the test program should actually be run as part of the probe.
On Apple systems with this bug, a dyld Abort crash like the above
will now show up in the compile log four times as the test program
fails to run. I'm okay with this.
src/lib/libcmd/Mamfile,
src/lib/libsum/Mamfile:
- When generating cmd.req and sum.req, actually attempt to run the
test program as part of the link test. This is done by adding
'&& ./1.${COTEMP}.x' where appropriate.
- Reformat the probe code (originally generated by nmake) for
legibility and manual maintenance.
Thanks to @tvalenta and @posguy99 for their help finding the cause
of the problem.
Resolves: #567
Resolves: #596
|
Note: if you still get the build error after pulling the fix, delete |
|
Confirming that CLT 14.2 can now be used to build |
FYI - I went back and looked at the version of ksh apple distributes. If I understand correctly, the And for any of those, I finally realize I wouldn't have ever noticed the difference b/c I don't have |
Fails with the below even on a clean checkout. macOS 12.6.2 (not 12.6.1), on Intel and M1.
With IFFEFLAGS=-d1
The text was updated successfully, but these errors were encountered: