libgcc12 @ 12.3.0: Fix build on Ventura 13.3.1 (a)

[lukaso]

Description

Fixes https://trac.macports.org/ticket/67424 but may need adjustment
for other platforms (assuming the pre-existing patch from 6 years ago is
still needed.

Type(s)

• bugfix
• enhancement
• security fix

Tested on

macOS 13.3.1 22E772610a arm64
Xcode 14.3 14E222b

Verification

Have you

• followed our Commit Message Guidelines?
• squashed and minimized your commits?
• checked that there aren't other open pull requests for the same change?
• referenced existing tickets on Trac with full URL?
• checked your Portfile with port lint --nitpick? warning not related to my change
• tried existing tests with sudo port test?
• tried a full install with sudo port -vst install?
• tested basic functionality of all binary files?
• checked that the Portfile's most important variants haven't been broken?

[kencu]

so libgcc12 is already building on Ventura.

Your issue is occurring because you’re forcing an SDK and deployment target.

We force a generic SDK for gcc12 because every decimal point update in the SDK broke gcc by preventing it from finding an SDK to use.

Be very careful with this PR. Things are more fragile than you might guess here.

[lukaso]

Where is the generic SDK coming from? It's not there on my machine. And MacOS has capabilities to find SDKs and inject them into the build process. (But yes, I'm not surprised this is a delicate piece of the puzzle.) This was also working until recently (pre-PR) so I'm not sure what changed.

[cjones051073]

so libgcc12 is already building on Ventura.

Your issue is occurring because you’re forcing an SDK and deployment target.

We force a generic SDK for gcc12 because every decimal point update in the SDK broke gcc by preventing it from finding an SDK to use.

Be very careful with this PR. Things are more fragile than you might guess here.

Yes, I remain unconvinced this is a good change at this time.

[lukaso]

I've done a little bit more digging. In order to use an older SDK, I download and store it here:

/Library/Developer/CommandLineTools/SDKs

And this is what the gcc12 library picks up. But the generic SDK is actually at:

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk

And the SDKs directory there will not allow me to download my SDK there. But the hack in gcc12 just manipulates the sdk to make it work, but is doing so in the wrong directory.

And the reason this is a problem in the first place is because the pre-built gcc12 doesn't seem to exist (backup on the build servers?)

In any case, the fix of manipulating the SDK config in the way it is being done looks wrong, since in reality SDKROOT can be set to anything. However, I agree my fix is not right either.

[lukaso]

I've updated the PR now to check if the file exists. If it does not, then adjust to the generic SDK.

What this means is:

• Custom SDK set, and found, use it (this is my use case)
• Point SDK set, and found, use it
• Point SDK set, and not found, use a generic SDK

[mascguy]

I've updated the PR now to check if the file exists. If it does not, then adjust to the generic SDK.

What this means is:
* Custom SDK set, and found, use it (this is my use case)
* Point SDK set, and found, use it
* Point SDK set, and not found, use a generic SDK

While this seems reasonable, it might be preferable to force use of a custom SDK via a new variant. (And fail with an error if it doesn't exist.) That achieves your goal, while also preventing any unanticipated change to default behavior.

Anyone else... thoughts?

[lukaso]

While this seems reasonable, it might be preferable to force use of a custom SDK via a new variant. (And fail with an error if it doesn't exist.) That achieves your goal, while also preventing any unanticipated change to default behavior.

I know I'm not "anyone else" 😄, but here are my 2 cents anyway.

Your suggestion does fix my problem. However, I think the patch as it stands (before my changes) is a workaround since the SDK should always be available at the sdkroot (this is the standard MacOS mechanism for indicating which SDK to use after all). Therefore, the fact that the sdk isn't always present where the sdkroot is pointing is a bug.

Therefore, the solution of blindly overriding the sdkroot isn't right, in my opinion. Equally, forcing people to use a specific variant to restore the default MacOS behaviour seems odd (e.g. +usestandardsdkroot). If anything, I would make the existing patch a variant, though I don't see much value in doing that either since it currently (based on my change) only overrides the behaviour if it will otherwise fail.

[mascguy]

While the change looks safe to me, I don't have the deep knowledge and insight into what could potentially break... even if it's only a 0.1% chance.

So I added a few of our most senior members, and managers, to help decide this. I'm confident we'll be able to find consensus, but we need to give folks time to review and consider any potential implications.

[kencu]

we specifically don’t use configure.sdkroot because it is too fragile. It changes all the time on newer systems, and breaks gcc when it does change, as that SDK is burned into the gcc build. There were dozens of tickets about this, over and over, every time there was a minor change by Apple in the SDK name. So don’t do that.

We specifically do use the generic SDK on newer systems as it always exists.

@lukaso :

What happened to your generic SDK in /Library/Developer/CommandLineTools/SDKs ?

If you don’t have one, you must have deleted it, and that is certain to be part of your troubles.

Please don’t commit this. We’ll have to undo it and rebuild every gcc again.

[kencu]

If you have a custom need for a gcc that is configured to use an older, non-standard SDK for a custom project you want to do, that is just fine. Use your custom gcc build as you wish. Lots of us do that kind of thing for specific purposes.

Using a portfile overlay works great.

But pushing a niche use case into the main repo when it will break everyone else eventually is just not a great plan.

[cjones051073]

I agree with @kencu , I am not in favour of merging this.

[lukaso]

What happened to your generic SDK in /Library/Developer/CommandLineTools/SDKs ?

I'm using a custom SDK (10.12 for Intel and 11.3 for Arm) which can only be placed in the command line tools directory and therefore the fix in the Portfile as it stands doesn't work, since it's in a different directory. The generic SDK is where it normally is.

But pushing a niche use case into the main repo when it will break everyone else eventually is just not a great plan.

I'm curious to understand how this breaks everyone else?

Using a portfile overlay works great.

Yes, that's what I'm doing now since this is still pending.

[kencu]

But pushing a niche use case into the main repo when it will break everyone else eventually is just not a great plan.

I'm curious to understand how this breaks everyone else?

I explained that above. Bakes in a fragile SDK path.

[lukaso]

I explained that above. Bakes in a fragile SDK path.

So I fully understand: the --with-sysroot flag means that anytime the Mac SDK is updated, gcc12 will break applications it builds, unless it's recompiled. Is that right?

If I turn the patch around and only do the fallback if there is no SDK to be found in the current approach, would that fix the problem with this patch?

[cjones051073]

I explained that above. Bakes in a fragile SDK path.

So I fully understand: the --with-sysroot flag means that anytime the Mac SDK is updated, gcc12 will break applications it builds, unless it's recompiled. Is that right?

If I turn the patch around and only do the fallback if there is no SDK to be found in the current approach, would that fix the problem with this patch?

For me no. The bottom line is I do not want any official option to use any SDK other than MacOSX.sdk or MacOSX13.sdk.

[kencu]

the way gcc is supposed to work, you can override the baked-in default SDK by setting SDKROOT or passing in the isysroot on the command line.

The baked-in value is supposed to be used only if nothing else is specified.

So you should be able to use your custom SDK choice with gcc however it was built…

Is that not working any longer? If not, that would need fixing, certainly.

[lukaso]

the way gcc is supposed to work, you can override the baked-in default SDK by setting SDKROOT or passing in the isysroot on the command line.

The baked-in value is supposed to be used only if nothing else is specified.

So you should be able to use your custom SDK choice with gcc however it was built…

Is that not working any longer? If not, that would need fixing, certainly.

Was this question for me? In my use case, gcc does not compile, so it is no longer working, but I don't think that's what you meant. 😄

[kencu]

the way gcc is supposed to work, you can override the baked-in default SDK by setting SDKROOT or passing in the isysroot on the command line.
The baked-in value is supposed to be used only if nothing else is specified.
So you should be able to use your custom SDK choice with gcc however it was built…
Is that not working any longer? If not, that would need fixing, certainly.

Was this question for me? In my use case, gcc does not compile, so it is no longer working, but I don't think that's what you meant. 😄

presumably if you put back the generic sdk you removed, gcc would compile again, and then gcc should build software against any sdk you want.

Where I’m going with this is that you don’t need to modify gcc to have it build software against the sdk you want to use; it can already do that as it is. and if it doesn’t, there is something to fix.

[lukaso]

presumably if you put back the generic sdk you removed, gcc would compile again, and then gcc should build software against any sdk you want.

I didn't remove the generic SDK. In fact, I don't think it's possible to mess with that directory on Ventura. What happened is I set the SDKROOT to a different location when doing my full build for GIMP, which among other things requires gcc. That location is the command line tools SDK directory since this is the only place where you can put different SDKs. There is no generic SDK there (though I suppose I could create a symlink there just to work around this problem, but that could have other unintended consequences when I'm building other ports).

Here's the setting (for example):

export SDKROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk

And I have to build gcc for 2 reasons (believe me, I don't want to).

1. For some reason the ventura build bot was broken for a while, so there was no pre-built gcc for the latest version, even when I explicitly changed the config for just gcc to use the prebuilt version (I don't need compatibility for older mac versions for gcc since it can itself compile targets with a custom SDKROOT)
2. When using MacPorts in non-sudo mode, in which case you always have to build.

Where I’m going with this is that you don’t need to modify gcc to have it build software against the sdk you want to use; it can already do that as it is. and if it doesn’t, there is something to fix.

Unfortunately for the reasons stated above, I haven't found a way to avoid building gcc. In fact, in order to build GIMP for release on MacOS down to version 10.12 I have to build (due to extensive and diverse dependencies):

• gcc12
• libgcc12
• cmake
• python27
• python38
• python310
• python311
• llvm-15
• llvm-16
• clang-16
• rust
• perl

None of these except python310 (GIMP3) and python27 (GIMP2) are required at runtime.
If you are interested in the ins and outs, you can see the full pipeline in this repo: https://gitlab.gnome.org/Infrastructure/gimp-macos-build/-/tree/master. Especially look at the files in the scripts directory.

If there's a better way to do all this, I would very much welcome any ideas.

[kencu]

Oh, I thought you said you removed:

/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk

and that would match your build error:

:info:build The directory that should contain system headers does not exist:
:info:build   /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include

So is it there? Or not? If it is not there, gcc (and probably other software) won't build on newer systems at least.

On my system, I see these SDKs, after a few Xcode upgrades:

% ls -la /Library/Developer/CommandLineTools/SDKs
total 0
drwxr-xr-x  8 root  wheel  256  2 Jun 19:58 .
drwxr-xr-x  5 root  wheel  160  9 Jan 22:50 ..
lrwxr-xr-x  1 root  wheel   14  2 Jun 19:57 MacOSX.sdk -> MacOSX13.3.sdk
drwxr-xr-x  7 root  wheel  224 19 Nov  2021 MacOSX12.1.sdk
drwxr-xr-x  7 root  wheel  224  2 Jun 19:58 MacOSX12.3.sdk
lrwxr-xr-x  1 root  wheel   14  2 Jun 19:57 MacOSX12.sdk -> MacOSX12.3.sdk
drwxr-xr-x  7 root  wheel  224  2 Jun 19:57 MacOSX13.3.sdk
lrwxr-xr-x  1 root  wheel   14  2 Jun 19:56 MacOSX13.sdk -> MacOSX13.3.sdk

[kencu]

By the way, from your ticket you said that on Ventura, in macports.conf you set this:

macosx_deployment_target 11.0
macosx_sdk_version 11.3

I guess my first question is why are you setting those? You shouldn't need to do that, and it can cause problems for you to be trying to micromanage those things.

Edit: Oh, I see why. You want to allow the built software to run on 11.0 and up, and to minimize troubles by using the matching SDK.

Hmmm.

Well gcc at least will want /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk to exist. I think you'll have to symlink that to some SDK. The usual thing to do would be to let xcode take care of that and it will symlink it to the latest SDK you have.

If you use the system for nothing other than this one project, gimp with a deployment target of 11.0, you could symlink it to the MacOSX11.3.sdk.

[kencu]

Somehow I think there is some small core thing you're doing here that is messing you up. I am not thinking there is anything in MacPorts that needs fixing at present, but something in what you're trying to do and how you're doing it.

I could be wrong, once we see how it all unfolds in the end, though.

[cjones051073]

Somehow I think there is some small core thing you're doing here that is messing you up. I am not thinking there is anything in MacPorts that needs fixing at present, but something in what you're trying to do and how you're doing it.

I could be wrong, once we see how it all unfolds in the end, though.

I agree. So far for me this is most likely some issue at @lukaso 's end and not an issue in MP that needs fixing.

Its also I think suspicious that you say you use

export SDKROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk

on macOS 13.

whilst I know its more common in e.g.Xcode projects to allow a build to target an older OS than the one being built on, thats not really what the above does. MacPorts does not really support anything other than using the SDK for the system you are on.

[lukaso]

Somehow I think there is some small core thing you're doing here that is messing you up. I am not thinking there is anything in MacPorts that needs fixing at present, but something in what you're trying to do and how you're doing it.
I could be wrong, once we see how it all unfolds in the end, though.

I agree. So far for me this is most likely some issue at @lukaso 's end and not an issue in MP that needs fixing.

Its also I think suspicious that you say you use

export SDKROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk

on macOS 13.

whilst I know its more common in e.g.Xcode projects to allow a build to target an older OS than the one being built on, thats not really what the above does. MacPorts does not really support anything other than using the SDK for the system you are on.

One of the reasons I chose MacPorts to use for building the official GIMP download package is because it supports setting the SDK to something other than the current OS (at least that was what I took away from discussing it with @mascguy). GIMP needs to be built to support 10.12 on Intel and 11.0 on ARM. Building only for the current OS would unfortunately be useless for my use case. I know it's not the core use case of MacPorts, but it's also not an optional one for me.

I believe Darktable also uses this method.

@kencu This is my command line tools directory:

$ ls -las /Library/Developer/CommandLineTools/SDKs
total 0
0 drwxr-xr-x  4 root  wheel  128  1 Apr 15:08 .
0 drwxr-xr-x  4 root  wheel  128 28 Mar 16:11 ..
0 drwxr-xr-x  7 1000  1000   224  3 May  2021 MacOSX11.3.sdk
0 drwxr-xr-x  3 root  wheel   96 28 Mar 16:11 MacOSX12.3.sdk

And the Circle CI runners also don't have the generic symlink.

The generic SDK link for me is at:

/Applications/Xcode. app/Contents/Developer/Platforms/MacOSX. platform/Developer/SDKs/MacOSX. sdk

$ ls -las /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/
total 0
0 drwxr-xr-x  5 root  wheel  160  8 Apr 03:28 .
0 drwxr-xr-x  6 root  wheel  192 24 Mar 11:08 ..
0 drwxr-xr-x  7 root  wheel  224 10 Mar 00:58 MacOSX.sdk
0 lrwxr-xr-x  1 root  wheel   10  8 Apr 03:25 MacOSX13.3.sdk -> MacOSX.sdk
0 lrwxr-xr-x  1 root  wheel   10  8 Apr 03:25 MacOSX13.sdk -> MacOSX.sdk

[kencu]

MacPorts gcc build requires this symlink to exist, and point to an SDK, ideally the current SDK, like mine does.

/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk

It is the default for the command line tools installation to have this symlink, so you must have deleted it along the way somewhere.

What is inside Xcode is not relevant for this (because the MacPorts gcc build doesn't use xcode, it uses the CLTs).

(As an aside, it looks like your CLTs must be out of date, as the SDKs you have in there are not current.)

[lukaso]

MacPorts gcc build requires this symlink to exist, and point to an SDK, ideally the current SDK, like mine does.

/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk

It is the default for the command line tools installation to have this symlink, so you must have deleted it along the way somewhere.

There's no mechanism for me to have deleted that symlink. I definitely didn't do it myself. I installed xcode and CLT the standard way. This Mac was wiped in January and installed from scratch so there's nothing legacy hanging around. I just upgraded to 13.4 (auto upgrade) and just opened XCode (which did install something for developing on MacOS), but no change to either directory.

Does MacPorts create that symlink, or something else messes about in there deleting things?

(As an aside, it looks like your CLTs must be out of date, as the SDKs you have in there are not current.)

$ softwareupdate --list
Software Update Tool

Finding available software
No new software available.

I can't find anything about keeping the CLT up to date (and in the past I always got updates to CLTs through the standard software update mechanism).

I guess what I'm saying is that I don't see how it could be that my Mac is out of date. And equally, that the hosted MacOS VMs on Circle CI are out of date and out of date in the same way.

[lukaso]

Here's what I see on the Circle CI runner once it's booted up:

total 0
0 drwxr-xr-x  5 root  wheel  160 Dec  7 20:07 .
0 drwxr-xr-x  6 root  wheel  192 Nov 17  2022 ..
0 drwxr-xr-x  7 root  wheel  224 Dec  7 20:05 MacOSX.sdk
0 lrwxr-xr-x  1 root  wheel   10 Dec  7 20:04 MacOSX13.1.sdk -> MacOSX.sdk
0 lrwxr-xr-x  1 root  wheel   10 Dec  7 20:04 MacOSX13.sdk -> MacOSX.sdk

$ ls -las /Library/Developer/CommandLineTools/SDKs
total 0
0 drwxr-xr-x  7 root  wheel  224 Dec  7 20:00 .
0 drwxr-xr-x  5 root  wheel  160 Dec  7 19:27 ..
0 lrwxr-xr-x  1 root  wheel   14 Dec  7 19:59 MacOSX.sdk -> MacOSX13.1.sdk
0 drwxr-xr-x  7 root  wheel  224 Dec  7 20:00 MacOSX12.3.sdk
0 lrwxr-xr-x  1 root  wheel   14 Dec  7 19:59 MacOSX12.sdk -> MacOSX12.3.sdk
0 drwxr-xr-x  7 root  wheel  224 Nov 12  2022 MacOSX13.1.sdk
0 lrwxr-xr-x  1 root  wheel   14 Dec  7 19:59 MacOSX13.sdk -> MacOSX13.1.sdk

Which supports your point...that the SDKs should be in the CLTs as well.

[kencu]

there is a known issue with the CLTs described here, and a fix:

https://trac.macports.org/wiki/ProblemHotlist#reinstall-clt

but what I would do if you can is save the older SDKs you want to keep somewhere safe, remove

/Library/Developer/CommandLineTools

and install the ones that match your xcode, ideally the very latest version, from the Apple Dev website.

Then copy or symlink the older SDKs back into

/Library/Developer/CommandLineTools/SDKs/

and you should be in business

[cjones051073]

I am going to close this, as its really not a change we want in the main ports tree.

@lukaso If you still have issues after fixing your CLT issues Ken outlines above, please open a Trac ticket instead so a fix can be discussed there before opening a PR.