dyld: lazy symbol binding failed: Symbol not found: _SCDynamicStoreCopyProxies #11502

ryandesign · 2023-07-23T10:20:27Z

I did this

Install curl 8.2.0 on OS X 10.8 (or on any Mac OS X or OS X version where curl was compiled with the macOS 10.11 SDK or earlier)

I expected the following

curl compiles but fails to run, for example:

$ curl --version
dyld: lazy symbol binding failed: Symbol not found: _SCDynamicStoreCopyProxies
  Referenced from: /opt/local/lib/libcurl.4.dylib
  Expected in: flat namespace

dyld: Symbol not found: _SCDynamicStoreCopyProxies
  Referenced from: /opt/local/lib/libcurl.4.dylib
  Expected in: flat namespace

Trace/BPT trap: 5

This was reported to MacPorts here: https://trac.macports.org/ticket/67803

The problem seems to be that curl always wants to use SCDynamicStoreCopyProxies on macOS, and this is part of the SystemConfiguration framework, and has been ever since Mac OS X 10.1. However, curl only links with that framework when building against the macOS 10.12 SDK or later.

Normally a missing symbol would be a link-time error but by using the flags -Wl,-undefined -Wl,dynamic_lookup you're telling the linker to ignore the problem and that the symbol will exist at runtime. It doesn't, though, because you haven't linked in the framework that has the symbol.

This configure test is faulty:

checking whether to link macOS CoreFoundation and SystemConfiguration framework... no

It is checking whether TARGET_OS_OSX is defined. That define was newly added in the macOS 10.12 SDK.

It looks like an attempt to fix this issue was made in #11417 but it didn't fix it in m4/curl-sysconfig.m4.

TARGET_OS_OSX was introduced to help developers distinguish between macOS and other Darwin-derived operating systems (iOS, tvOS, watchOS). Is that distinction necessary here? If not, you can use TARGET_OS_MAC which encompasses all of them. Another way to identify any Darwin-based Apple OS is to check if both __APPLE__ and __MACH__ are defined.

curl/libcurl version

8.2.0

operating system

Darwin redacted 12.6.0 Darwin Kernel Version 12.6.0: Wed Mar 18 16:23:48 PDT 2015; root:xnu-2050.48.19~1/RELEASE_X86_64 x86_64

The text was updated successfully, but these errors were encountered:

ryandesign · 2023-07-23T10:29:52Z

~~#11417 was merged after the 8.2.0 release so I haven't tested it but~~ I'm still dubious of the use of TARGET_OS_OSX at all.

ryandesign · 2023-07-23T11:00:10Z

If the macOS distinction is important—if you need this code to happen on macOS but not iOS/tvOS/watchOS—a condition that was used in harfbuzz for that purpose was:

#if TARGET_OS_MAC && !(defined(TARGET_OS_IPHONE) && TARGET_OS_IPHONE)

The now-not-particularly-accurately-named TARGET_OS_IPHONE means "any Darwin-derived OS that is not macOS". (It was introduced back when iOS, which was then called iPhoneOS, was the only non-macOS Darwin OS.)

bagder · 2023-07-23T14:39:31Z

If the macOS distinction is important—if you need this code to happen on macOS but not iOS/tvOS/watchOS

We want it to work as widely as possible. But I don't know what that requires.

ryandesign · 2023-07-23T18:48:01Z

The question is what was the intent of #7121: to use it on any Darwin OS or only on macOS? It says:

if USE_RESOLVE_ON_IPS is set (which is currently only set on macOS)

but looking at the code it is actually defined on all Apple platforms:

curl/lib/curl_setup.h

Lines 258 to 260 in 57f56cc

    
           #if defined(__APPLE__) && !defined(USE_ARES) 
        
           #include <TargetConditionals.h> 
        
           #define USE_RESOLVE_ON_IPS 1

It also says:

I need to check if it exists on targets like iOS

I grep'd through the SDKs and this function is mentioned in non-macOS SDKs:

% find /Volumes/BigSur/Applications/Xcode.app/Contents/Developer/Platforms -name 'SystemConfiguration.tbd' -or -name 'SCDynamicStoreCopySpecific.h' 2>/dev/null | xargs grep --color SCDynamicStoreCopyProxies            
/Volumes/BigSur/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/System/Library/Frameworks/SystemConfiguration.framework/SystemConfiguration.tbd:                       _SCDynamicStoreCopyProxies, _SCDynamicStoreCopyProxiesWithOptions, 
/Volumes/BigSur/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/System/Library/Frameworks/SystemConfiguration.framework/Headers/SCDynamicStoreCopySpecific.h:	@function SCDynamicStoreCopyProxies
/Volumes/BigSur/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/System/Library/Frameworks/SystemConfiguration.framework/Headers/SCDynamicStoreCopySpecific.h:SCDynamicStoreCopyProxies		(
/Volumes/BigSur/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/System/Library/Frameworks/SystemConfiguration.framework/SystemConfiguration.tbd:                       _SCDynamicStoreCopyProxies, _SCDynamicStoreCopyProxiesWithOptions, 
/Volumes/BigSur/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/System/Library/Frameworks/SystemConfiguration.framework/Headers/SCDynamicStoreCopySpecific.h:	@function SCDynamicStoreCopyProxies
/Volumes/BigSur/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/System/Library/Frameworks/SystemConfiguration.framework/Headers/SCDynamicStoreCopySpecific.h:SCDynamicStoreCopyProxies		(
/Volumes/BigSur/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/SystemConfiguration.framework/SystemConfiguration.tbd:                       _SCDynamicStoreCopyNotifiedKeys, _SCDynamicStoreCopyProxies, 
/Volumes/BigSur/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/SystemConfiguration.framework/SystemConfiguration.tbd:                       _SCDynamicStoreCopyProxiesWithOptions, _SCDynamicStoreCopyValue, 
/Volumes/BigSur/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration.tbd:                       _SCDynamicStoreCopyNotifiedKeys, _SCDynamicStoreCopyProxies, 
/Volumes/BigSur/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration.tbd:                       _SCDynamicStoreCopyProxiesWithOptions, _SCDynamicStoreCopyValue, 
/Volumes/BigSur/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/Headers/SCDynamicStoreCopySpecific.h:	@function SCDynamicStoreCopyProxies
/Volumes/BigSur/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/Headers/SCDynamicStoreCopySpecific.h:SCDynamicStoreCopyProxies		(
/Volumes/BigSur/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/System/Library/Frameworks/SystemConfiguration.framework/SystemConfiguration.tbd:                       _SCDynamicStoreCopyProxies, _SCDynamicStoreCopyProxiesWithOptions, 
/Volumes/BigSur/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/System/Library/Frameworks/SystemConfiguration.framework/Headers/SCDynamicStoreCopySpecific.h:	@function SCDynamicStoreCopyProxies
/Volumes/BigSur/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/System/Library/Frameworks/SystemConfiguration.framework/Headers/SCDynamicStoreCopySpecific.h:SCDynamicStoreCopyProxies		(
/Volumes/BigSur/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/System/Library/Frameworks/SystemConfiguration.framework/SystemConfiguration.tbd:                       _SCDynamicStoreCopyProxies, _SCDynamicStoreCopyProxiesWithOptions, 
/Volumes/BigSur/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/System/Library/Frameworks/SystemConfiguration.framework/Headers/SCDynamicStoreCopySpecific.h:	@function SCDynamicStoreCopyProxies
/Volumes/BigSur/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/System/Library/Frameworks/SystemConfiguration.framework/Headers/SCDynamicStoreCopySpecific.h:SCDynamicStoreCopyProxies		(

But this Stack Overflow post shows someone encountering these errors:

'SCDynamicStoreCreate' is unavailable: not available on iOS
'SCDynamicStoreCopyValue' is unavailable: not available on iOS

with someone responding:

Those are only available for Mac OS

bagder · 2023-07-23T21:09:58Z

what was the intent

I'm not sure that's terribly important. The intent of that PR was to fix the problem of the author, which I believe it did. That seems have been to make libcurl to behave better on macOS.

Does this logic also apply to other apple operating systems and versions or is it perhaps restricted to a subset? And if so, can the change be improved?

If so, I hope someone offers a PR that does it. The intent of the PR from two years ago is no longer that interesting.

ryandesign · 2023-07-24T01:09:38Z

I tried a build for iOS after changing the conditions from TARGET_OS_OSX to TARGET_OS_MAC (which includes iOS and other Darwin OS variants) and the build failed:

macos.c:52:28: error: 'SCDynamicStoreCopyProxies' is unavailable: not available on iOS
    CFDictionaryRef dict = SCDynamicStoreCopyProxies(NULL);
                           ^

That confirms for me that this should really only be used on macOS, not other Darwin OS derivatives, and I'll prepare a PR to fix it so it'll work on older SDKs.

ryandesign · 2023-07-24T05:04:08Z

I haven't decided yet what conditional to use to identify macOS. Elsewhere in curl this conditional is used:

curl/lib/vtls/sectransp.c

Line 70 in 3c0a910

#if (TARGET_OS_MAC && !(TARGET_OS_EMBEDDED || TARGET_OS_IPHONE))

However Apple now says TARGET_OS_EMBEDDED is deprecated. I'm also keeping in mind the fact that newer clang compilers default to -Werror,-Wundef-prefix=TARGET_OS_ and so you either have to be certain that the TARGET_OS_ macros you use are defined on every OS or you have to check that they're defined first.

The conditional to identify macOS is currently repeated in four files. The breakage in 8.2.0 was caused by making a change to the conditional in only three out of the four files. Would it be better to define that conditional in fewer files so that it's less likely to go out of sync? I was thinking of defining it only here:

curl/m4/curl-sysconfig.m4

Line 33 in 3c0a910

#if (TARGET_OS_OSX)

however it would need to be duplicated in the CMake build system too. But then the configure script or CMake could define a macro in curl_config.h which the rest of the .c and .h files could use. If this is a good idea, what would be a good macro name? Are there other existing curl macros for identifying other operating systems?

ryandesign · 2023-07-25T10:23:53Z

I still think reducing the number of places the same conditional is repeated would be a good idea but for now I've submitted a PR that just fixes the conditionals.

I'll attempt to reproduce the PHP FPM problem and will file a separate issue if I find anything.

fds242 · 2023-07-25T12:15:07Z

As feared, this change will make it a tad hairier to avoid including this code. Still, compiling with -D __TARGETCONDITIONALS__ -D TARGET_OS_MAC=0 should work, or faking a Mac Catalyst build through --target=<arch>-apple-ios##-macabi

bagder · 2023-07-25T15:48:20Z

@fds242 if you can provide evidence of problems with this function call, preferably in a way to reproduce the issue, then we could work on fixing the root cause instead of you trying to disable parts of libcurl that is not written to be disabled easily. That would be great!

fds242 · 2023-07-25T16:30:21Z

@badger It is inherent incompatibility that logically cannot be fixed, only disabled or removed. Assuming we accept as noted in #11254 that this function should not be called in a fork. (I have not personally found Apple documentation to this effect on this specific call.) PHP-FPM is a classic single-threaded Unix server that creates worker forks of itself, as configured, to carry out the work. By the time a worker fork loads the PHP curl shared extension, and hence any libcurl code has a chance to run, it is already inside a fork. Whether documented or not, this sure causes severe issues, unrelated PHP extensions suddenly now causing the worker fork to unceremoniously crash with no error message, just because a libcurl that included this call was also loaded.

Only workarounds I could imagine are outside of curl,
a) linking PHP directly to libcurl rather than keeping the curl extension as a shared module (this is not normally done since PHP internals don't and can't take advantage of curl)
b) manually adding this call to SCDynamicStoreCopyProxies into php-fpm's code
Both different ways of achieving the same idea of making sure the call is first executed at server start, rather than inside a fork.

But, again, it continues to make no sense to me for libcurl to even attempt make this call. Even if you disregard the terrible crashes it causes, it's quite akin to how libcurl would never make a setlocale() call behind your back, but calls to setlocale() are included in tool/operate.c for the curl tool, as they should be. It is the purview of the application, not the library, to make calls that affect further system calls for the entire process.

In the end I'm satisfied I could continue building libcurl without it, I was just hoping for the change to be more like … ah, I see changes are actively being made, and looking much closer to how I hoped it would be on making my first comments.

If the define to CURL_OSX_CALL_COPYPROXIES could be wrapped in an ifndef check, it would be perfection.

#ifndef CURL_OSX_CALL_COPYPROXIES
#define CURL_OSX_CALL_COPYPROXIES 1
#endif

Then you could very easily -D CURL_OSX_CALL_COPYPROXIES 0

bagder · 2023-07-25T16:44:54Z

I'm not satisfied with that.

ryandesign · 2023-07-25T23:01:16Z

@fds242 I've attempted to use PHP FPM with the curl and pdo_pgsql modules and I haven't been able to make it crash, so I may not be testing the same way you were. Could you file a separate issue please with more information on your setup (macOS version, PHP version, which web server and version, a minimal reproduction recipe)?

fds242 · 2023-07-26T04:57:55Z

@ryandesign Apologies for the off-topic reply here, but I simply don't know when I'd have the time to finish writing up & filing a proper separate issue myself. Thank you for wanting to look into it.

After too much trial-and-error, the key to triggering a 100% immediately reproducible crash is having a libpq (the postgresql client library) built with Kerberos support (--with-gssapi), loaded alongside curl containing the unfortunate call to SCDynamicStoreCopyProxies. You don't even have to try to invoke anything curl (since it now unconditionally executes the call on load), it simply has to be loaded to trigger a crash as soon as you attempt to connect to an actually existing PostgreSQL server. Only the version of curl is sensitive, all the relevant changes landed in 8.2.0. But everything I tested was the current non-beta release version. Web server is also irrelevant, Apache, Caddy, nginx all the same. I'm still on a x86_64 Mac. This I don't know if changes reproducibility. Compilers/SDKs/macOS version targets I tested a wide variety and made no difference.

Finally here are some draft build & test instructions that should make it easy to reproduce:

# no need for anything via Homebrew, MacPorts, or manual installs, only Xcode

# everything will just go here, easy to trash later
WORKDIR="$HOME/Desktop/crashtest"

mkdir "$WORKDIR" && cd "$WORKDIR"

# assuming you already downloaded them
tar -xjf ~/Downloads/curl-8.2.0.tar.bz2
tar -xjf ~/Downloads/postgresql-15.3.tar.bz2
tar -xjf ~/Downloads/php-8.2.8.tar.bz2

( cd curl-8.2.0/ && CPPFLAGS="-D TARGET_OS_OSX=1" ./configure --prefix="$WORKDIR" --without-ssl --disable-ldap && make -j4 install )

( cd postgresql-15.3/ && ./configure --prefix="$WORKDIR" --with-gssapi --without-ldap && make -j4 -C src/interfaces/libpq install && make -C src/include install )

( cd php-8.2.8/ && CURL_CFLAGS="-I$WORKDIR/include" CURL_LIBS=$("$WORKDIR"/bin/curl-config --libs) ./configure --prefix="$WORKDIR" --disable-all --enable-fpm --with-curl=shared,"$WORKDIR" --enable-pdo --with-pdo-pgsql=shared,"$WORKDIR" && make -j4 install )



## this keeps the default php-fpm listen setting of TCP localhost:9000, ensure nothing listens there already
mv -i etc/php-fpm.d/www.conf.default etc/php-fpm.d/www.conf
mv -i etc/php-fpm.conf.default etc/php-fpm.conf

## load both extensions
echo "extension = pdo_pgsql" > "$WORKDIR"/lib/php.ini
echo "extension = curl" >> "$WORKDIR"/lib/php.ini

## server has to actually exist (and support Kerberos?) to trigger the immediate crash, but valid working credentials don't matter
echo "<?php new PDO('pgsql:host=your-actual-working-postgresql-server');" > test.php

## the Caddy webserver is super easy to configure & test with
>Caddyfile cat <<-EOF
{
    http_port 9009
}
http:// {
    root * $WORKDIR
    php_fastcgi localhost:9000
}
EOF


# start PHP-FPM at last, this will keep this terminal occupied for easy debugging
"$WORKDIR/sbin/php-fpm" --nodaemonize

# get Caddy from https://github.com/caddyserver/caddy/releases
# make sure cwd is where we put the Caddyfile earlier
# this will also keep this terminal occupied
caddy run


# then you can finally test with for example:
curl -i http://localhost:9009/test.php

# you will get 502 Bad Gateway from Caddy when the php-fpm child crashes, and see the crash in php-fpm's output

# then see how the crash goes away if you stop loading the curl extension, or install a libcurl with the SCDynamicStoreCopyProxies call removed, etc.
# don't forget to restart php-fpm if changes were made, web server does not have to be restarted

Now SCDynamicStoreCopyProxies is called (and the required frameworks are linked in) on all versions of macOS and only on macOS. Fixes crash due to undefined symbol when built with the macOS 10.11 SDK or earlier. CURL_OSX_CALL_COPYPROXIES is renamed to CURL_MACOS_CALL_COPYPROXIES and is now only defined when SCDynamicStoreCopyProxies will actually be called. Previously, it was defined when ENABLE_IPV6 was not defined but SCDynamicStoreCopyProxies is not called in that case. TARGET_OS_OSX is only defined in the macOS 10.12 SDK and later and only when dynamic targets are enabled. TARGET_OS_MAC is always defined but means any Mac OS or derivative including macOS, iOS, tvOS, and watchOS. TARGET_OS_IPHONE means any Darwin OS other than macOS. Follow-up to c73b2f8 Closes curl#11502

curl/curl#11502 osx-arm64 builds fine without the patch, but I applied the patch to both macos arches we support because feature used by curl, and that generates the problem may be enabled at some point.

curl/curl#11502

Now SCDynamicStoreCopyProxies is called (and the required frameworks are linked in) on all versions of macOS and only on macOS. Fixes crash due to undefined symbol when built with the macOS 10.11 SDK or earlier. CURL_OSX_CALL_COPYPROXIES is renamed to CURL_MACOS_CALL_COPYPROXIES and is now only defined when SCDynamicStoreCopyProxies will actually be called. Previously, it was defined when ENABLE_IPV6 was not defined but SCDynamicStoreCopyProxies is not called in that case. TARGET_OS_OSX is only defined in the macOS 10.12 SDK and later and only when dynamic targets are enabled. TARGET_OS_MAC is always defined but means any Mac OS or derivative including macOS, iOS, tvOS, and watchOS. TARGET_OS_IPHONE means any Darwin OS other than macOS. Follow-up to c73b2f8 Fixes curl#11502 Closes curl#11516

bagder added appleOS specific to an Apple operating system help wanted labels Jul 23, 2023

ryandesign mentioned this issue Jul 23, 2023

lib/hostip.c: make NAT64 address synthesis on macOS working #7121

Closed

This comment was marked as off-topic.

Sign in to view

ryandesign mentioned this issue Jul 25, 2023

macOS: Really fix target detection #11516

Closed

bagder closed this as completed in 8b7cbe9 Jul 29, 2023

rafaelmartins pushed a commit to AnacondaRecipes/curl-feedstock that referenced this issue Aug 30, 2023

Fix macos bug

bd72a16

curl/curl#11502

rafaelmartins pushed a commit to AnacondaRecipes/curl-feedstock that referenced this issue Aug 31, 2023

Fix macos bug

7369eda

curl/curl#11502

rafaelmartins pushed a commit to AnacondaRecipes/curl-feedstock that referenced this issue Aug 31, 2023

Fix macos bug

f7f5ec2

curl/curl#11502

rafaelmartins pushed a commit to AnacondaRecipes/curl-feedstock that referenced this issue Aug 31, 2023

Fix macos bug

3446eba

curl/curl#11502

vszakats mentioned this issue Jul 4, 2024

GHA/macos: improve, fix gcc/llvm, add new test matrix #14097

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dyld: lazy symbol binding failed: Symbol not found: _SCDynamicStoreCopyProxies #11502

dyld: lazy symbol binding failed: Symbol not found: _SCDynamicStoreCopyProxies #11502

ryandesign commented Jul 23, 2023

ryandesign commented Jul 23, 2023 •

edited

Loading

ryandesign commented Jul 23, 2023

bagder commented Jul 23, 2023

ryandesign commented Jul 23, 2023

bagder commented Jul 23, 2023

ryandesign commented Jul 24, 2023

ryandesign commented Jul 24, 2023

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

ryandesign commented Jul 25, 2023

fds242 commented Jul 25, 2023

bagder commented Jul 25, 2023

fds242 commented Jul 25, 2023

bagder commented Jul 25, 2023

ryandesign commented Jul 25, 2023

fds242 commented Jul 26, 2023

dyld: lazy symbol binding failed: Symbol not found: _SCDynamicStoreCopyProxies #11502

dyld: lazy symbol binding failed: Symbol not found: _SCDynamicStoreCopyProxies #11502

Comments

ryandesign commented Jul 23, 2023

I did this

I expected the following

curl/libcurl version

operating system

ryandesign commented Jul 23, 2023 • edited Loading

ryandesign commented Jul 23, 2023

bagder commented Jul 23, 2023

ryandesign commented Jul 23, 2023

bagder commented Jul 23, 2023

ryandesign commented Jul 24, 2023

ryandesign commented Jul 24, 2023

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

ryandesign commented Jul 25, 2023

fds242 commented Jul 25, 2023

bagder commented Jul 25, 2023

fds242 commented Jul 25, 2023

bagder commented Jul 25, 2023

ryandesign commented Jul 25, 2023

fds242 commented Jul 26, 2023

ryandesign commented Jul 23, 2023 •

edited

Loading