Injecting functions into binaries with two-level namespaces

[Wowfunhappy]

Continuing discussion from #210

I don't think you need to use ObjC for your stub module, or do you?

I don't think so, the example with Unity is pure C. (Although it feels almost strange to call it that since it's a single one-line function.)

And I wonder, DYLD_INSERT_LIBRARIES does do what it promises, despite the fact that applications don't look inside it. I just did some testing with libc++ 17 from my LLVM-17 install tree. Normally they have their ID set to the libc++ and libc++abi dylibs that were used during the build (a priori the system ones), so inserting them still causes the system libraries to be loaded first. But if you change their ID (and make libc++.dylib link to the desired libc++abi) then they are the only libc++ libraries showing up when running with DYLD_PRINT_LIBRARIES=1. IOW, what if you set the ID of your UnityMavericksWorkarounds.dylib to /usr/lib/libSystem.B.dylib, could you then DYLD_INSERT_LIBRARIES it so you don't have to use install_name_tool?

How do you change the ID of a library? My understanding is that with two-level namespaces, dyld is looking for the full path to the library.

I generally prefer to use install_name_tool anyway because it works even when the app isn't launched from the Terminal. (I know you can add environment variables by editing info.plist, but then those variables are also injected into any processes that app launches which can cause problems.)

(@krackers you might find this discussion interesting.)

[RJVB]

I don't think you need to use ObjC for your stub module, or do you?

I don't think so, the example with Unity is pure C.

It certainly is, but compiling it as ObjC will probably link in the ObjC runtime, which might have some side-effects(?). It's probably also slower but that might not be apparent for such a small file.

How do you change the ID of a library?

install_name_tool -id foo libfoo.dylib
It's what gets stored in dependent binaries. The ID can be an absolute path, in which case the library will be loaded from that location (or from somewhere in DYLD_LIBRARY_PATH if it's not in the designated position). It can also be a special token like @rpath/libfoo.dylib, in which case dyld will search the RPATH stored in the binary for a libfoo.dylib.

(I know you can add environment variables by editing info.plist, but then those variables are also injected into any processes that app launches which can cause problems.)

That's true of course. If you're just injecting a library that adds a function there shouldn't be much side-effects though, and on Mac it will also depend on exactly how those processes are spawned. AFAIK the general recommendation is not to use system() or fork() + exec*() but rather the programmatic equivalent of using open in a shell. IIRC that means NSLaunch, which doesn't (by default) inherit the environment.

Either way, both approaches have their advantages and scopes of application. Using install_name_tool is probably better for production binaries, using the env. variables is practical to test if a solution works, have a baseline etc.

[RJVB]

Well, anyway, I did some very quick testing of a inject library I wrote for Linux *). It provides an overload for rename(2) for instance, which catches the error raised when renaming across devices and uses a fallback that will succeed. It also provides copy_file_range() implementation (with a similar fallback).

I built the wrapper library, ran optool on it to re-export libSystem.B.dylib and then tested simply what library actually provides a function that can be loaded from the resulting library (using dlsym)

> dlsym libwrapped_syscalls.dylib copy_file_range
libwrapped_syscalls.dylib::copy_file_range: 0x10845e55a (/Volumes/Debian/Users/bertin/work/src/new/libwrapped_syscalls.dylib::copy_file_range)
> dlsym libwrapped_syscalls.dylib rename
libwrapped_syscalls.dylib::rename: 0x10eb067d3 (/Volumes/Debian/Users/bertin/work/src/new/libwrapped_syscalls.dylib::rename)
> dlsym libwrapped_syscalls.dylib unlink
libwrapped_syscalls.dylib::unlink: 0x7fff85425c67 (/usr/lib/system/libsystem_kernel.dylib::unlink)

My hypothesis was that maybe we could fool two-level-namespace apps to use an inserted library to set the ID of that library to the one it attempts to extend:

> install_name_tool -id /usr/lib/libSystem.B.dylib ./libwrapped_syscalls.dylib
> dlsym libwrapped_syscalls.dylib copy_file_range
dlsym: error retrieving libwrapped_syscalls.dylib::copy_file_range (dlsym(0x7fff6cdfa378, copy_file_range): symbol not found)
> dlsym libwrapped_syscalls.dylib rename
libwrapped_syscalls.dylib::rename: 0x7fff85425ca5 (/usr/lib/system/libsystem_kernel.dylib::rename)
> dlsym libwrapped_syscalls.dylib unlink
libwrapped_syscalls.dylib::unlink: 0x7fff85425c67 (/usr/lib/system/libsystem_kernel.dylib::unlink)

As you can see, changing the library's ID has the opposite effect of what I hoped: it simply becomes a copy of the library it re-exports. The added function(s) have disappeared (note the address of rename!).

For this particular test it is irrelevant if you run optool on the library or not, BTW.

*) There Chromium and some other applications do things like using sendfile or rename assuming that they're remaining within a single filesystem. That isn't always a valid assumption, so actions like installing or upgrading extensions used to fail for me because my /tmp, ~/.cache and ~/.config are on different ZFS datasets...

[Wowfunhappy]

For your test, can't you use dyld_interpose? The reason I typically can't use this is I'm trying to add a symbol that does not already exist, in which case interposing won't work.

[krackers]

it simply becomes a copy of the library it re-exports.

Fwiw this is what I would expect should happen, since DYLD (or any dynamic loader really) should aim to avoid loading duplicate versions of a library. So I'd expect that only one of the two would actually be loaded, and which one it is probably depends on the ordering or something.

[Wowfunhappy]

It certainly is, but compiling it as ObjC will probably link in the ObjC runtime, which might have some side-effects(?). It's probably also slower but that might not be apparent for such a small file.

The unity example is compiled like this:

clang -compatibility_version 9999 -o /Path/To/UnityMavericksWorkarounds.dylib -dynamiclib /Path/To/UnityMavericksWorkarounds.m

How do I compile it as a C file instead of an Objective-C file?

That's true of course. If you're just injecting a library that adds a function there shouldn't be much side-effects though, and on Mac it will also depend on exactly how those processes are spawned.

So let's take the Dictionary app. I have an mitm proxy set up to work around Mavericks's outdated https support. Unfortunately, Dictionary.app ignores OS X's system proxy settings, so I can't connect to Wikipedia.

Krackers wrote a ProxyFix library to fix this:

#include <stdio.h>
#include <objc/runtime.h>
#include <Foundation/Foundation.h>
#include <dlfcn.h>
#include <AppKit/AppKit.h>

#define DYLD_INTERPOSE(_replacement,_replacee) \
	__attribute__((used)) static struct{ const void* replacement; const void* replacee; } _interpose_##_replacee \
				__attribute__ ((section ("__DATA,__interpose"))) = { (const void*)(unsigned long)&_replacement, (const void*)(unsigned long)&_replacee };

CFReadStreamRef myCFReadStreamCreateForHTTPRequest(CFAllocatorRef alloc, CFHTTPMessageRef request) {
	CFReadStreamRef ref = CFReadStreamCreateForHTTPRequest(alloc, request);
	CFDictionaryRef systemProxyDict = CFNetworkCopySystemProxySettings();
	CFReadStreamSetProperty(ref, kCFStreamPropertyHTTPProxy, systemProxyDict);
	return ref;
}

DYLD_INTERPOSE(myCFReadStreamCreateForHTTPRequest, CFReadStreamCreateForHTTPRequest);

To use this, I modified Dictionary.app's info.plist:

<key>LSEnvironment</key>
<dict>
    <key>DYLD_INSERT_LIBRARIES</key>
    <string>@executable_path/../Frameworks/ProxyFix.dylib</string>
</dict>

This works great! But if I try to print something, and in the print dialog select "Open PDF in Preview", Preview.app will crash!

Process:         Preview [64118]
Path:            /Applications/Preview.app/Contents/MacOS/Preview
Identifier:      com.apple.Preview
Version:         7.0 (826.4)
Build Info:      Preview-826004000000000~2
Code Type:       X86-64 (Native)
Parent Process:  launchd [170]
Responsible:     Preview [64118]
User ID:         501

Date/Time:       2024-03-07 18:08:01.978 -0500
OS Version:      Mac OS X 10.9.5 (13F1911)
Report Version:  11
Anonymous UUID:  552BC867-8AAA-03C1-82A7-8F2945B54FF3


Crashed Thread:  0

Exception Type:  EXC_BREAKPOINT (SIGTRAP)
Exception Codes: 0x0000000000000002, 0x0000000000000000

Application Specific Information:
dyld: launch, loading dependent libraries

Dyld Error Message:
  could not load inserted library '@executable_path/../Frameworks/ProxyFix.dylib' because image not found

Binary Images:
       0x102fef000 -        0x1031d2fef  com.apple.Preview (7.0 - 826.4) <40BAB370-D0EA-399C-94E0-F67F8131AAC7> /Applications/Preview.app/Contents/MacOS/Preview
    0x7fff64360000 -     0x7fff64393887  dyld (239.5) <1D3130FE-FE7E-3C4C-8E74-EB51895B6BA5> /usr/lib/dyld

[RJVB]

For your test, can't you use `dyld_interpose`?

I don't see how? I wasn't familiar with the feature but it looks to be a formalised/modern version of the overloading I do (by getting the function address with `dlsym(2)` and then calling that in a new function of the same name). I also see you still need DYLD_INSERT_LIBRARIES, undoubtedly with the same caveats as always re: two-level namespace applications. My goal here was just to get a quick indication of whether the idea I had *could* work. From what I saw it simply can't because the symbols of interest are no longer visible in a library if you change its ID. I have not found a reason to use this particular wrapper library on Darwin. The re-export trick with optool would probably be useful if one only needed to provide the backported functions in the MacPorts LegacySupport library to pre-built binaries.

[Wowfunhappy]

undoubtedly with the same caveats as always re: two-level namespace applications.

No, that's just it—I don't entirely understand how dyld_interpose works, but it has no trouble with two-level namespaces.

[krackers]

Yes, dyld_interpose is able to handle two-level namespaces properly, because DYLD is actually aware of it when it binds symbols and gives it priority over everything else allowing it to pierce the namespace barrier

[krackers]

The re-export trick with optool would probably be useful if one only needed to provide the backported functions in the MacPorts LegacySupport library to pre-built binaries

IIRC macports already does this, they offer a variant which reexports system symbols precisely for easy drop in compatability

[RJVB]

On Thursday March 07 2024 15:11:15 Jonathan wrote: The unity example is compiled like this: `clang -compatibility_version 9999 -o /Path/To/UnityMavericksWorkarounds.dylib -dynamiclib /Path/To/UnityMavericksWorkarounds.m` How do I compile it as a C file instead of an Objective-C file?

AFAIK Clang looks at the file extension to determine what dialect to use, absent more specific instructions. A pure C file has a `.c` extension... ;)

So let's take the Dictionary app. I have an mitm proxy set up to work around Mavericks's outdated https support.

Unrelated, but what problems does that solve exactly - the endless errors related to calendars and address books for instance (which AFAIK are more a problem of outdated certificates)?

<key>LSEnvironment</key> <dict> <key>DYLD_INSERT_LIBRARIES</key> ***@***.***_path/../Frameworks/ProxyFix.dylib</string> </dict>

[...]

Dyld Error Message: could not load inserted library ***@***.***_path/../Frameworks/ProxyFix.dylib' because image not found

You have 2 solutions here: 1) copy the ProxyFix.dylib library into the Preview.app bundle too (where it wouldn't normally be used) 2) give ProxyFix.dylib an absolute path ID and install it at that location. A variant would be to use just "ProxyFix.dylib" as the ID and install the library in one of the standard locations where dyld searches by default (this variant *may* require building it as a framework, I can't remember).

[Wowfunhappy]

The problem is not outdated certificates (which is easy to fix), but outdated cipher suites. Many/most websites, including Wikipedia, do not support old cipher suites because they are considered insecure. Mavericks does not support any modern cipher suites. So HTTPS doesn't work in apps that use Apple's native libraries.

You have 2 solutions here:

I consider both of these solutions worse than leaving the problem unsolved. 🙂 Everything should live in the app bundle, and the problem is moderately difficult to run into. But this is why I shy away from using DYLD_INSERT_LIBRARIES in most cases. (Except for testing where as you said it's definitely useful!)

AFAIK Clang looks at the file extension to determine what dialect to use, absent more specific instructions. A pure C file has a .c extension... ;)

Ah! Okay. I will try that at some point...

[krackers]

@Wowfunhappy You can fix this by unsetting the env once it's loaded, inside a __constructor___ block

[krackers]

Also as to issue it fixes, it is a Dictionary.app specific bug where it does not obey system proxy settings (or rather more of a CoreFoundation quirk, it's low-level enough that connections made using it directly will not lookup system proxy settings. But apparently no one except the authors of Dictionary.app were insane enough to do their networking directly in corefoundation)

[Wowfunhappy]

@krackers Thanks, now I remember you've told me about this before and I need to actually do it. Note to self I need to use https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/unsetenv.3.html

[RJVB]

Mavericks does not support any modern cipher suites. So HTTPS doesn't work in apps that use Apple's native libraries.

I had a look at "mitn proxy", found a site that was jubilantly enthusiastic about what their product can do but didn't understand a word of it. Or rather, I understood the words but not how it could be relevant for everyday solutions that don't involve reverse-engineering code or (presumably) jail-breaking iOS devices or hacking them in other ways ;)

I consider both of these solutions worse than leaving the problem unsolved. 🙂 Everything should live in the app bundle, and the problem is moderately difficult to run into.

The "open PDF in Preview" feature is one of the printing functions I use most commonly, so I wouldn't say the particular example is "moderately difficult to run into" ;) And in that particular example you can still set the ID to the absolute path to the library inside the Dictionary.app bundle. The thing with Apple's app bundles is that they're claimed to be position independent but in reality tend to be full with assumptions that they do in fact live in their default location. IIRC Dictionary.app is one of them so you would not be crippling it any more than it already is. Anyway, if you're hacking a system like this you better (IMHO) make sure that the code you inject is harmless beyond its intended purpose. I haven't looked at all at what that Proxy library does exactly but I would assume that it's written in such a way that it doesn't change anything for applications that do obey the system proxy settings. In fact, personally I'd want this kind of patch to be applied in such a way that any application that might need it benefits from it automatically, provided the above holds.

[krackers]

not how it could be relevant for everyday solutions

it's effectively reverse proxy running locally on your machine. Since osx does not support modern cipher suite (or tls 1.3 for that matter), it can stand as a go-between, and will make the connection on your behalf, then send you the info over downgraded tls. This is only relevant for applications which use default osx securetransport SSL library, obviously chrome or CLI apps which bundle gnutls/openssl don't need this.

(Theoretically since securetransport is open-source it should be possible to build your own and replace the library wholesale but apparently source dumps are incomplete, funhappy ran into issues with this)

[Wowfunhappy]

@krackers Why is this a "reverse proxy" and not a "forward proxy"? I never understood this terminology. :)

[Wowfunhappy]

(Theoretically since securetransport is open-source it should be possible to build your own and replace the library wholesale but apparently source dumps are incomplete, funhappy ran into issues with this)

For reference, my messy very bad no-good attempt at this: https://apple.stackexchange.com/a/399650/150839

[RJVB]

On Friday March 08 2024 13:29:52 krackers wrote: IIRC macports already does this, they offer a variant which reexports system symbols precisely for easy drop in compatability

Not a variant (in the MacPorts sense of the term) but a copy of the library that re-exports libSystem, which gets used exactly as I imagined. I had forgotten I'd noticed that version of the library but can't recall that I ever looked at what it was used for... Thanks for reminding me :)

[RJVB]

it's effectively reverse proxy running locally on your machine. Since osx does not support modern cipher suite (or tls 1.3 for that matter), it can stand as a go-between, and will make the connection on your behalf, then send you the info over downgraded tls.

I guessed it might be that. I guess it can't hurt to set this up on my own Mac, is there a recipe somewhere I can follow (and doesn't it increase the CPU overhead for traffic that doesn't go through Apple's https APIs)?

[Wowfunhappy]

I direct less technical users to use "Legacy Mac Proxy" on https://jonathanalland.com/old-osx-projects.html, but you may prefer to set up it up yourself. Under the hood, this is using the Squid proxy server. Feel free to extract the installer package and take a look at squid.conf.

and doesn't it increase the CPU overhead for traffic that doesn't go through Apple's https APIs

Yes, it does. There's just no perfect way to do this, unfortunately. The proxy also isn't entirely without other side effects, for example I can't seem to make pip work without extra environment variables: https://forums.macrumors.com/threads/fixing-https-issues-on-old-versions-of-os-x.2281326/page-13?post=32582522#post-32582522

If you primarily use OS X as a Unix box with software linked against its own SSL libraries, you may not want to use a proxy. I use lots of Mac-native Cocoa apps, so a proxy is basically required. I need Dashboard and Calendar.app and Mail.app and SourceTree and Transmission to connect to the modern internet!

I'd love an alternate solution. Krackers once suggested a SIMBL plugin which redirects the relevant Objective-C calls to something based on OpenSSL, which is something I would try in an alternate universe where I had unlimited free time...

[Wowfunhappy]

Also, Squid may be overkill. If I was to start over, I would probably try writing a very minimal proxy in Go: https://forums.macrumors.com/threads/fixing-https-issues-on-old-versions-of-os-x.2281326/page-10?post=31873567#post-31873567

[RJVB]

So ideally you'd use the opposite of that ProxyLib patch to get only specific applications to use a proxy otherwise NOT configured at the system level? That'd avoid the overhead issue. Or ... use a dedicated little server to run the proxy on?

I make extensive [...] Calendar.app

I notice a lot of errors that appear to come from that, and the address book, but both still seem to work...

Mail.app

That one was nice in 10.6 but I stopped using it when it started to insist on downloading full messages even from the IMAP server I run on the localhost (for archiving my email)...

I would probably try writing a very minimal proxy in Go

Assuming you managed to install a new enough Go version? BTW, I just read somewhere that 10.11 dropped support for the DYLD_ env. variables, or that certain key system binaries like /bin/sh started "eliding" (unsetting?) them. Is that correct, even when you disable the SIP circus?

[Wowfunhappy]

Or ... use a dedicated little server to run the proxy on?

Yes, you could do that! Set it up on your router, or a Raspberry Pi or another Linux box or something!

I prefer to run it locally, because I have a Mavericks laptop that I travel with, and also because the CPU usage is trivial (I just didn't want to deny that it exists).

That one was nice in 10.6 but I stopped using it when it started to insist on downloading full messages even from the IMAP server I run on the localhost (for archiving my email)...

Assuming you managed to install a new enough Go version?

I think Go ≤ 1.18 works on 10.9 with MacPorts Legacy Support? That's much more than recent enough. Might need to go back further to retain Snow Leopard support; that still shouldn't be a problem.

I notice a lot of errors that appear to come from that, and the address book, but both still seem to work...

It depends on the service you're connecting to. Personally I can't connect to my work calendar without the proxy. Apple Mail "works" with many services without the proxy (I don't actually think it affects IMAP connections since they aren't HTTP), but many remote images won't load, which is something I care about.

BTW, I just read somewhere that 10.11 dropped support for the DYLD_ env. variables, or that certain key system binaries like /bin/sh started "eliding" (unsetting?) them. Is that correct, even when you disable the SIP circus?

Sorry, I basically don't know about anything newer than 10.9.

[krackers]

So ideally you'd use the opposite of that ProxyLib patch to get only specific applications to use a proxy...

Yeah would technically also work, although all proper Cocoa applications should transparently work with proxies, it's only apparently pip (which isn't even a cocoa app) that breaks for some reason because it tries to be too clever for its own good or something.

Assuming you managed to install a new enough Go version?

Well even if you stick with "old go", go 1.11 is serviceable enough to get things done. But yes, thanks again to macports legacy support you can get 1.18 working without any issue (although things don't quite "just work" anymore in terms of compilation, if you go this route it's easier to install go on some other platform, compile the binary, than manually inject the legacy support dylib).

BTW, I just read somewhere that 10.11 dropped support for the DYLD_ env. variables, or that certain key system binaries like /bin/sh started "eliding" (unsetting?) them. Is that correct, even when you disable the SIP circus?

Maybe you are referring to hardened runtime. I don't know much about this other than it exists, and it's completely pointless because you can resign binaries to strip them of this hardened runtime protection.

[RJVB]

>So ideally you'd use the opposite of that ProxyLib patch to get only specific applications to use a proxy... Yeah would technically also work, although all proper Cocoa applications should transparently work with proxies

The idea for this was that you'd "patch" all Cocoa applications that you needed to do modern https with so they'd use a "hidden" proxy. A bit like what Windscribe do with their VNC browser plugin

Maybe you are referring to hardened runtime.

No, the source is a readme from one of the devs working on GCC/Darwin; I think it was concerned specifically with DYLD_LIBRARY_PATH which is evidently a loss not to have in complex build systems. But it did mention "the DYLD_ variables"...

[Wowfunhappy]

The idea for this was that you'd "patch" all Cocoa applications that you needed to do modern https with so they'd use a "hidden" proxy. A bit like what Windscribe do with their VNC browser plugin

If I was going with an approach like this, I think I'd want to forget about the proxy and swizzle/interpose the functions in NSURLConnection and/or Foundation.

The advantage of the current approach is that it is fundamentally not a hack, I'm just using features that are natively supported in OS X. OS X supports proxy servers, and OS X supports adding certificates via Keychain Access, so telling OS X to use a proxy server which re-signs all traffic to use a custom certificate (and just so happens to also support modern cipher suites on the other end) is all Apple-supported functionality! It's not even a particularly novel thing to do—aiui mitm proxies are often used by large enterprises (for better or worse) to monitor and filter web traffic on their premises.

And for the most part it all works transparently! When it doesn't work, it's fundamentally because of a bug in the original software, like the Dictionary app flat out ignoring the system proxy. (What I'm doing to the Dictionary app is a hack, but that's a one-off targeted bug fix.)

[RJVB]

On Sat, 9 Mar 2024 at 17:03, Jonathan ***@***.***> wrote: I direct less technical users to use "Legacy Mac Proxy" on https://jonathanalland.com/old-osx-projects.html, but you may prefer to set up it up yourself.

Checking the Readme on that dmg I see you suggest pointing Firefox to the proxy. That surprises me; the v78 based version I run isn't that recent but I don't think I ever came across websites it cannot connect to. Certain sites don't work (well) anymore but that's for other reasons AFAICT. Got an example?

…

Message ID: ***@***.*** com>

[Wowfunhappy]

You have that backwards, I suggest setting Firefox to ignore the proxy!

[RJVB]

You have that backwards, I suggest setting Firefox to _ignore_ the proxy!

OOPPS :)