docs: Add caveats for glibc version targeting feature

[polarathene]

I have summarized my findings and linked to the issue that details them for more information.

This helps communicate some surprises, both with dynamic and static linking support when considering cargo zigbuild as a solution. Some are UX issues with cargo zigbuild command, while others are upstream quirks with zig cc.

[messense]

This is expected because even if you want to use glibc 2.32, you can still end up being compatible with glibc 2.31 because all of the versioned symbols you linked are < 2.32.

glibc doesn't increase symbols versions for every functions in a new minor version, only these with breaking changes.

[polarathene]

This is expected because even if you want to use glibc 2.32, you can still end up being compatible with glibc 2.31 because all of the versioned symbols you linked are < 2.32.

My intent is just to highlight the difference in behaviour from cargo build when dynamically linking that same glibc version, cargo zigbuild is being more lenient.

That may be an improvement if it's not impacting functionality. I just wanted to highlight that there was a difference, so there isn't any confusion if it built with the intended version when it doesn't fail under the same conditions that a build via cargo build would.

How would you word it differently?

EDIT: As covered in last section below

• The difference was Zig doesn't align with glibc 2.32 which raised the symbol version for pthread_getattr_np due to moving the symbol from libpthread to libc.
• This is also apparently important for Rust to handle proper stack overflow detection, so I'm not sure if Zig should be misaligned with that? 🤷‍♂️

---

Reproduction (static vs dynamic vs zig dynamic)

Below is a reproduction for cargo build based on rust-lang/libc#2054 (comment) regarding openpty usage with glibc 2.32.

• A static build with a lower version works correctly on older glibc versions.
• A static build with newer versions works correctly on older glibc versions, and also on newer glibc (instead of segfaulting).
• Dynamic linking with glibc 2.32 will prevent it running on earlier glibc, but it'll work fine on newer versions.
• Dynamic linking with a version lower than glibc 2.32 becomes compatible with earlier versions of glibc too.
• However cargo zigbuild can dynamically link with:
  • glibc 2.32 target and the failure on earlier glibc is avoided.
  • glibc >2.32 target will introduce failure with a dependency requirement on glibc 2.33.

Reproduction example

use libc;
use std::{mem,ptr};
fn main() {
    let mut slave = mem::MaybeUninit::<libc::c_int>::uninit();
    let mut master = mem::MaybeUninit::<libc::c_int>::uninit();
    let p;

    unsafe {
       p = libc::openpty(
            master.as_mut_ptr(),
            slave.as_mut_ptr(),
            ptr::null_mut(),
            ptr::null_mut(),
            ptr::null_mut()
        );
    }

    println!("p:{}",p);
}

[package]
name = "example"
version = "0.1.0"
edition = "2021"

[dependencies]
libc = "0.2.153"

# Fedora 33 container with glibc-static package:
$ ldd --version
ldd (GNU libc) 2.32

$ RUSTFLAGS="-C target-feature=+crt-static" cargo build --release --target x86_64-unknown-linux-gnu

# Fedora 31 container:
$ ldd --version
ldd (GNU libc) 2.30
# Fedora 32 container:
$ ldd --version
ldd (GNU libc) 2.31

# Same runtime output below for both containers

# static build:
# https://github.com/rust-lang/libc/issues/2054#issuecomment-829119942
$ ./example
example: dl-call-libc-early-init.c:37: _dl_call_libc_early_init: Assertion `sym != NULL' failed.

# dynamic build (doesn't fail with `cargo zigbuild`):
$ ./example
./example: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by ./example)

# Fedora 40 container:
$ ldd --version
ldd (GNU libc) 2.39

# static build:
$ ./example
Segmentation fault

# dynamic build:
$ ./example
p:0

When building on Fedora 32 (glibc 2.31) container instead:

• Static builds will:
  • Segfault on both Fedora 33 and 40.
  • No failure on Fedora 31 container (glibc 2.30)
• Dynamic builds:
  • No GLIBC_2.31 not found on Fedora 31, works correctly.
  • Fedora 33 and 40 both work correctly too.
  • cargo zigbuild for 2.32

However when a Fedora 31 (glibc 2.30) container runs a build via Fedora 34 (glibc 2.33):

• Static is successful.
• Dynamic fails:

  ./example: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by ./example)
  ./example: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by ./example)
  

A static build with glibc 2.33 is successful across all of these Fedora containers.

• That is because glibc 2.33 was when the openpty method received proper static support.
• Which clearly explains the varied behaviour of dynamic/static builds at runtime with earlier versions.
• This comment suggests that previously, static builds for the method would have been limited to host with that specific glibc version, but that's not quite accurate given the Fedora 31 + 32 compatibility despite glibc 2.30 vs 2.31. So for static builds it depends on their static build support for that glibc target version.

---

Dynamic linking differences

TL;DR:

• cargo zigbuild will match cargo build when linking to glibc 2.33 with the built program failing to run on hosts with earlier glibc.
• Meanwhile only cargo build would also have that same failure experience when building with glibc 2.32 version too, but otherwise fine with glibc 2.31 build running on host with glibc 2.30 🤔

I tracked it down, the difference doesn't appear related to openpty. I'm not entirely sure why the symbol versions differ between cargo build and cargo zigbuild, presumably Zig knows better to not match the pthread_getattr_np version for glibc 2.32? 🤷‍♂️ (or it's a bug)

Dynamic linking differences (resolved)

UPDATE: I've learned how to inspect the symbol versions to understand what was enforcing the min supported glibc version between cargo build and cargo zigbuild:

• cargo build had the min glibc versions of 2.32 because of pthread_getattr_np. Prior versions of glibc version this symbol much lower, which matches cargo zigbuild.
• glibc 2.33 is enforced by cargo zigbuild as it shares the same symbol version bump for stat64 (technically renamed, not just version bump?)

What introduces a difference to cargo zigbuild between glibc 2.32 vs 2.33 version targeting? Presumably something related to this commit (that was identified as relevant to the compatibility for static builds) (UPDATE: findings below)

From the Fedora mailing list June 2020:

I am surprised that this is not handled by symbol versioning.
This function seems to exist for long long time, so why did it break just now?

The symbol was moved from libpthread to libc.
The new symbol version is required so that newly linked applications depend on glibc 2.32 as the minimum glibc version.

Related bug report:

Rust does need that symbol for stack overflow detection. I found the specific glibc change here

While this may seem Fedora specific due to the container image choice and reference links, Debian 12 (Bookworm) via rust:latest with glibc 2.36 also produces this min requirement.

# Build hosts with dynamic linking:

# Fedora 33:
$ readelf -W --version-info --dyn-syms target/x86_64-unknown-linux-gnu/release/example | grep 2.32
     9: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_getattr_np@GLIBC_2.32 (5)
  008:   2 (GLIBC_2.2.5)   5 (GLIBC_2.32)    4 (GLIBC_2.2.5)   6 (GLIBC_2.18)
  0x0110:   Name: GLIBC_2.32  Flags: none  Version: 5
# For comparison to glibc 2.33 Fedora 34 change:
readelf -W --version-info --dyn-syms target/x86_64-unknown-linux-gnu/release/example | grep stat64
    19: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __xstat64@GLIBC_2.2.5 (2)
    33: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __fxstat64@GLIBC_2.2.5 (2)
    59: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __lxstat64@GLIBC_2.2.5 (2)

# Fedora 34:
# Same:
$ readelf -W --version-info --dyn-syms target/x86_64-unknown-linux-gnu/release/example | grep 2.32
     9: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_getattr_np@GLIBC_2.32 (5)
  008:   2 (GLIBC_2.2.5)   5 (GLIBC_2.32)    4 (GLIBC_2.2.5)   6 (GLIBC_2.18)
  0x0110:   Name: GLIBC_2.32  Flags: none  Version: 5
# glibc 2.33 requirement added due to stat64 + fstat64 symbol versions:
$ readelf -W --version-info --dyn-syms target/x86_64-unknown-linux-gnu/release/example | grep 2.33
    20: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND stat64@GLIBC_2.33 (9)
    59: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fstat64@GLIBC_2.33 (9)
  014:   9 (GLIBC_2.33)    4 (GLIBC_2.2.5)   2 (GLIBC_2.2.5)   2 (GLIBC_2.2.5)
  038:   2 (GLIBC_2.2.5)   4 (GLIBC_2.2.5)   4 (GLIBC_2.2.5)   9 (GLIBC_2.33)
  0x00e0:   Name: GLIBC_2.33  Flags: none  Version: 9


# Fedora 33 with `cargo zigbuild`:
$ cargo zigbuild --release --target x86_64-unknown-linux-gnu.2.33
# Nothing versioned for glibc 2.32:
$ readelf -W --version-info --dyn-syms target/x86_64-unknown-linux-gnu/release/example | grep 2.32
# glibc 2.33 target carries the same stat64 change:
$ readelf -W --version-info --dyn-syms target/x86_64-unknown-linux-gnu/release/example | grep 2.33
    25: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fstat64@GLIBC_2.33 (7)
    37: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND stat64@GLIBC_2.33 (7)
  018:   2 (GLIBC_2.2.5)   7 (GLIBC_2.33)    5 (GLIBC_2.2.5)   5 (GLIBC_2.2.5)
  024:   9 (GLIBC_2.3)     7 (GLIBC_2.33)    5 (GLIBC_2.2.5)   5 (GLIBC_2.2.5)
  0x00c0:   Name: GLIBC_2.33  Flags: none  Version: 7
# The `cargo build` pthread symbol linked to glibc 2.32, while zig doesn't:
$ readelf -W --version-info --dyn-syms target/x86_64-unknown-linux-gnu/release/example | grep pthread_getattr
    18: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_getattr_np@GLIBC_2.2.5 (5)

# Fedora 32, also has low pthread version like `cargo zigbuild` produced:
$ readelf -W --version-info --dyn-syms target/x86_64-unknown-linux-gnu/release/example | grep pthread_getattr
     7: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_getattr_np@GLIBC_2.2.5 (4)

# Reference
# https://linux.die.net/man/1/nm
# https://johannst.github.io/notes/development/symbolver.html

# Fedora 33 + 32,
# `@@` denotes default version to link for static build:
# `U` => Undefined symbol
$ nm target/x86_64-unknown-linux-gnu/release/example | grep openpty
                 U openpty@@GLIBC_2.2.5

# Fedora 34:
$ nm target/x86_64-unknown-linux-gnu/release/example | grep openpty
                 U openpty@GLIBC_2.2.5

# Fedora 33 with `cargo zigbuild`, no symbol version:
nm target/x86_64-unknown-linux-gnu/release/example | grep openpty
                 U openpty

For static builds, _dl_open will be always present when linking glibc, regardless if your program would use it or not:

$ readelf -a target/x86_64-unknown-linux-gnu/release/example | grep dl_open
  3512: 00000000000a4fb4     0 NOTYPE  LOCAL  HIDDEN    11 .annobin__dl_open.end
  3513: 00000000000a5120  2623 FUNC    LOCAL  DEFAULT   11 dl_open_worker
  3517: 00000000000a511f     0 NOTYPE  LOCAL  HIDDEN    11 .annobin_dl_open[...]
  3518: 00000000000a5b5f     0 NOTYPE  LOCAL  HIDDEN    11 .annobin_dl_open[...]
  3688: 000000000014b700    32 OBJECT  LOCAL  DEFAULT   26 _dl_open_hook
  5059: 00000000000a4d60   596 FUNC    LOCAL  DEFAULT   11 _dl_open

# Roughly equivalent information:
$ nm -a target/x86_64-unknown-linux-gnu/release/example | grep _dl_open
00000000000aa903 t .annobin___libc_register_dl_open_hook.end
00000000000aa8bf t .annobin___libc_register_dl_open_hook.start
00000000000a4fb4 t .annobin__dl_open.end
00000000000a4d52 t .annobin__dl_open.start
00000000000a5b5f t .annobin_dl_open_worker.end
00000000000a511f t .annobin_dl_open_worker.start
00000000000aa8c0 t __libc_register_dl_open_hook
00000000000a4d60 t _dl_open
000000000014b700 d _dl_open_hook