environment.memoryAllocator no longer sets LD_PRELOAD. How can it now be disabled for specific commands?

[8573]

Issue description

Originally, the NixOS configuration option environment.memoryAllocator.provider worked by pointing the environment variable LD_PRELOAD at the chosen memory allocator library — in my case, scudo. I could disable this for a given command foo by running LD_PRELOAD= foo at the command line.

As of commit 44b6999, this has been changed no longer to use the environment variable LD_PRELOAD but rather the file /etc/ld-nix.so.preload. Some searching of Google has not shown me any way to turn off this sort of preloading for a specific command. Is there such a way?

cc @joachifm

Technical details

• system: "x86_64-linux"
• host os: Linux 4.19.58-hardened, NixOS, 19.09pre185176.1036dc66416 (Loris)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.2.2
• channels(root): "nixos-19.09pre185176.1036dc66416"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

[8573]

For example, I would run LD_PRELOAD= inkscape and LD_PRELOAD= gimp because scudo (and graphene-hardened) seemed to prevent these graphical applications from working acceptably (with scudo, they crash frequently, including whenever I tried to use a file-picker dialog, and I think it was worse with graphene-hardened).

[8573]

The hardened allocators also seem to break man and, now, programs written in Rust (which seemed to work fine with LD_PRELOAD=/path/to/scudo).

[joachifm]

Setting LD_PRELOAD to the libc allocator should still work, does it not?

I guess this comes down to whether we want the allocator to apply as broadly & robustly as possible or make it easy to override ad-hoc ... If users want hardened allocators but also want easy ad-hoc overrides I guess we can revert ld-so.preload to regain that ability, but I hadn't really envisioned this as a feature suitable for interactive systems.

As it stands, it seems to me it's going to be difficult to provide both good ux for interactive systems and good coverage, at least from our end, given the number of bugs in apps & the allocators themselves (unclear to me which is at fault in the cases mentioned above). Ideally, these bugs would be reported & hopefully fixed.

[joachifm]

Another point in favor of ld-so.preload is that it works for setuid binaries. I really think it's the superior solution, despite being a little inconvenient.

[8573]

@joachifm:

Setting LD_PRELOAD to the libc allocator should still work, does it not?

I thought programs were made to use the libc allocator by there being no other allocator preloaded. Is there a file containing the libc allocator somewhere whose path I can put in LD_PRELOAD to test this suggestion?

[...] I hadn't really envisioned this as a feature suitable for interactive systems.

Ah, I had wondered whether that might be the case. Could this be documented in the description of environment.memoryAllocator.provider?

Also, it seems I hadn't tested thoroughly enough regarding the problem with Rust programs I mentioned: it does seem to be a problem caused (or, should I say, revealed) by the hardened allocator in general rather than by the specific method of preloading the hardened allocator, as Rust programs seem to segfault with both methods of preloading Scudo and run fine with the default libc allocator. (I would suspect that the reason they formerly worked fine is that, until Rust version 1.32, Rust programs used jemalloc rather than the system allocator — I thought I had tested since that change, but I guess not.)

That Rust programs, which aren't necessarily interactive, are broken would seem to make this relevant even if the hardened allocators are intended only for non-interactive installations of NixOS, but—

Ideally, these bugs would be reported & hopefully fixed.

—I agree that it would be best to fix the underlying bugs rather than hiding them.

[joachifm]

@8573

First of all, thank you for testing this :)

Ah, I had wondered whether that might be the case. Could this be documented in the description of environment.memoryAllocator.provider?

Yes. I've done a poor job communicating the potential breakage from the hardened profile, but that will be fixed in due time. To elaborate a little, the hardened defaults err on the side of breakage, with the expectation that admins will relax constraints that are unsuitable for their needs.

I thought programs were made to use the libc allocator by there being no other allocator preloaded. Is there a file containing the libc allocator somewhere whose path I can put in LD_PRELOAD to test this suggestion?

I was thinking of adding libc.so.6 or whatever, but I was perhaps too quick to respond here; I'm unsure there is a way to prevent ld-so.preload from being used. What I suggested implies that symbols overridden earlier in the sequence are unaffected by later overrides (ld-so.preload is last), but on reflection that doesn't make sense, sorry.

graphene-hardened-malloc upstream suggests running apps in mount namespace with empty ld-so.preload file (or without the hardened malloc .so). Maybe firejail or somesuch could be used to do this conveniently.

That Rust programs, which aren't necessarily interactive, are broken would seem to make this relevant even if the hardened allocators are intended only for non-interactive installations of NixOS, but—

Uncovering bugs is the reason why this is default-on in the hardened profile on unstable. It likely won't make it into the upcoming release, there is just too much breakage, unfortunately. It would be good to have a tracking issue where we could collect known problems with debug output/whatever.

[delroth]

Note that at least with Scudo you have the option of disabling parts of the hardening via environment variables, e.g. SCUDO_OPTIONS="DeleteSizeMismatch=0". This does not directly answer your question but might provide what you need to get more troublesome apps running.

(e.g. I discovered yesterday that ffmpeg is broken by design and mixes memalign/realloc because "it works on all the systems we've tested". https://trac.ffmpeg.org/ticket/6403 . Everything using libsigc++ is also buggy.)

[8573]

Note that at least with Scudo you have the option of disabling parts of the hardening via environment variables, e.g. SCUDO_OPTIONS="DeleteSizeMismatch=0". This does not directly answer your question but might provide what you need to get more troublesome apps running.

Ah, I didn't know about that. SCUDO_OPTIONS='DeallocationTypeMismatch=0' allows man to work, but even disabling all the Scudo features doesn't save the Rust programs.

[delroth]

Do you have an easy repro for the Rust case you're talking about? I've been testing alacritty recently and it's working fine.

(Happy to take the off-topic elsewhere. If there is an easy repro case, we should just file a bug to upstream Rust.)

[8573]

@delroth: rustc -vV, which should print—

rustc 1.34.2 (6c2484dc3 2019-05-13)
binary: rustc
commit-hash: 6c2484dc3c532c052f159264e970278d8b77cdc9
commit-date: 2019-05-13
host: x86_64-unknown-linux-gnu
release: 1.34.2
LLVM version: 8.0

—instead prints—

rustc 1.34.2 (6c2484dc3 2019-05-13)
binary: rustc
commit-hash: 6c2484dc3c532c052f159264e970278d8b77cdc9
commit-date: 2019-05-13
host: x86_64-unknown-linux-gnu
release: 1.34.2
Segmentation fault

I doubt I can be very helpful debugging it, as I don't know how to use a debugger (I'm not a developer by trade).

My own Rust irc-bot also doesn't work under Scudo: when I try to run the executable file, which is clearly there, the shell (whether Bash or zsh) claims the file doesn't exist.

[stale]

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the
   related code and @ mention them in a comment.
2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
   irc.freenode.net.