macos: rustc built without jemalloc sporadically fails with realloc failure

[ehuss]

Building rustc on macos without jemalloc (the default in config.toml) will trigger frequent SIGABRT. The error is:

rustc(14225,0x70000fab4000) malloc: *** error for object 0x600005c70800: pointer being realloc'd was not allocated
rustc(14225,0x70000fab4000) malloc: *** set a breakpoint in malloc_error_break to debug

The backtrace is:

Backtrace

* thread #2, name = 'rustc', stop reason = breakpoint 1.1
  * frame #0: 0x00007ff80c53f5d7 libsystem_malloc.dylib`malloc_error_break
    frame #1: 0x00007ff80c531376 libsystem_malloc.dylib`malloc_vreport + 440
    frame #2: 0x00007ff80c5345ed libsystem_malloc.dylib`malloc_report + 151
    frame #3: 0x00007ff80c525542 libsystem_malloc.dylib`realloc + 328
    frame #4: 0x000000010c886c64 librustc_driver-cea70d83b3706378.dylib`_RINvNtCsk2H0Gr1Y1z1_5alloc7raw_vec11finish_growNtNtB4_5alloc6GlobalECsetNeWcxqv12_12rustc_expand + 52
    frame #5: 0x000000010c88721d librustc_driver-cea70d83b3706378.dylib`_RNvMs_NtCsk2H0Gr1Y1z1_5alloc7raw_vecINtB4_6RawVecNtNtCsetNeWcxqv12_12rustc_expand3mbe9TokenTreeE16reserve_for_pushBP_ + 125
    frame #6: 0x000000010c7f7222 librustc_driver-cea70d83b3706378.dylib`_RNvNtNtCsetNeWcxqv12_12rustc_expand3mbe6quoted5parse + 4002
    frame #7: 0x000000010c7f63fe librustc_driver-cea70d83b3706378.dylib`_RNvNtNtCsetNeWcxqv12_12rustc_expand3mbe6quoted5parse + 382
    frame #8: 0x000000010c8138eb librustc_driver-cea70d83b3706378.dylib`_RINvXs0_NtNtNtCsasXF9FuNFFy_4core4iter8adapters3mapINtB6_3MapINtNtNtBc_5slice4iter4IterNtNtNtCsetNeWcxqv12_12rustc_expand3mbe12macro_parser10NamedMatchENCNvNtB1r_11macro_rules25compile_declarative_macros_0ENtNtNtBa_6traits8iterator8Iterator4folduNCINvNvB3i_8for_each4callNtB1r_9TokenTreeNCNvXs_NtNtCsk2H0Gr1Y1z1_5alloc3vec11spec_extendINtB4K_3VecB4l_EINtB4I_10SpecExtendB4l_BN_E11spec_extend0E0EB1t_ + 267
    frame #9: 0x000000010c861d47 librustc_driver-cea70d83b3706378.dylib`_RNvXNtNtCsk2H0Gr1Y1z1_5alloc3vec14spec_from_iterINtB4_3VecNtNtCsetNeWcxqv12_12rustc_expand3mbe9TokenTreeEINtB2_12SpecFromIterBU_INtNtNtNtCsasXF9FuNFFy_4core4iter8adapters3map3MapINtNtNtB2b_5slice4iter4IterNtNtBW_12macro_parser10NamedMatchENCNvNtBW_11macro_rules25compile_declarative_macros_0EE9from_iterBY_ + 247
    frame #10: 0x000000010c8bd753 librustc_driver-cea70d83b3706378.dylib`_RNvNtNtCsetNeWcxqv12_12rustc_expand3mbe11macro_rules25compile_declarative_macro + 2451
    frame #11: 0x000000010b75c904 librustc_driver-cea70d83b3706378.dylib`_RNvMs_NtCs9b1qYhXJ0Cf_13rustc_resolve6macrosNtB6_8Resolver13compile_macro + 68
    frame #12: 0x000000010b70fd37 librustc_driver-cea70d83b3706378.dylib`_RNvMs3_NtCs9b1qYhXJ0Cf_13rustc_resolve19build_reduced_graphNtB5_24BuildReducedGraphVisitor12define_macro + 295
    frame #13: 0x000000010b710e08 librustc_driver-cea70d83b3706378.dylib`_RNvXs4_NtCs9b1qYhXJ0Cf_13rustc_resolve19build_reduced_graphNtB5_24BuildReducedGraphVisitorNtNtCs9cCHMvxMcfg_9rustc_ast5visit7Visitor10visit_item + 56
    frame #14: 0x000000010b783a5c librustc_driver-cea70d83b3706378.dylib`_RINvNtCs9cCHMvxMcfg_9rustc_ast5visit9walk_itemNtNtCs9b1qYhXJ0Cf_13rustc_resolve19build_reduced_graph24BuildReducedGraphVisitorEBM_ + 572
    frame #15: 0x000000010b71304c librustc_driver-cea70d83b3706378.dylib`_RNvXs4_NtCs9b1qYhXJ0Cf_13rustc_resolve19build_reduced_graphNtB5_24BuildReducedGraphVisitorNtNtCs9cCHMvxMcfg_9rustc_ast5visit7Visitor10visit_item + 8828
    frame #16: 0x000000010b783a5c librustc_driver-cea70d83b3706378.dylib`_RINvNtCs9cCHMvxMcfg_9rustc_ast5visit9walk_itemNtNtCs9b1qYhXJ0Cf_13rustc_resolve19build_reduced_graph24BuildReducedGraphVisitorEBM_ + 572
    frame #17: 0x000000010b71304c librustc_driver-cea70d83b3706378.dylib`_RNvXs4_NtCs9b1qYhXJ0Cf_13rustc_resolve19build_reduced_graphNtB5_24BuildReducedGraphVisitorNtNtCs9cCHMvxMcfg_9rustc_ast5visit7Visitor10visit_item + 8828
    frame #18: 0x000000010b7d332d librustc_driver-cea70d83b3706378.dylib`_RINvMs6_NtCsetNeWcxqv12_12rustc_expand6expandNtB6_11AstFragment10visit_withNtNtCs9b1qYhXJ0Cf_13rustc_resolve19build_reduced_graph24BuildReducedGraphVisitorEB1f_ + 685
    frame #19: 0x000000010b7519c7 librustc_driver-cea70d83b3706378.dylib`_RNvXNtCs9b1qYhXJ0Cf_13rustc_resolve6macrosNtB4_8ResolverNtNtCsetNeWcxqv12_12rustc_expand4base14ResolverExpand36visit_ast_fragment_with_placeholders + 487
    frame #20: 0x000000010c7e0cc3 librustc_driver-cea70d83b3706378.dylib`_RNvMs1_NtCsetNeWcxqv12_12rustc_expand6expandNtB5_13MacroExpander19collect_invocations + 707
    frame #21: 0x000000010c7dbe80 librustc_driver-cea70d83b3706378.dylib`_RNvMs1_NtCsetNeWcxqv12_12rustc_expand6expandNtB5_13MacroExpander21fully_expand_fragment + 256
    frame #22: 0x000000010c7dbba2 librustc_driver-cea70d83b3706378.dylib`_RNvMs1_NtCsetNeWcxqv12_12rustc_expand6expandNtB5_13MacroExpander12expand_crate + 1186
    frame #23: 0x0000000108c098bc librustc_driver-cea70d83b3706378.dylib`_RINvMNtCs3NuVQBhYW7C_13rustc_session5utilsNtNtB5_7session7Session4timeINtNtCsasXF9FuNFFy_4core6result6ResultNtNtCs9cCHMvxMcfg_9rustc_ast3ast5CrateNtCsdjh14Z7klAl_12rustc_errors13ErrorReportedENCNvNtCsBlFisMo4u3_15rustc_interface6passes20configure_and_expands_0EB3a_ + 1116
    frame #24: 0x0000000108b749c0 librustc_driver-cea70d83b3706378.dylib`_RNvNtCsBlFisMo4u3_15rustc_interface6passes20configure_and_expand + 528
    frame #25: 0x0000000108b7a354 librustc_driver-cea70d83b3706378.dylib`_RNvMs0_NtCsBlFisMo4u3_15rustc_interface7queriesNtB5_7Queries9expansion + 836
    frame #26: 0x0000000108a5520f librustc_driver-cea70d83b3706378.dylib`_RINvMs2_NtCsBlFisMo4u3_15rustc_interface7queriesNtNtB8_9interface8Compiler5enterNCNCNvCsQmx6cSBOMI_12rustc_driver12run_compilers_0s0_0INtNtCsasXF9FuNFFy_4core6result6ResultINtNtB2d_6option6OptionNtB6_6LinkerENtCsdjh14Z7klAl_12rustc_errors13ErrorReportedEEB1m_ + 1327
    frame #27: 0x0000000108a4aaa8 librustc_driver-cea70d83b3706378.dylib`_RINvCskTEkA9M7v6o_10rustc_span15with_source_mapINtNtCsasXF9FuNFFy_4core6result6ResultuNtCsdjh14Z7klAl_12rustc_errors13ErrorReportedENCINvNtCsBlFisMo4u3_15rustc_interface9interface23create_compiler_and_runBJ_NCNvCsQmx6cSBOMI_12rustc_driver12run_compilers_0Es_0EB3n_ + 376
    frame #28: 0x0000000108a5639f librustc_driver-cea70d83b3706378.dylib`_RINvMs_Cs7KVLOe8Wn3E_10scoped_tlsINtB5_9ScopedKeyNtCskTEkA9M7v6o_10rustc_span14SessionGlobalsE3setNCNCINvNtCsBlFisMo4u3_15rustc_interface4util51setup_callbacks_and_run_in_thread_pool_with_globalsNCINvNtB1H_9interface12run_compilerINtNtCsasXF9FuNFFy_4core6result6ResultuNtCsdjh14Z7klAl_12rustc_errors13ErrorReportedENCNvCsQmx6cSBOMI_12rustc_driver12run_compilers_0E0B3G_E00B3G_EB57_ + 1311
    frame #29: 0x0000000108a52ea2 librustc_driver-cea70d83b3706378.dylib`_RINvNtNtCs57FQuS03DEe_3std10sys_common9backtrace28___rust_begin_short_backtraceNCINvNtCsBlFisMo4u3_15rustc_interface4util51setup_callbacks_and_run_in_thread_pool_with_globalsNCINvNtB1m_9interface12run_compilerINtNtCsasXF9FuNFFy_4core6result6ResultuNtCsdjh14Z7klAl_12rustc_errors13ErrorReportedENCNvCsQmx6cSBOMI_12rustc_driver12run_compilers_0E0B3l_E0B3l_EB4M_ + 146
    frame #30: 0x0000000108a3fae5 librustc_driver-cea70d83b3706378.dylib`_RNSNvYNCINvMNtCs57FQuS03DEe_3std6threadNtBa_7Builder15spawn_uncheckedNCINvNtCsBlFisMo4u3_15rustc_interface4util51setup_callbacks_and_run_in_thread_pool_with_globalsNCINvNtB1c_9interface12run_compilerINtNtCsasXF9FuNFFy_4core6result6ResultuNtCsdjh14Z7klAl_12rustc_errors13ErrorReportedENCNvCsQmx6cSBOMI_12rustc_driver12run_compilers_0E0B3b_E0B3b_Es_0INtNtNtB3g_3ops8function6FnOnceuE9call_once6vtableB4C_ + 165
    frame #31: 0x00000001007157c7 libstd-f20edb373511b023.dylib`std::sys::unix::thread::Thread::new::thread_start::h13e1d52aa9cad226 + 39
    frame #32: 0x00007ff80c707514 libsystem_pthread.dylib`_pthread_start + 125
    frame #33: 0x00007ff80c70302f libsystem_pthread.dylib`thread_start + 15

This happens almost 100% of the the time for me, though it can be sporadic.

Doing something like ./x.py build --stage=2 src/tools/cargo is enough to reproduce. It almost always fails while building core for stage2.

I've spent some time trying to debug it, but have not figured anything out.

Using jemalloc makes the error go away.

Currently using master (8ad3c1d), though this has been going on for a while. If I get a chance, I'll try to bisect to see when it started. But for me, I think it started after I updated to Monterey (macOS 12.0).

I have XCode 13.1.

Other users have reported this at rust-lang/cargo#10022.

[ehuss]

More information about this is over at #92185 (comment). These are likely the same issue, but I am not certain.

[djc]

I also ran into this a while ago, reported it against rayon initially: rayon-rs/rayon#901.

There was a non-Rust issue we found that reported similar issues: ibmdb/python-ibmdb#648 (comment).

Analysis we've done so far: it seems that something is broken with the macOS 12-provided /usr/lib/libstdc++.6.dylib. Swapping to the homebrew-provided libstdc++ seems to fix the crash, and using link-cplusplus to swap to libc++ also seems to fix the crash. Switching to jemalloc or mimalloc seemed to fix this.

Does anyone have the ear of folks at Apple?

[saethlin]

Does anyone have the ear of folks at Apple?

Yes, I've reached out to a friend.

[lilyball]

@saethlin Has a radar been filed for this, and if so, what’s the radar number?

[ldionne]

Hey folks, I'm the person who'd be closest to being in charge of the old libstdc++.dylib at Apple. For background information, we only ship libstdc++.dylib for backwards ABI compatibility with older applications that were linked against it years ago. If I'm not mistaken, we don't even ship it on Apple Silicon macs anymore.

The right fix would be to always use libc++, since that is (and has been for a while) the one supported C++ Standard Library on Apple platforms. Sorry if that sounds like a silly suggestion -- I don't have much context on this issue from your side. Is there a reason why you folks are using the older libstdc++.dylib?

[ehuss]

I don't think it is linked to libstdc++.dylib

> otool -L rustc
rustc:
	@rpath/librustc_driver-e04c6e4bb2b707be.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libstd-e0d1108e51bfb0d5.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
	/usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)

#92173 (comment) might have been discussing a different project?

[lilyball]

I agree, this issue showed up in Rust built via Nix (NixOS/nixpkgs#144704) and the build setup there uses libc++ on macOS.

[lopopolo]

Is this issue related to the crashes folks are observing in clippy on macOS in the new 1.59.0 release?

• Clippy crash (version 1.59 on macOS) rust-clippy#8470

[lilyball]

This issue here is about when rustc is built without jemalloc, whereas that issue is occurring with binaries downloaded via rustup and which presumably are using jemalloc.

[ehuss]

@lopopolo Yea, it's the same issue.

@lilyball clippy is not built with jemalloc.

[lopopolo]

@ehuss should this be tagged as a stable-to-stable regression and up the priority? or is there another place I should be following along?

[ehuss]

I'd leave it up to the clippy team to set the priority for fixing clippy specifically.
Fixing for rustc is probably low priority, as the official builds use jemalloc.
I'm not sure where to follow along. Here, or #92185 or on the clippy repo.

[lopopolo]

Thanks for the context and the links @ehuss!

[dtzxporter]

Some more details:

• It only happens for me when the block size is 256 bytes.
• Using malloc inception with DYLD_INSERT_LIBRARIES and a module linked to the latest libSystem.B, the issue does NOT occur.
• It seems to be an issue with free as hooking just realloc still triggered the issue, and as soon as I hooked free, the issue went away.
• It does not occur when MallocStackLogging env var is set either.

[rdornin]

I just want to report that issue is currently blocking me in development. If there are any work around for the macos darwin tool chain and clippy errors... would love to hear about it.

[7596ff]

Analysis we've done so far: it seems that something is broken with the macOS 12-provided /usr/lib/libstdc++.6.dylib. Swapping to the homebrew-provided libstdc++ seems to fix the crash, and using link-cplusplus to swap to libc++ also seems to fix the crash. Switching to jemalloc or mimalloc seemed to fix this.

I've been tracking this issue. My fix was to downgrade to my project's MSRV, 1.57. But after the latest "Command Line Tools for Xcode" update (13.3), I found that I could run clippy again on both stable-x86_64-apple-darwin and stable-aarch64-apple-darwin. I haven't found anything in the changelog that is directly related to this, but it fixed the problem for me, and I think this could be of use.

[ffmichaelm]

I am running into the issue myself. What seems to be a workaround for everybody?

[dtzxporter]

The definitive fix is to update to Xcode 13.3, you may need to use the App Store or Settings App (I've had it come differently on two different machines)

I believe one of the malloc implementations shipped with the 'shared library' feature of Xcode was broken.

From my testing, I believe it's completely unrelated to the rust alloc code.

[ffmichaelm]

I got XCode 13.3 and removed my rustup and reinstalled my version (1.51). Do I need to do anything else special because it seems to not have done anything?

[ffmichaelm]

so I am unsure if it is the same issue but things were fine on big sir and then I upgrade. My particular project has an enormous amount of dependencies (>800). Occasionally when doing a cargo clean, cargo update some libraries fail for apparently no reason (SIGABORT). If I rerun multiple times things seem to eventually build.

error: could not compile `web3`

Caused by:
  process didn't exit successfully: `rustc --crate-name web3 --edition=2018.......(signal: 6, SIGABRT: process abort signal)
warning: build failed, waiting for other jobs to finish...```

Sometimes its web3, sometimes in json-rpc, seems to be random packages.

   Compiling rusoto_kms v0.45.0
error: could not compile `rusoto_kms`

Caused by:
  process didn't exit successfully: `rustc --crate-name rusoto_kms --edition=2018```

Very verbose gives me no more info

[dtzxporter]

You can run it with lldb to check the callstack.

Our issue was actually logging the malloc issue (Source of this ticket and clippy issue):

rustc(14225,0x70000fab4000) malloc: *** error for object 0x600005c70800: pointer being realloc'd was not allocated
rustc(14225,0x70000fab4000) malloc: *** set a breakpoint in malloc_error_break to debug

The above issue appears fixed as of Xcode 13.3.

[ffmichaelm]

Should I create a new ticket?

[dtzxporter]

Should I create a new ticket?

If possible, debug cargo/rustc and print the callstack, if it's different, then definitely!

[dtzxporter]

lldb cargo check (or w/e crashes)
lldb:> process launch
lldb:> process continue
-> when it catches sigabrt
lldb:> thread backtrace

[dtzxporter]

Sorry forgot process launch

[ffmichaelm]

   Compiling diesel v1.4.8
error: could not compile `cookie_store`

Caused by:
  process didn't exit successfully: `rustc --crate-name cookie_store --edition=2018 ........ --cap-lints allow` (signal: 6, SIGABRT: process abort signal)
warning: build failed, waiting for other jobs to finish...
Process 64899 stopped
* thread #5, stop reason = signal SIGUSR1
    frame #0: 0x00007ff80dc1da76 libsystem_pthread.dylib`_pthread_cond_wait + 1256
libsystem_pthread.dylib`_pthread_cond_wait:
->  0x7ff80dc1da76 <+1256>: callq  0x7ff80dc1a97e            ; pthread_testcancel
    0x7ff80dc1da7b <+1261>: leaq   -0x88(%rbp), %rax
    0x7ff80dc1da82 <+1268>: movq   0x10(%rax), %rax
    0x7ff80dc1da86 <+1272>: movq   %rax, 0x8(%rbx)
Target 0: (cargo) stopped.
(lldb)
Process 64899 resuming
Process 64899 stopped
* thread #5, stop reason = signal SIGUSR1
    frame #0: 0x00007ff80dc1da86 libsystem_pthread.dylib`_pthread_cond_wait + 1272
libsystem_pthread.dylib`_pthread_cond_wait:
->  0x7ff80dc1da86 <+1272>: movq   %rax, 0x8(%rbx)
    0x7ff80dc1da8a <+1276>: jmp    0x7ff80dc1daab            ; <+1309>
    0x7ff80dc1da8c <+1278>: movq   -0x58(%rbp), %rcx
    0x7ff80dc1da90 <+1282>: movq   -0x40(%rbp), %rdi
Target 0: (cargo) stopped.

[dtzxporter]

You would have to debug rustc directly I believe. It's kinda hard to explain over github. Are you in the discord/zulip?

[dtzxporter]

What is the discord link? Like I said I don't know if that will help as when I run these individually they seem fine. Is there a way to get the rustc output when doing cargo b?

https://discord.com/invite/rust-lang

cargo build --verbose will print the full rustc command you can run in the same dir to produce the same output

Then you can run lldb on that rustc command.

[ffmichaelm]

rustc(69444,0x700006d18000) malloc: *** error for object 0x600002ff1b00: pointer being realloc'd was not allocated
rustc(69444,0x700006d18000) malloc: *** set a breakpoint in malloc_error_break to debug
Abort trap: 6```

[ffmichaelm]

I am able to build ./x.py build --stage=2 src/tools/cargo just fine

[ffmichaelm]

So it is the same error as described but I am able to reproduce it in a separate fashion, which I am unsure if that makes it the same issue,

[ffmichaelm]

Upon further investigation I took the 1.51-dev I build and installed myself (with cargo) and tried rebuilding my project using that and it worked. When I install rust with rustup does it not also compile it in some fashion? Why does that not work?

[mikemiles-dev]

Now that I’ve built my own rust is there some doc how I can package and distribute a tool chain myself?

[shahn]

I am seeing a crash inside ./x.py test with stage0 rustfmt as well. As I understand it, that's part of the pre-built stuff used to bootstrap, so any kind of fix would require a new stage0?

Exporting MallocStackLogging=1 before the ./x.py test makes the crash go away, but of course slows down everything and creates a ton of debug output. But at least I can run the tests locally again :)

[roy-work]

I'm also seeing this bug, I think?

clippy-driver(8777,0x7000079c2000) malloc: *** error for object 0x600000f89700: pointer being realloc'd was not allocated
clippy-driver(8777,0x7000079c2000) malloc: *** set a breakpoint in malloc_error_break to debug
zsh: abort       rustc --crate-name <name> --edition=2018 <crate path>/src/lib.rs

I'm on v14 of CLI tools (on macOS), though, given how much time has passed.

version: 14.1.0.0.1.1666437224

This is on,

» rustc --version
rustc 1.60.0-nightly (b17226fcc 2022-02-18)

Was there some fix, here? Are later nightlies still affected? (For … historical reasons … my company is on nightly. We try to track the nightly right before the branch to a stable, though, so this is my best guess for 1.60. But this blocks us getting to 1.60, currently. Should I just attempt to skip 1.60? (Would this be fixed in the nightly-before 1.61?))

[ianks]

I can reliably create this using libruby as well, which uses malloc as default allocator. Specifically on an M1 mac. I've confirmed this happens with:

#[global_allocator]
static ALLOCATOR: System = System;

[RalfJung]

Fixing for rustc is probably low priority, as the official builds use jemalloc.

Looks like Miri might also be affected. So this issue means every single user of rustc_driver will waste a few hours/days debugging strange macOS crashes. Would be good to do something that makes this discoverable.

Is there a reason that librustc_driver doesn't just set jemalloc itself, to avoid having every single tool repeat this procedure?

EDIT: looks like that does not work for terrible sad reasons.

[tautschnig]

For anyone seeing this in their GitHub CI: In Kani, we have chosen to upgrade to macos-13 runners (currently in beta): model-checking/kani#2987. This appears to solve the problem for us.

[goffrie]

For the record, we also ran into "pointer being freed was not allocated" running clippy-driver on macos12, and also chose to upgrade the OS.