Treat thread-local statics on main thread as static roots for leakage analysis

[max-heller]

Miri currently treats allocations as leaked if they're only referenced in thread-local statics. For threads other than the main thread, this is correct, since the thread can terminate before the program does, but references in the main thread's locals should be treated as living for the duration of the program since the thread lives for the duration of the program.

This PR adds thread-local statics and TLS keys as "static roots" for leakage analysis, but does not yet bless the example program from #2881. See #2881 (comment)

Closes #2881

[oli-obk]

I did some extra logging, and the issue turns out to be that all the static roots you added are deallocated at the end of the program, causing the leaked allocations within to not be detected anymore.

If you do not leak, then the thread local statics (even of the main thread) get freed. As an example: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=3c4634970fddcc8784f4376a8e1241fd passes without leaking.

This means we cannot just apply what was proposed in #2881 (comment)

Knowing this, do you still think it's worth it to add this extra logic for something that goes away if the main thread TLS actually cleans up its memory?

[RalfJung]

Oh right, we'd have to also skip deallocating main thread TLS.

[RalfJung]

@rustbot author

[max-heller]

Knowing this, do you still think it's worth it to add this extra logic for something that goes away if the main thread TLS actually cleans up its memory?

Does this mean the value actually doesn't live for the lifetime of the program, or is it still valid to consider the program as ended before the main thread cleans up its TLS?

Oh right, we'd have to also skip deallocating main thread TLS.

Would this mean missing out on analysis of main thread TLS destructors? E.g. if a destructor contained UB we'd no longer be able to detect it?

Instead of skipping deallocation, could we run the leakage check before the main thread cleans up?

@rustbot label -S-waiting-on-author +S-waiting-on-review

[RalfJung]

We should definitely run the destructor. We just shouldn't deallocate the backing storage.

[RalfJung]

So this will test that we allow leaks in one of the 2 kinds of TLS (TLS statics vs OS-supported TLS keys). We should make sure to have tests covering both (by directly using the underlying mechanisms).

[max-heller]

Added #[thread_local]s to the tests

[RalfJung]

I think we want one test that uses #[thread_local] and one that uses pthread TLS keys directly.

Or we only treat #[thread_local] as static roots? Then we cover all targets with target_thread_local. Doing special magic to library stuff like the pthread keys is somewhat dubious anyway...

[oli-obk]

looks like the msvc thread locals need a similar treatment. I think they have a completely separate code path

[max-heller]

@rustbot author

[max-heller]

looks like the msvc thread locals need a similar treatment. I think they have a completely separate code path

separate from this path, you mean?

miri/src/shims/windows/foreign_items.rs

Lines 241 to 269 in 6f688d5

	// Thread-local storage
	"TlsAlloc" => {
	// This just creates a key; Windows does not natively support TLS destructors.
	// Create key and return it.
	let [] = this.check_shim(abi, Abi::System { unwind: false }, link_name, args)?;
	let key = this.machine.tls.create_tls_key(None, dest.layout.size)?;
	this.write_scalar(Scalar::from_uint(key, dest.layout.size), dest)?;
	}
	"TlsGetValue" => {
	let [key] = this.check_shim(abi, Abi::System { unwind: false }, link_name, args)?;
	let key = u128::from(this.read_scalar(key)?.to_u32()?);
	let active_thread = this.get_active_thread();
	let ptr = this.machine.tls.load_tls(key, active_thread, this)?;
	this.write_scalar(ptr, dest)?;
	}
	"TlsSetValue" => {
	let [key, new_ptr] =
	this.check_shim(abi, Abi::System { unwind: false }, link_name, args)?;
	let key = u128::from(this.read_scalar(key)?.to_u32()?);
	let active_thread = this.get_active_thread();
	let new_data = this.read_scalar(new_ptr)?;
	this.machine.tls.store_tls(
	key,
	active_thread,
	new_data,
	&*this.tcx,
	&mut this.machine.static_roots,
	)?;

[RalfJung]

Sorry for pointing you to shims/tls.rs, that was probably the wrong starting point. I think in a first step we just want to replace this here

miri/src/eval.rs

Lines 250 to 252 in 8bbc93c

	// Need to call this ourselves since we are not going to return to the scheduler
	// loop, and we want the main thread TLS to not show up as memory leaks.
	this.terminate_active_thread()?;

by code that instead marks the thread_local statics of the main thread as "allowed to leak".
Then we should have tests:

• for having some leaked memory in a #[thread_local] static.
• for having some leaked memory in a tread_local! { static }. This test should ignore-windows since some Windows targets don't use #[thread_local] for their ``tread_local!`.

[RalfJung]

@max-heller this is waiting for you to resolve the remaining comments. See the open threads above, in particular here and here.

Once this is ready for review again, please use @rustbot ready.

[max-heller]

Addressed the open requests, sorry for the delay.
@rustbot ready

[RalfJung]

No worries, it's not like I had fast reaction times on this PR in the past. :)

[RalfJung]

@rustbot author

[max-heller]

@rustbot ready

[RalfJung]

@rustbot author
I assume this is the last round, so please also squash this into a single commit.

[max-heller]

@rustbot ready

[RalfJung]

Awesome, thanks for staying with us until this is finally done. :)
@bors r+

[bors]

📌 Commit 2756f11 has been approved by RalfJung

It is now in the queue for this repository.

[bors]

⌛ Testing commit 2756f11 with merge 6db4c80...

[bors]

☀️ Test successful - checks-actions
Approved by: RalfJung
Pushing 6db4c80 to master...

[bors]

☀️ Test successful - checks-actions
Approved by: RalfJung
Pushing 6db4c80 to master...