clarify rules for ZST Boxes #77844

RalfJung · 2020-10-12T08:17:19Z

LLVM's rules around getelementptr inbounds with offset 0 are a bit annoying, and as a consequence we have no choice but say that a Box<()> pointing to previously allocated memory that has since been freed is UB. Clarify the docs to reflect this.

This is based on conversations on the LLVM mailing list.

Here's my initial mail: https://lists.llvm.org/pipermail/llvm-dev/2019-February/130452.html
The first email of the March part of that thread: https://lists.llvm.org/pipermail/llvm-dev/2019-March/130831.html
First email of the April part: https://lists.llvm.org/pipermail/llvm-dev/2019-April/131693.html

The conclusion for me at least was that getelementptr inbounds with offset 0 is not the identity function, but can sometimes return poison even when the input is a regular pointer -- specifically, it returns poison when this pointer points into something that LLVM "knows has been deallocated", i.e., a former LLVM-managed allocation. It is however the identity function on pointers obtained by casting integers.

Note that there are formal proposals for LLVM semantics where getelementptr inbounds with offset 0 isn't quite the identity function but never returns poison (it affects the provenance of the pointer but in a way that doesn't matter if this pointer is never used for memory accesses), and indeed this is likely necessary to consistently describe LLVM semantics. But with the informal LLVM LangRef that we have right now, and with LLVM devs insisting otherwise, it seems unwise to rely on this.

rust-highfive · 2020-10-12T08:17:23Z

r? @sfackler

(rust_highfive has picked a reviewer for you, use r? to override)

danielhenrymantilla

Damn, that is a pity. Good catch anyways!

danielhenrymantilla · 2020-10-12T16:37:31Z

library/alloc/src/boxed.rs

+//! produces a valid pointer, but a pointer pointing into previously allocated memory that since got
+//! freed is not valid.


Suggested change

//! produces a valid pointer, but a pointer pointing into previously allocated memory that since got

//! freed is not valid.

//! produces a valid pointer, [unless it is a pointer pointing into previously allocated memory that since got

//! freed][`::core::ptr`].

Otherwise it may not be 100% clear which of the two rules "win"s in case of, for instance:

dealloc(ptr : *mut ()); // deallocated Box::<()>::from_raw(ptr as usize as *mut ()) // ^^^^^^^^^^^^ // aligned non-zero integer

I think I should probably change the text to say "integer literal", would that also solve your concern?

ChrisDenton · 2020-10-12T17:54:10Z

So for clarity, it is memory safe to magic up a boxed ZST if the pointer does not come via a "standard allocator"? For example:

use core::ptr;
use core::mem;

// Assumes it's safe to bypass the ZST's constructor(s)
unsafe fn zst_box<T>() -> Box<T> {
    assert_eq!(mem::size_of::<T>(), 0);
    // `NonNull::dangling` ensures that the pointer is well-aligned
    // and does not come from an allocator.
    let ptr = ptr::NonNull::dangling().as_ptr();
    Box::from_raw(ptr)
}

RalfJung · 2020-10-13T07:31:10Z

So for clarity, it is memory safe to magic up a boxed ZST if the pointer does not come via a "standard allocator"? For example:

Yes. That code does not violate Box memory layout. (It could be UB for other reasons as your comment indicates, e.g. T := ! makes this UB.)

Dylan-DPC-zz · 2020-10-31T11:27:31Z

r? @Mark-Simulacrum

Mark-Simulacrum · 2020-11-05T21:00:34Z

I am tempted to just r+ this, but it seems like it at least warrants a cc @rust-lang/lang. Going to r? @nikomatsakis as T-lang lead.

nikomatsakis · 2020-11-06T17:24:21Z

Nominating to mention it at our next meeting but I expect to r+ and land after that.

nikomatsakis · 2020-11-16T22:22:16Z

library/alloc/src/boxed.rs

+//! For zero-sized values, the `Box` pointer still has to be [valid] for reads and writes and
+//! sufficiently aligned. In particular, casting any aligned non-zero integer literal to a raw
+//! pointer produces a valid pointer, but a pointer pointing into previously allocated memory that
+//! since got freed is not valid.


Are there any limitations here -- i.e., it's only a problem if the integer value is somehow derived from a freed value via some trackable provenance -- or is it UB even if you just pull an integer out of thin air that happens to be the same as a freed value?

If the latter, can we give some concrete advice about what value people should use for zero-sized values? I at least am feeling a bit confused about what kind of value is valid to use, given that it can't be null but must be aligned.

These are good questions. Unfortunately, LLVM is not specified precisely enough to properly answer them. We are building on quicksand here and can just take a guess about how it will behave. ;)

My personal interpretation is that it is okay to do this if the pointer has no provenance. That should always be the case for pointers created from an integer literal, or any other computations where there are no pointers involved.

How would you feel about adding some text to that effect, even if it is quite tentative? What does the compiler do if we create a Box::new(T) where struct T; has some kind of high alignment?

So you are saying I should use the term "provenance"? I had avoided that, because I don't think we are defining it anywhere outside the UCG glossary.

What does the compiler do if we create a Box::new(T) where struct T; has some kind of high alignment?

This is implemented in the stdlib actually, not the compiler. It uses the alignment as ptr value:

rust/library/alloc/src/alloc.rs

Line 163 in c4f836a

0 => Ok(NonNull::slice_from_raw_parts(layout.dangling(), 0)),

No, I am not suggesting you should use the word provenance necessarily. What I was suggesting is saying something like "The recommended way to build a Box to a ZST is to use an integer equal to the alignment as the pointer value."

And/or we could mention ptr::NonNull::dangling() as a canonically-valid-and-always-up-to-date implementation 🙂

That would be even better, good point.

I added a comment along these lines.

nikomatsakis · 2020-11-19T14:21:11Z

To clarify the state of this PR: I am waiting for an addition to the comment that suggests the use of dangling, as suggested here.

RalfJung · 2020-11-20T09:27:00Z

library/core/src/ptr/mod.rs

+//!   valid for zero-sized accesses. This corresponds to writing your own allocator; allocating
+//!   zero-sized objects is not very hard. In contrast, when you use the standard allocator, after
+//!   memory got deallocated, even zero-sized accesses to that memory are invalid.
+//!   The canonical way to obtain a pointer that is valid for zero-sized accesses is [`NonNull::dangling`].


This is rather unfortunate naming: the text says "the pointer must not be dangling" and then proceeds to recommend calling NonNull::dangling...

Is there another term we could use instead of the first "dangling"?

- the pointer must not be "dangling" in the sense of pointing to deallocated memory + the pointer must not be pointing to deallocated memory

maybe?

I tried something like this, WDYT?

It's great!👌

nikomatsakis · 2020-11-20T22:14:47Z

@bors r+ rollup

bors · 2020-11-20T22:14:49Z

📌 Commit a7677f7 has been approved by nikomatsakis

clarify rules for ZST Boxes LLVM's rules around `getelementptr inbounds` with offset 0 are a bit annoying, and as a consequence we have no choice but say that a `Box<()>` pointing to previously allocated memory that has since been freed is UB. Clarify the docs to reflect this. This is based on conversations on the LLVM mailing list. * Here's my initial mail: https://lists.llvm.org/pipermail/llvm-dev/2019-February/130452.html * The first email of the March part of that thread: https://lists.llvm.org/pipermail/llvm-dev/2019-March/130831.html * First email of the April part: https://lists.llvm.org/pipermail/llvm-dev/2019-April/131693.html The conclusion for me at least was that `getelementptr inbounds` with offset 0 is *not* the identity function, but can sometimes return `poison` even when the input is a regular pointer -- specifically, it returns `poison` when this pointer points into something that LLVM "knows has been deallocated", i.e., a former LLVM-managed allocation. It is however the identity function on pointers obtained by casting integers. Note that there [are formal proposals](https://people.mpi-sws.org/~jung/twinsem/twinsem.pdf) for LLVM semantics where `getelementptr inbounds` with offset 0 isn't quite the identity function but never returns `poison` (it affects the provenance of the pointer but in a way that doesn't matter if this pointer is never used for memory accesses), and indeed this is likely necessary to consistently describe LLVM semantics. But with the informal LLVM LangRef that we have right now, and with LLVM devs insisting otherwise, it seems unwise to rely on this.

Rollup of 8 pull requests Successful merges: - rust-lang#77844 (clarify rules for ZST Boxes) - rust-lang#79067 (Refactor the abi handling code a bit) - rust-lang#79182 (Fix links to extern types in rustdoc (fixes rust-lang#78777)) - rust-lang#79231 (Exhaustively match in variant count instrinsic) - rust-lang#79238 (Direct RUSTC_LOG (tracing/log) output to stderr instead of stdout.) - rust-lang#79256 (Fix typos) - rust-lang#79264 (Get rid of some doctree items) - rust-lang#79272 (Support building clone shims for arrays with generic size) Failed merges: r? `@ghost` `@rustbot` modify labels: rollup

rust-highfive assigned sfackler Oct 12, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Oct 12, 2020

clarify rules for ZST Boxes

c555aab

RalfJung force-pushed the zst-box branch from af2b8a2 to c555aab Compare October 12, 2020 08:32

jyn514 added A-docs Area: documentation for any part of the project, including the compiler, standard library, and tools T-lang Relevant to the language team, which will review and decide on the PR/issue. A-zst Area: Zero-sized types (ZST). labels Oct 12, 2020

danielhenrymantilla approved these changes Oct 12, 2020

View reviewed changes

explicitly talk about integer literals

0f572a9

camelid added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 30, 2020

rust-highfive assigned Mark-Simulacrum and unassigned sfackler Oct 31, 2020

rust-highfive assigned nikomatsakis and unassigned Mark-Simulacrum Nov 5, 2020

nikomatsakis added the I-nominated label Nov 6, 2020

nikomatsakis added S-waiting-on-team Status: Awaiting decision from the relevant subteam (see the T-<team> label). and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 9, 2020

nikomatsakis reviewed Nov 16, 2020

View reviewed changes

nikomatsakis added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-team Status: Awaiting decision from the relevant subteam (see the T-<team> label). labels Nov 19, 2020

RalfJung commented Nov 20, 2020

View reviewed changes

RalfJung removed the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Nov 20, 2020

RalfJung added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Nov 20, 2020

reference NonNull::dangling

a7677f7

RalfJung force-pushed the zst-box branch from c776caa to a7677f7 Compare November 20, 2020 10:10

nikomatsakis approved these changes Nov 20, 2020

View reviewed changes

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 20, 2020

Dylan-DPC-zz mentioned this pull request Nov 21, 2020

Rollup of 9 pull requests #79252

Closed

Dylan-DPC-zz mentioned this pull request Nov 21, 2020

Rollup of 8 pull requests #79273

Merged

bors merged commit 6cd02a8 into rust-lang:master Nov 21, 2020

rustbot added this to the 1.50.0 milestone Nov 21, 2020

RalfJung deleted the zst-box branch November 22, 2020 15:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

clarify rules for ZST Boxes #77844

clarify rules for ZST Boxes #77844

RalfJung commented Oct 12, 2020 •

edited

Loading

rust-highfive commented Oct 12, 2020

danielhenrymantilla left a comment

danielhenrymantilla Oct 12, 2020

RalfJung Oct 13, 2020

ChrisDenton commented Oct 12, 2020

RalfJung commented Oct 13, 2020

Dylan-DPC-zz commented Oct 31, 2020

Mark-Simulacrum commented Nov 5, 2020

nikomatsakis commented Nov 6, 2020

nikomatsakis Nov 16, 2020

RalfJung Nov 17, 2020

nikomatsakis Nov 17, 2020

RalfJung Nov 18, 2020

nikomatsakis Nov 18, 2020

danielhenrymantilla Nov 18, 2020

nikomatsakis Nov 19, 2020

RalfJung Nov 20, 2020

nikomatsakis commented Nov 19, 2020

RalfJung Nov 20, 2020

danielhenrymantilla Nov 20, 2020

RalfJung Nov 20, 2020

danielhenrymantilla Nov 20, 2020

nikomatsakis commented Nov 20, 2020

bors commented Nov 20, 2020

		//! produces a valid pointer, but a pointer pointing into previously allocated memory that since got
		//! freed is not valid.

clarify rules for ZST Boxes #77844

clarify rules for ZST Boxes #77844

Conversation

RalfJung commented Oct 12, 2020 • edited Loading

rust-highfive commented Oct 12, 2020

danielhenrymantilla left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChrisDenton commented Oct 12, 2020

RalfJung commented Oct 13, 2020

Dylan-DPC-zz commented Oct 31, 2020

Mark-Simulacrum commented Nov 5, 2020

nikomatsakis commented Nov 6, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nikomatsakis commented Nov 19, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nikomatsakis commented Nov 20, 2020

bors commented Nov 20, 2020

RalfJung commented Oct 12, 2020 •

edited

Loading