Reads of undef memory must not cause the behavior to be undefined in general.

[mahkoh]

The following is a list of behavior which is forbidden in all Rust code

• Reads of undef (uninitialized) memory

This cannot be because memcpy will read padding bytes which are undef. It's also not true in practice because in

let x: u8 = undef;
let y: u16 = x as u16 + 0xab00;
let z: u16 = y & 0xff00;

z will be 0xab00 and not undef.

[steveklabnik]

/cc @rust-lang/lang

[eefriedman]

This cannot be because memcpy will read padding bytes which are undef.

I'm not sure what you mean here; the fact that rustc sometimes generates a call to memcpy is mostly irrelevant to the semantics of Rust code.

z will be 0xab00 and not undef.

The LLVM add instruction behaves this way, but the rustc "+" operator isn't guaranteed to translate directly to the LLVM add instruction. In fact, it doesn't in overflow checking mode.

[mahkoh]

@eefriedman

1. Not if memcpy is itself written in rust code.
2. It doesn't make a difference. x as u16 will have the upper byte zeroed.

[mahkoh]

The reason for the current rule seems to be: "I don't know how to formulate the real rules so I'll simply disallow it completely."

[eefriedman]

Not if memcpy is itself written in rust code.

There isn't any fundamental need for it to be legal to write memcpy in Rust... it's part of the runtime. Granted, it would be convenient in some cases. Maybe we can add a special-case for "copying" an undef Copy value .

It doesn't make a difference. x as u16 will have the upper byte zeroed.

You're not looking at this at the right level. "add" is opaque; in theory, it could involve indexing into an array using the values of the operands, which could crash the program if undef is involved.

Anyway, trying to make promises about how exactly arithmetic is implemented leads down a path which isn't really productive.

[mahkoh]

There isn't any fundamental need for it to be legal to write memcpy in Rust

Whew

Maybe we can add a special-case for "copying" an undef Copy value .

That's the opposite of what should be done. The formulation must be more abstract so that everything legal can be done with undef while still disallowing all that that causes the behavior to be undefined. Maybe one even has to go so far as to add a primitive size one type that cannot be interpreted as any other type (without transmute) and has exactly one value that spans all u8 values. One might interpret it as a one byte type with one byte padding but it's not really padding.

You're not looking at this at the right level.

I see that adding an example did more harm than good.

[huonw]

It seems like spec'ing the example of addition/bitmasks would also require expanding the notion of "undefined value" to be at the bit level, and for the language to have some idea about results of (special cases of) operators. The latter seems like a rather open ended space, with very complicated properties encodable. This isn't a blocker, but it does mean touching this requires some care. (Byte-level undef would work for that specific example, but it seems restrictive---what if u8 was bool and 0xFF00 was 0xFFFE---and even then, operators still need to be understood at the byte level.)

[mahkoh]

require expanding the notion of "undefined value" to be at the bit level

Does the rust manual define undef at all? IIRC it only links to the llvm manual which already talks about undef at the bit level.

It seems like spec'ing the example of addition/bitmasks would also

The example is not important. It's just supposed to show that, at the llvm level, working with undef can lead to definite results and doesn't necessarily lead to undef propagation. Actually using undef values in rust code for anything remotely complicated can easily lead to UB because of undef propagation. However, there is no reason for the manual to claim that reading undef always leads to UB which is much stricter than what llvm requires.

[huonw]

Rust isn't LLVM, and we don't necessarily want to make every guarantee that it does. I'm personally not that happy that we defer to LLVM for many definitions for convenience, and I expect this to not be the case in the future e.g. if an actual spec is written. (Feel free to read it as if I said "introducing a more formal Rust undef, which is tracked at the bit level" in place of "expanding the notion ... bit level".)

On the point of undefined values, you're right that we just link to LLVM's definition of undef, but we do so in the context of reading undef memory, which doesn't say anything about an in-register value as we'd have for arithmetic, so even the most pedantic reading is vague. Also, I don't recall any team discussion featuring undef values where anything other than the whole value was considered undef. Summary: this is under-spec'd and the existing underlying/assumed sense of this area at the Rust level is almost certainly not for individual bits.

Of course, you're also right that using LLVM's undef can lead to definite results even without tracking bits (e.g. let x = if undef { 0 } else { 0 }; will always give x == 0), but see the first sentence.

[mahkoh]

I'm personally not that happy that we defer to LLVM for many definitions for convenience, and I expect this to not be the case in the future e.g. if an actual spec is written.

You're already de facto guaranteeing the current behavior by having it work. Significant changes cannot be made without silently (!) breaking code which is the complete opposite of stability.

Even if you had reliable normative information, undefined operations that have behaved reliably for some time cannot always be made behave differently without causing many problems (e.g. signed integer overflow in C.)

But there is no reliable normative information and thus people have to rely on what works in the current implementation for just about everything. E.g. the only official information about the behavior of transmute that can be found is

Unsafely transforms a value of one type into a value of another type.

Both types must have the same size.

This doesn't even guarantee that the returned value has anything to do with the input value. Precisely because transmute is completely unspecified, the current implementation must be treated as normative.

The same applies to undef: memcpy works now and it works according to llvm so it has to continue to work.

[mahkoh]

For example, the following is discouraged by a lint but does not cause any problems:

fn f(&self) -> &Self;

fn f_mut(&mut self) -> &mut Self {
    unsafe { transmute(self.f()) }
}

At the same time there is other, unreliable information flying around that says that transmuting & to &mut is always undefined. Strictly speaking this is correct because transmute is mostly undefined, but if that's the argument then transmuting &mut to & is equally undefined.

The lack of any kind of information regarding transmute means that you have no way to distinguish between safe, dangerous, and forbidden operations. All of them are equally undefined.

[steveklabnik]

Precisely because transmute is completely unspecified, the current implementation must be treated as normative.

I strongly disagree with this. We should be seeking to specify such things, not just accepting whatever random behavior happens to work in these corners.

[arielb1]

@mahkoh

There's a difference between what we specify and what we allow LLVM to assume.

The reason there is a lint about transmuting & to &mut is because that causes instant death by aliasing rule violations - we basically allow ourselves to add MIR optimizations that will destroy your code in that case.

We still have not really decided how much "instant aliasing death" is a thing - @thestinger preferred to have access-based aliasing rules and I think he's got a point - but we didn't specify that it was not a thing.

On the memcpy issue: the part of C's semantics that allows memcpy to be implemented in C code is quite ugly. I would prefer not to specify anything like that.
sne point that I would like to make clear is the difference between well-defined, unspecified and undefined behaviour:

• well-defined behaviour refers to things that ordinary programs are allowed to assume. The compiler must not break it.
• undefined behaviour refers to things that the compiler is allowed to assume (to not happen). All programs must not try to provoke it.
• unspecified behaviour refers to things that are not a part of the specification. The compiler is not granted a carte blanche to abuse them, but neither do applications.

There might very well be some other specification that defines some unspecified behaviour (either to something well-defined or undefined). For example, system calls are specified by your favourite OS's documentation.

At this moment, we have no plans to publicly specify everything that is needed by a stable libcore. This means that anyone who wants to implement one (currently that's either us or you) must look at the compiler source for a specification and coordinate with us to avoid breakage. This of course does not mean that undefined behaviour is triggered - that would give the compiler a license to destroy all Rust code. There are boundaries, they are just somewhat unclear. I am sorry that this interferes with your project.

[nikomatsakis]

@mahkoh

You're already de facto guaranteeing the current behavior by having it work. Significant changes cannot be made without silently (!) breaking code which is the complete opposite of stability.

This same reasoning can be used to argue that, e.g., we should never change our sorting algorithm, because it may invoke the comparator in a different order, and so forth. We've also made it clear that various low-level details are expected to change, and that authors of unsafe code (in particular) will need to track the language as it evolves.

That said, we should definitely consider "common practice" when deciding what kinds of things are undefined behavior. This is only partially because of existing code -- what I am most concerned about is just that if the rules are too complex and abstract (that is, too divorced from some abstract model of how the machine operators), people won't be able to keep them in their head, and so they will write noncomformant code that does surprising things when optimized.

From what I can see, C has this problem in spades. Infinite loops, TBAA, etc all lead to making it actually surprisingly hard to write "correct" C code that does anything clever. But of course people write all kinds of clever things in C, many of which are compiler issues waiting to happen.

I think @mahkoh has a point that it would be nice to affirm that particular idioms (e.g., a naively written memcpy that "seems right") will work without leading to undefined behavior. I'm just not sure if that's an urgent priority: it's a rather complex equation, since we must also consider what LLVM will do (and to what extent we can control that), and so forth, and we don't want to wind up guaranteeing too much. Put another way, I am sympathetic with the aims of this RFC, but I also wonder if it would be better to try to tackle the problem of "stabilizing" unsafe code patterns in a more wholesale fashion, rather than going at it piecemeal.

[mahkoh]

@arielb1

The reason there is a lint about transmuting & to &mut is because that causes instant death by aliasing rule violations - we basically allow ourselves to add MIR optimizations that will destroy your code in that case.

There is no lint against transmuting (&, &) to (&mut, &mut) and the manual only talks about llvm's aliasing rules that must not be violated. If you're saying that you might break such things freely in the future, then this certainly doesn't just affect "my project".

On the memcpy issue: the part of C's semantics that allows memcpy to be implemented in C code is quite ugly.

There are many special cases for char but apart from that I don't recall anything particularly ugly. It would certainly be better if memcpy did not have to be written with u8's and then rely on LLVM to optimize it.

The point that I would like to make clear is the difference between well-defined, unspecified and undefined behaviour:

I disagree with your definitions. Here are mine:

• undefined behavior is behavior upon an operation that has not been explicitly defined by the specification
• unspecified behavior is behavior that depends on the implementation and that the implementation need not specify
• implementation defined behavior is behavior that is defined by each implementation

With these definitions, undefined behavior is what you called unspecified behavior. I think the C++11 standard agrees with my definition:

undefined behavior
behavior for which this International Standard imposes no requirements
[ Note: Undefined behavior may be expected when this International Standard omits any explicit definition of behavior ...

There is a significant difference between undefined behavior and unspecified behavior so we have to agree on what we're talking about.

At this moment, we have no plans to publicly specify everything that is needed by a stable libcore

I don't think anything in this issue is restricted to code in a libcore. In fact, libcore doesn't contain a memcpy so I'm not sure how libcore is related to this issue. A memcpy might be written in many situations: when you write a kernel; when you need a particularly optimized memcpy; when you need a memcpy that can be inlined; etc. And transmutes are certainly used in lots of code.

@nikomatsakis

We've also made it clear that various low-level details are expected to change, and that authors of unsafe code (in particular) will need to track the language as it evolves.

I don't recall this and breaking random unsafe code seems to go completely against the rest of your stability guarantees. Please link to the text where you said this.