RFC for unsafe blocks in unsafe fn

[RalfJung]

No longer treat the body of an unsafe fn as being an unsafe block. To avoid a breaking change, this is a warning now and may become an error in a future edition.

Rendered

Cc @rust-lang/wg-unsafe-code-guidelines

[Diggsey]

If this were to be accepted, it would be much better to get the warning in before the 2018 epoch hits stable. If there's no time to get it in for 2018 then I don't think it should be accepted.

[oli-obk]

We can start as allow by default. The fact that const unsafe fn already behaves this way and that clippy can uplift this lint to warn, will already make sure to migrate large parts of the ecosystem.

Oh... you are already mentioning that below in the unresolved questions...

[RalfJung]

Yeah, I had to follow the RFC structure, didn't I? ;)

[RalfJung]

@Diggsey There will be another edition. And once we no longer warn about unsafe blocks in unsafe fn being redundant, we can indeed phase this in more smoothly e.g. with Clippy.

[burdges]

We needed this years ago, so the sooner the better.

[Ixrec]

C# has unsafe blocks in addition to unsafe methods. Though it's not super helpful since I'm not aware of the C# community ever discussing burden of proof issues like this RFC does, probably because 99.99% of the time the answer in that language is "unsafe just isn't worth it". I couldn't even find a C# style guide that mentions the existence of unsafe code, much less has guidelines for making it less dangerous.

[Centril]

Furthermore, while other languages may not have blocks many do have escape hatches (which are relevant to this section)... Examples I found:

1. Haskell: http://hackage.haskell.org/package/base-4.12.0.0/docs/System-IO-Unsafe.html#v:unsafePerformIO and http://hackage.haskell.org/package/base-4.8.0.0/docs/Unsafe-Coerce.html

2. Idris: https://github.com/idris-lang/Idris-dev/blob/master/libs/prelude/Builtins.idr#L189

3. Agda: https://agda.readthedocs.io/en/v2.5.4.2/language/postulates.html

4. Coq: https://github.com/coq/coq/wiki/CoqAndAxioms also, http://adam.chlipala.net/cpdt/html/Universes.html#lab78

5. ReasonML: https://reasonml.github.io/docs/en/interop -- actually, this is sort of a block...

6. OCaml: https://caml.inria.fr/pub/docs/manual-ocaml/libref/Obj.html

7. SML: http://www.smlnj.org/doc/SMLofNJ/pages/unsafe.html

8. Ada: http://www.adaic.org/resources/add_content/docs/95style/html/sec_5/5-9-1.html

[RalfJung]

@Centril But this RFC is specifically about blocks and nesting of unsafe escape hatches. I do not think any of the examples you mention apply there, do they?

@Ixrec Thanks, I had no idea! And it looks like unsafe operations can be used freely in unsafe functions. :/

[Centril]

@RalfJung not with that specificity no; the languages have the "block" form, e.g.:

x = unsafePerformIO $ do
    foo
    bar
    ...

what they lack is the unsafe function form.

[RalfJung]

That's still quite different from Rust. It's just a normal function to the compiler, no checks for "unsafe operations" or so are performed. I do not see a close enough relation to this RFC.

[Centril]

@RalfJung alright; fair enough. Let's leave this bit (the comment thread) open for interested readers who want to see the associated material linked. :)

[Centril]

Other possible drawbacks to list:

1. It will become less ergonomic to write unsafe code (it's justified I think, but worth mentioning...).

2. People might just do this:

unsafe fn frobnicate(x: T, y: U, ...) -> R {
    unsafe {
        ... // Actual code.
    }
}

and then nothing has been gained. I don't know what the risk of this is, but worth mentioning.

[RalfJung]

I added (a variant of) 1.

For 2., I think something has been gained: It is not possible to incrementally improve this function's unsafe locality. Or maybe it is not worth it, then that has at least been explicitly documented in the code.

[Centril]

Yeah I'm not entirely sure 2. is a drawback or not; I usually try to write the section as what someone else might think is a potential drawback (but not necessarily me) -- i.e. this is the section where I try to bring out my inner Dr. Phil / empathy =P

[RalfJung]

The drawbacks section now says

Many unsafe fn are actually rather short (no more than 3 lines) and will
likely end up just being one large unsafe block. This change would make such functions less ergonomic to write.

That should cover 2, right?

[Centril]

@RalfJung the concern is actually slightly different here; it is that people will just go and write one big unsafe { ... } when they shouldn't.

[mark-i-m]

Isn't that already possible today?

[Centril]

@mark-i-m yes; sure -- the concern is that the change we do here might not have any noticable effect cause people could be lazy and...

[Centril]

I've added T-dev-tools for the possible clippy lint for now. If no such clippy lint is proposed in the final version before final review of the RFC I'll remove that team.

[scottmcm]

I think this RFC need more examples of realistic code where this would help, and an explanation of why it helps in enough cases that it's worse the obvious pain in current cases. That seems especially true since "safe" code is just as suspect when it's around unsafe.

Another alternative: an ununsafe block (obvious strawman name) that disallows calling unsafe code again (that can be undone with another unsafe block, of course).

More generally, async fn puts an async block around the body (among other things), so it doesn't seem insane that unsafe fn puts an unsafe block around the body. Though we won't have effect polymorphism any time soon, is there some inconsistency here that should be fixed at a different level?

[mark-i-m]

@scottmcm There are some places in the language where you are forced to use unsafe fn. For example, SIMD or alternate calling conventions or implementing traits with unsafe functions.

Here are some example from a project I worked on:

1. Allocator trait: https://github.com/mark-i-m/os2/blob/47136c645878e0295142213bf63e03fe4e0bca45/kernel/memory/heap.rs#L26-L43
   You might notice that it's very non-obvious from the body of these implementations if they actually use any unsafe operations. IIRC, they don't.

2. Weird calling conventions: https://github.com/mark-i-m/os2/blob/47136c645878e0295142213bf63e03fe4e0bca45/kernel/process/sched.rs#L158-L179
   This function is part of the context-switching code of an OS kernel. The stack is in a very weird state when this is called. In this case, the caller does have a proof obligation (it should only be called from a particular part of the kernel). It also happens that there are one or two patches of unsafe operations. It would be really nice to separate these concerns

[sfackler]

Another concrete use case I find valuable:

Callback functions used when interacting with C libraries are almost always unsafe extern "C" fns, since they're usually passed raw pointers. However, the actual scope of unsafety in the implementations of the callbacks is commonly limited to casting those raw pointers in to Rust references. Currently, that's not called out visually since the entire function body is already an unsafe block but this RFC would enable more tightly scoped blocks.

[RalfJung]

More generally, async fn puts an async block around the body (among other things), so it doesn't seem insane that unsafe fn puts an unsafe block around the body. Though we won't have effect polymorphism any time soon, is there some inconsistency here that should be fixed at a different level?

unsafe is not an effect and behaves nothing like an effect. :)

async says "this function is externally observable to not behave like a normal function, not even calling it works the normal way". unsafe just means "this is a normal function but you have some preconditions". unsafe can be discharged: By proving some things (just proving and checking, not actually doing anything!), you can make an unsafe function safe (think: get_unchecked vs get). This is impossible with effects. You cannot remove async from your function after proving some things about it or adding some assert!.

There is some syntactical similarity between async and unsafe, but semantically speaking they are worlds apart.

[Centril]

unsafe is not an effect and behaves nothing like an effect. :)

well... not everyone agrees (as you know) ;) https://internals.rust-lang.org/t/what-does-unsafe-mean/6696/2
cc @eternaleye

[RalfJung]

Some further examples of longer unsafe fn that look like they would benefit from a more clear demarcation of where the danger is in there:

• align_to and align_to_mut
• Rc helper methods from_raw and allocate_for_ptr
• slice::merge
• The WASM mutex implementation
• do_exec for Fuchsia

Basically any time your unsafe fn contains any non-trivial logic, the implicit unsafe block is not your friend.

However, I also have to admit that the vast majority of unsafe fn are less than 3 lines and just call another unsafe fn or perform a raw ptr deref or so. For all of them, this change would mostly be syntactic noise, which is unfortunate.

[RalfJung]

well... not everyone agrees (as you know) ;)

I am aware. However, I gave my usual arguments above, and AFAIK they have not been refuted yet, so I will keep claiming that everyone who says unsafe is an effect is wrong. ;) In this particular case I think it is actually actively harmful to think of unsafe this way, because it emphasizes a focus on the "additional power" aspect of unsafe, instead of focusing on the "proof obligation" aspect. I think the latter is vastly more important, and the language agrees with me: If the focus was "additional power", we would not let you write unsafe blocks on a safe fn. If the focus was additional power, we would e.g. use unsafe to mark the presence of interior mutability and we would want a guarantee like "calling a safe fn will never write to shared data" (akin to "calling a non-async fn will never yield to another task"). We do not have this guarantee, because this is not what unsafe is for. It is not an effect. We could have an annotation for "writes to shared data", and I agree that would be an effect.

I think whoever claims that unsafe is an effect should formally define what you can then say about a function that is not marked unsafe, in terms of its observable behavior. Because that's what effects make for: To make statements like "does not panic" or "does not allocate" or "does not yield". "Has been manually proven correct" is very, very different from that in that it can be hidden behind an abstraction barrier.

But anyway, this is getting off-topic. ;)

[SimonSapin]

I’m worried about the migration of existing code.

I’d like to see this RFC make it a requirement that rustfix / cargo fix need to fully support automating the necessary code changes, before this lint can warn by default.

But this is tricky, simply wrapping the entire body of a function into a new unsafe block sort of defeats the purpose of this change. On the other hand it would likely be very noisy to minimize the size of generated unsafe blocks as much as possible by wrapping each unsafe fn call (or other operation that needs it) separately. Finding a balance between those likely requires case-by-case subjective human judgment.

[RalfJung]

But this is tricky, simply wrapping the entire body of a function into a new unsafe block sort of defeats the purpose of this change.

I wouldn't necessarily say so. It still provides benefits for new unsafe code written later, and it permits gradual migration of existing unsafe code.

[SimonSapin]

Good point, my "sort of" only applies to existing code. I didn’t meant to argue against this RFC, I was only pondering the merits of different ways to deal with the migration. Sorry if I implied otherwise.

[RalfJung]

It's okay. :) I will add something about migrations to the RFC text.

[newpavlov]

I wonder how many of existing unsafe fns would simply wrap the whole function body with unsafe block with the proposed change. If it's more than 90% (for my code I think its true), I think it will be better to introduce some kind of ununsafe/safe block which will turn on safety checks for a wrapped code. I would hate if code like this will be common:

unsafe fn foo() {
    unsafe {
        // ..
    }
}

And while treating unsafe fn as an effect is debatable (I agree with @RalfJung argumentation here), but I think that the current behaviour is consistent and easy to understand, while the snippet above can be somewhat surprising for new users.

[gnzlbg]

@mark-i-m I didn't intend to suggest that we should set that precedent or provide such a macro. I just wanted to point out that if in some application domains typing unsafe twice becomes too painful, user-defined proc macros might be able to alleviate this pain a bit.

[Lokathor]

I'm with mark-i-m on the proc-macro thing. Both of those options seems clearly worse than what we have today. I don't want any magic transformations happening when unsafe is on the scene. Everything should be extremely clear because you're playing with nuclear fire, and we're playing with it by hand.

@RalfJung The function body is one line. "If it ever executes unsafe code", yes of course that's a bug! That's why you don't go adding random other lines into any place in any module that ever uses unsafe code. unsafe code is definitionally unverifiable by the rules of rust alone. It has to be analyzed by a human. That means that we want to make the job of the human as easy as possible.

Right now, we have things like this:

pub unsafe fn set_len(&mut self, new_len: usize){
  self.len = new_len;
}

And your claim is that "it's all safe operations, so you should write a safe internal function to do it and then call that safe internal function"? No. That's wrong. Please read the Rustonomicon again. One unsafe block anywhere in a module should be assumed to taint the entire module.

The line self.len = new_len; is safe only because Rust is insufficient at this time. That's an unsafe, highly dangerous operation that should never be executed from safe code. That's just as dangerous as doing ptr.offset(val), and there should not be a safe internal method that someone in the future might think to call from within safe code to change the length. We don't want to think of assigning a new value to the length field to be thought of as a safe operation.

What we want is the polar opposite: an RFC where you can mark a field as being an unsafe field where you can only read or write from it in an unsafe block. Now that would be an improvement to our safety system.

[RalfJung]

And your claim is that "it's all safe operations, so you should write a safe internal function to do it and then call that safe internal function"?

No, I never claimed that. It should be an unsafe fn whose body is checked for not performing any unsafe operations. That's not the same thing as a "safe internal function".

One unsafe block anywhere in a module should be assumed to taint the entire module.

I am well aware.

Maybe set_len wasn't a great example (though I still stand by my point). I listed some more examples further up this thread for why it is harmful that the body of an unsafe fn is an implicit unsafe block. There exist important and non-trivial unsafe fn that are way longer than one line, and their author should get all the help from the compiler that we can give them. Currently, we give them less than the standard amount of help by not even pointing out when they accidentally use an unsafe operation outside of an unsafe block.

[Lokathor]

Oh, hold on, my deepest apologies @RalfJung, it was @burdges who wanted the safe internal function split up.

Still, if we agree that the whole module is tainted, wouldn't you also agree that assuming that we can reduce the unsafety by making the scope of the unsafe blocks smaller is largely a misleading notion?

[eternaleye]

@Lokathor You're conflating two distinct concerns.

Making the bodies safe by default is not about reducing the scope of unsafe potentially causing problems - it's about increasing the precision with which proof obligations (unsafe fn) and obligations being discharged (unsafe {}) are denoted and distinguished from one another.

The whole module is tainted, yes... but only if there is an obligation that is neither discharged nor deferred to the module's user.

[RalfJung]

To add to what @eternaleye said: The point of unsafe blocks is also to make sure that you don't call an unsafe function without realizing. That's why it is good practice to keep your unsafe blocks small -- not because the surrounding code cannot also cause problems if it is incorrect, but because this excludes some of the problems it could cause. We have to carefully review the surrounding code to uphold whatever is required for the unsafe block, but at least we don't have to also review it for causing UB on its own. That's useful.

Quoting from the first motivational paragraph of this RFC:

Even if the programmer does not read the documentation, calling an unsafe function (or performing another unsafe operation) outside an unsafe block will lead to a compile error, hopefully followed by reading the documentation.

Your arguments go besides the point here.

[burdges]

There are numerous examples linked up thread:
#2585 (comment)
#2585 (comment)
I'll step through one in more detail though:

There are five unsafe fns in btree::map four of which have entirely safe bodies, except for calling the fifth unwrap_unchecked, which itself is entirely safe, except for only doing a test in debug builds.

In other word, unwrap_unchecked is marked unsafe, not even so that the programmer proves the existence of the leaf at the call site, but so that they prove that their tests catch any case where this fails, which keeps the unsafety fairly local. Yet, this aim is largely defeated by being called from unsafe fns where you must check every call site.

If you read this code, all your time gets spent flipping over to btree::node to determine if the various private methods invoked are themselves unsafe fns with proof obligations, which is an enormous distraction from figuring out if the code satisfies the proof obligation of unwrap_unchecked by instead reading the tests.

[Lokathor]

People keep linking me to #2585 (comment) but I've got a thumbs up on it so I do not know why you think I have not seen it.

I'd be more convinced if unsafe code had a way of classifying the dangers at any given moment and masking out some of them but not all of them was possible, so then the compiler can tell you want you missed. Like checked exceptions in other languages. As it stands, the idea that you're proving some things but not others doesn't really line up with how an unsafe block works.

[eternaleye]

@Lokathor Again, you're conflating two things. The annotation unsafe fn is telling the compiler "there exist proof obligations here" without telling the compiler what they are because doing so is difficult and likely burdensome. That's the kind of thing that does require full-on dependent types, etc. Similarly, unsafe {} is telling the compiler "given the current context, all proof obligations within this block have been satisfied" - again, without actually specifying them, because doing so would require vastly increased language complexity.

However, separating the two makes it much, much more feasible to improve that in the future - including through using structured comments parsed with external tooling, proc macros, or other means.

unsafe is important not for marking what the proof obligations being acted on are, but for marking where they are being acted on. That's the first step - and a crucial one - to being able to do the rest.

EDIT: Notably, there exist proof obligations that are checked by the compiler. This is what a type system is. unsafe is only needed for obligations that are problematic for the compiler to check, so asking for unsafe code to have "a way of classifying the dangers" so that "the compiler can tell you [what] you missed" is asking the Rust developers to make unsafe unnecessary.

[ubsan]

I've personally never seen code that accidentally did unsafe things in an unsafe fn, and I've written a lot of unsafe code. This seems like a change without a lot of real-world code that would benefit. I also tend to disagree with the idea of "minimizing the scope of unsafe" - one should make the scope of unsafe as small as makes code readable

[RalfJung]

I've read a lot of unsafe code, and I often find it non-trivial to look out for accidental unsafe operations.

[Lokathor]

I'm not conflating anything that doesn't already go together. The caller having a proof burden despite you using an unsafe block obviously means that you didn't really discharge all of the proof burden involved. That's why I don't like it. If you were enhancing unsafe so that it was something like:

unsafe (alignment, ptr_location, valid_bits) {
  self.ptr.read()
}

And you could discharge some proofs but not all, and then the rest would float up to the caller, that would be amazing and fruitful. I'd support that in a heartbeat.

[ubsan]

@Lokathor that's more of an "effect" model of unsafety, and I think that'd be really cool!

[cramertj]

+1 to @RalfJung's comment about reading vs. writing-- I personally wouldn't find this that big of a deal for code that I write myself, but it'd be really helpful when reviewing others code. (I have, on occasion, however, found bugs in others' code that resulted from "unaware-unsafe")

[eternaleye]

@Lokathor

The caller having a proof burden despite you using an unsafe block obviously means that you didn't really discharge all of the proof burden involved.

This is a misunderstanding both of "proof burden" and of "discharge".

Discharging a proof burden does not mean that you show it is always true no matter what. Discharging a proof burden means that you show it is implied by things that are already believed to be true in context.

unsafe fn says (to its callers) "I need you to make sure (some things) are true", as well as (to its body) "you are allowed to presume that (those same things) are true". Meanwhile, unsafe {} says that "the things my body requires to be true are implied by things my outer scope tells me are true".

For example, consider:

// UB if `! (1..=v.len()).contains(p)`
unsafe fn foo<T>(v: &Vec<T>, p: usize) -> &T {
    // IF this function's proof obligation is satisfied, THEN the proof obligation
    // inside here is satisfied
    unsafe {
        // UB if `! (0..v.len()).contains(p-1)`
        v.index_unchecked(p-1)
    }
}

It is clearly an unsafe function (it has UB if called with a value outside 1..=v.len()), and it calls an unsafe function (unchecked indexing) - but their proof obligations are different: unchecked indexing has UB if its value is outside 0..v.len(). However, the proof obligation of unchecked indexing is fully satisfied from:

1. The proof obligation of foo and
2. That its argument is one less than foo's argument.

[burdges]

We improved tiny-keccek's performance by replacing some unsafe code with safe code, especially removing one unsafe fn if I recall. We likely risk more small performance penalties with larger unsafe code blocks. As with humans reading code, these larger unsafe blocks mean the compiler "understands" less about our proof obligations and how we satisfy them.

[RalfJung]

@burdges Now that's weird. Can you point to the concrete change that lead to a performance improvemenet?

unsafe blocks itself have no effect at all on codegen, they are just use by the type checker.

[burdges]

In that case, it's presumably merely that Rust's optimizations expect safe coding styles, but Niko suggests that some future optimizations might only work when far enough away from unsafe code.

Anyways, the original PR is debris/tiny-keccak#8 but the merged version is debris/tiny-keccak#11 Another popped up this year in debris/tiny-keccak#36 but only because they never merged the bounds check eliding from debris/tiny-keccak@ccf709b

[RalfJung]

@burdges Thanks! I doubt it is about the safe vs. unsafe distinction that anything got faster there. Seems more like you also went from slices to arrays, which helps because more is statically known.

[graydon]

Mixed feelings. This was dealt with at length in the initial design and the argument that you mention here that most unsafe functions are short and this would make them read noisily as unsafe fn foo() { unsafe { .. } } carried the decision then. I'm not sure that's substantially changed.

[Centril]

read noisily as unsafe fn foo() { unsafe { .. } } carried the decision then. I'm not sure that's substantially changed.

If we could write unsafe fn foo() = unsafe { .. }; I think that would make it less weird. Allowing fn foo(...) = expr; would have other benefits as well (tho lets not go too deeply into that here).

[pqnet]

It could be solved the other way around, by introducing a safe { } block. Old unsafe function would still work, the extra bit of clutter would not affect much unsafe marked functions which are doing (mostly) safe things (the need to mark them safe is because they were problematically big anyway). It would need another keyword though :-(

[phil-opp]

Regarding the noise:

I strongly prefer the noise of unsafe fn foo() { unsafe { .. } } over the noise that is currently required to get a safe environment inside an unsafe fn:

unsafe fn foo() {
    /// Documentation that explains why this inner function is needed
    fn foo_inner() {
        // the actual implementation, but with more rightward drift
    }
    foo_inner()
}

This is a lot of boilerplate, especially if the actual implementation is very short. So it's tempting to just use the default unsafe environment, even if it results in a larger unsafe block than necessary. This seems like a bad default for Rust, which normally tries to make the safe implementation simpler.

[mark-i-m]

It looks like the discussion has largely settled and most of it seems to be about whether this would be a noise improvement or a noise increase. Does that sound right?

[burdges]

Assuming this RFC passes, then clippy could start warning not merely about unnecessary unsafe, but about unsafe blocks being larger than necessary, although the logic for this sounds delicate. If we did that now then we'd force people into excessive nested fns.

[gnzlbg]

It looks like the discussion has largely settled and most of it seems to be about whether this would be a noise improvement or a noise increase. Does that sound right?

I think there are three questions whose answers build on each other.

Question one is whether doing things either way can be useful depending on the project. If the answer is yes, we could just add an allow-by-default lint that raises an error if an unsafe { } is not used to call other unsafe code from unsafe fn. This would allow those that want the explicitness to opt-in to this behavior.

Question two is what should the default behavior be. That is, should the above lint be allow or warn by default ? (note that we can't turn this lint to error by default in either of the current editions)

Question three is whether that lint should be turned into an error in the next edition.

---

We don't have to resolve all these questions now. I personally think that both the RFC and the discussion show that adding such a lint would be an useful thing to do, and that allows people to choose whatever fits them or their projects best. This would also let us collect experience about how "noisy" such a lint actually is.

We could leave what the defaults should be as a question to be resolved in the tracking issue, and require that resolving it should have its own FCP, were a summary comment based on experience with the feature can kickstart the discussion to avoid repeating all the arguments.

It only makes sense to discuss whether this lint should error in the next edition iff we decide that it should warn by default in the current ones. It might well be that we recognize that, while this lint is a better default, it isn't the best behavior for some projects, and therefore, it should always be possible to disable it So even if we make it warn by default, that doesn't necessarily mean that it will become an impossible to disable error in future editions.

[gnzlbg]

I think this would be an useful lint to have, and I'd like to get more experience with it by using it in my projects and seeing how others use it in theirs before I fully make my mind about whether it should be enabled by default or not.

[phil-opp]

We would also need to change the unused_unsafe lint, since it currently warns when unsafe is used inside an unsafe function:

warning: unnecessary `unsafe` block
 --> src/main.rs:6:5
  |
5 | unsafe fn test () {
  | ----------------- because it's nested under this `unsafe` fn
6 |     unsafe { foo() }
  |     ^^^^^^ unnecessary `unsafe` block
  |
  = note: #[warn(unused_unsafe)] on by default