Tests fails under miri

[gzp79]

It seems as miri test is broken. Could you please investigate it ?

Undefined Behavior: dereferencing pointer failed: 0x1fee70[noalloc] is a dangling pointer (it has no provenance)
   --> src\smallbox.rs:362:18
    |
362 |         unsafe { &*self.as_ptr() }
    |                  ^^^^^^^^^^^^^^^ dereferencing pointer failed: 0x1fee70[noalloc] is a dangling pointer (it has no provenance)
    |
    = help: this indicates a bug in the program: it performed an invalid operation, and caused Undefined Behavior
    = help: see https://doc.rust-lang.org/nightly/reference/behavior-considered-undefined.html for further information
    = note: BACKTRACE:
    = note: inside `<smallbox::SmallBox<usize, space::S1> as std::ops::Deref>::deref` at src\smallbox.rs:362:18: 362:33
note: inside `smallbox::tests::test_basic`
   --> src\smallbox.rs:468:17
    |
468 |         assert!(*stacked == 1234);
    |                 ^^^^^^^^
note: inside closure
   --> src\smallbox.rs:466:21
    |
465 |     #[test]
    |     ------- in this procedural macro expansion
466 |     fn test_basic() {
    |                     ^
    = note: this error originates in the attribute macro `test` (in Nightly builds, run with -Z macro-backtrace for more info)

note: some details are omitted, run with `MIRIFLAGS=-Zmiri-backtrace=full` for a verbose backtrace

error: aborting due to previous error

test smallbox::tests::test_basic ... error: test failed, to rerun pass `--lib`

Caused by:
  process didn't exit successfully: `C:\Users\zpgaa\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\bin\cargo-miri.exe runner D:\work\3rdparty\smallbox\target\miri\x86_64-pc-windows-msvc\debug\deps\smallbox-932a3225d7a60a4d.exe` (exit code: 1)

smallbox on  master is 📦 v0.8.1 via 🦀 v1.69.0-nightly took 2s

[andylokandy]

Miri fails because Smallbox employs a technique called "pointer-usize-pointer roundtrips", which cannot be expressed within the constraints of Strict Provenance. This doesn't necessarily indicate that the code has undefined behavior, but it means that tools like MIRI cannot prove that it doesn't. The changes required to comply with Strict Provenance rules are currently under the nightly feature gate #[feature(strict_provenance)], so we are unable to act at this time.

[GoldsteinE]

I don’t think ptr-int-ptr roundtrips are the only problem here. Consider this patch:

Patch

diff --git a/src/smallbox.rs b/src/smallbox.rs
index 499dd7b..8308f99 100644
--- a/src/smallbox.rs
+++ b/src/smallbox.rs
@@ -182,8 +182,8 @@ impl<T: ?Sized, Space> SmallBox<T, Space> {

         // Overwrite the pointer but retain any extra data inside the fat pointer.
         let mut ptr = ptr;
-        let ptr_ptr = &mut ptr as *mut _ as *mut usize;
-        ptr_ptr.write(ptr_addr as usize);
+        let ptr_ptr = &mut ptr as *mut _ as *mut *const u8;
+        ptr_ptr.write(ptr_addr);

         ptr::copy_nonoverlapping(val as *const _ as *const u8, ptr_copy, size);

@@ -221,11 +221,9 @@ impl<T: ?Sized, Space> SmallBox<T, Space> {
     unsafe fn as_ptr(&self) -> *const T {
         let mut ptr = self.ptr;

-        if !self.is_heap() {
-            // Overwrite the pointer but retain any extra data inside the fat pointer.
-            let ptr_ptr = &mut ptr as *mut _ as *mut usize;
-            ptr_ptr.write(self.space.as_ptr() as *const () as usize);
-        }
+        // Overwrite the pointer but retain any extra data inside the fat pointer.
+        let ptr_ptr = &mut ptr as *mut *const T as *mut *const Space;
+        ptr_ptr.write(self.space.as_ptr());

         ptr
     }

It gets rid of such roundtrips and tests still fail under Miri, with both Stacked Borrows and Tree Borrows.

Error with Stacked Borrows

test smallbox::tests::test_basic ... error: Undefined Behavior: trying to retag from <167105> for SharedReadOnly permission at alloc63930[0x8], but that tag does not exist in the borrow stack for this location
   --> src/smallbox.rs:357:18
    |
357 |         unsafe { &*self.as_ptr() }
    |                  ^^^^^^^^^^^^^^^
    |                  |
    |                  trying to retag from <167105> for SharedReadOnly permission at alloc63930[0x8], but that tag does not exist in the borrow stack for this location
    |                  this error occurs as part of retag at alloc63930[0x0..0x10]
    |
    = help: this indicates a potential bug in the program: it performed an invalid operation, but the Stacked Borrows rules it violated are still experimental
    = help: see https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md for further information
help: <167105> was created by a SharedReadOnly retag at offsets [0x0..0x8]
   --> src/smallbox.rs:226:23
    |
226 |         ptr_ptr.write(self.space.as_ptr());
    |                       ^^^^^^^^^^^^^^^^^^^
    = note: BACKTRACE (of the first span):
    = note: inside `<smallbox::SmallBox<(usize, usize), space::S1> as std::ops::Deref>::deref` at src/smallbox.rs:357:18: 357:33
note: inside `smallbox::tests::test_basic`
   --> src/smallbox.rs:463:17
    |
463 |         assert!(*heaped == (0, 1));
    |                 ^^^^^^^
note: inside closure
   --> src/smallbox.rs:458:21
    |
457 |     #[test]
    |     ------- in this procedural macro expansion
458 |     fn test_basic() {
    |                     ^
    = note: this error originates in the attribute macro `test` (in Nightly builds, run with -Z macro-backtrace for more info)

note: some details are omitted, run with `MIRIFLAGS=-Zmiri-backtrace=full` for a verbose backtrace

error: aborting due to previous error

error: test failed, to rerun pass `--lib`

Error with Tree Borrows

test smallbox::tests::test_basic ... error: Undefined Behavior: using uninitialized data, but this operation requires initialized memory
    --> /nix/store/dc9whnls6hmwbja38fny60xn1apl97np-rust-default-1.70.0-nightly-2023-04-10/lib/rustlib/src/rust/library/core/src/cmp.rs:1330:5
     |
1330 | /     partial_eq_impl! {
1331 | |         bool char usize u8 u16 u32 u64 u128 isize i8 i16 i32 i64 i128 f32 f64
1332 | |     }
     | |_____^ using uninitialized data, but this operation requires initialized memory
     |
     = help: this indicates a bug in the program: it performed an invalid operation, and caused Undefined Behavior
     = help: see https://doc.rust-lang.org/nightly/reference/behavior-considered-undefined.html for further information
     = note: BACKTRACE:
     = note: inside `std::cmp::impls::<impl std::cmp::PartialEq for usize>::eq` at /nix/store/dc9whnls6hmwbja38fny60xn1apl97np-rust-default-1.70.0-nightly-2023-04-10/lib/rustlib/src/rust/library/core/src/cmp.rs:1310:52: 1310:59
     = note: inside `std::tuple::<impl std::cmp::PartialEq for (usize, usize)>::eq` at /nix/store/dc9whnls6hmwbja38fny60xn1apl97np-rust-default-1.70.0-nightly-2023-04-10/lib/rustlib/src/rust/library/core/src/tuple.rs:32:37: 32:72
note: inside `smallbox::tests::test_basic`
    --> src/smallbox.rs:463:17
     |
463  |         assert!(*heaped == (0, 1));
     |                 ^^^^^^^^^^^^^^^^^
note: inside closure
    --> src/smallbox.rs:458:21
     |
457  |     #[test]
     |     ------- in this procedural macro expansion
458  |     fn test_basic() {
     |                     ^
     = note: this error originates in the macro `partial_eq_impl` which comes from the expansion of the attribute macro `test` (in Nightly builds, run with -Z macro-backtrace for more info)

note: some details are omitted, run with `MIRIFLAGS=-Zmiri-backtrace=full` for a verbose backtrace

error: aborting due to previous error

error: test failed, to rerun pass `--lib`

Interestingly, Miri with Tree Borrows mentions reading uninitialized memory, which is certainly UB, regardless of which borrowing model will be adopted at the end. I haven’t been able to pinpoint why this happens though, and it might be a bug in Miri.

Anyway, I think that deserves further investigation.

[andylokandy]

It gets rid of such roundtrips and tests still fail under Miri, with both Stacked Borrows and Tree Borrows.

@GoldsteinE I believe that this patch doesn't satisfy the 'strict provenance' requirement, which necessitates manipulating the pointer using the API associated with #[feature(strict_provenance)]. Otherwise, information about the memory to which the pointer is directed will be lost in the miri context (when built with miri, the pointers do carry extra information about the memory, and the strict_provenance API preserves that information).

[GoldsteinE]

Strict provenance requirements aren’t checked by Miri by default AFAIK, so SB/TB failures can’t be caused by them. Miri treats as casts as behaving with “angelic non-determinism”, so if there is any “exposed” provenance that can be used without causing UB, it will be used.

[GoldsteinE]

I’m also not sure which strict provenance APIs you’re talking about. It seems like there’s no roundtrip at all, so no special APIs are needed?

[andylokandy]

Yeah, I know talk is cheap. Therefore, I tested the new feature-gated ptr APIs and as expected, Miri passed without any issues.

[GoldsteinE]

That’s cool! Does it work with sptr backport?

[andylokandy]

Here are the changes:

---

[andylokandy]

That’s cool! Does it work with sptr backport?

It needs an additional #![feature(set_ptr_value)] feature gate, but I didn't find a polyfill for it yet.

[GoldsteinE]

I also feel like “works with strict provenance APIs, but not ptr-to-ptr casts” could be a bug in Miri. Maybe it’s worth reporting this case to Miri and/or UCG.

[andylokandy]

In your patch, the address of ptr is still directly derived from an usize ptr_addr. This is a ptr-usize-ptr roundtrip.

let ptr_ptr = &mut ptr as *mut _ as *mut *const u8;
ptr_ptr.write(ptr_addr);

[GoldsteinE]

Oops, fair enough. I’ll try to make a working patch without sptr APIs later today.

[andylokandy]

Oh my bad, ptr_addr is indeed a pointer...

[GoldsteinE]

@andylokandy Could you provide your sptr patch as a file? I’ll try to tinker with it some more.

[andylokandy]

@GoldsteinE You can see this #22