Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[vm/ffi] MSAN unpoison stores into Pointers and TypedDatas #52399

Closed
4 tasks
dcharkes opened this issue May 15, 2023 · 6 comments
Closed
4 tasks

[vm/ffi] MSAN unpoison stores into Pointers and TypedDatas #52399

dcharkes opened this issue May 15, 2023 · 6 comments
Assignees
Labels
area-vm Use area-vm for VM related issues, including code coverage, FFI, and the AOT and JIT backends. library-ffi

Comments

@dcharkes
Copy link
Contributor

dcharkes commented May 15, 2023

We don't unpoisen the memory when doing storing data into Pointers.

If we lookup malloc in the process, this malloc is instrumented, we store data into the memory with dart:ffi and later we use that memory in C++ code again which is instrumented, the memory shows up as uninitialized in MSAN.

We should add unpoison calls to the FFI stores. We already do something similar in FFI calls:

#if defined(USING_MEMORY_SANITIZER)
{
RegisterSet kVolatileRegisterSet(CallingConventions::kVolatileCpuRegisters,
CallingConventions::kVolatileXmmRegisters);
__ movq(temp, RSP);
__ PushRegisters(kVolatileRegisterSet);
// Outgoing arguments passed on the stack to the foreign function.
__ movq(CallingConventions::kArg1Reg, temp);
__ LoadImmediate(CallingConventions::kArg2Reg, stack_space);
__ CallCFunction(
compiler::Address(THR, kMsanUnpoisonRuntimeEntry.OffsetFromThread()));
// Incoming Dart arguments to this trampoline are potentially used as local
// handles.
__ movq(CallingConventions::kArg1Reg, is_leaf_ ? FPREG : saved_fp);
__ LoadImmediate(CallingConventions::kArg2Reg,
(kParamEndSlotFromFp + InputCount()) * kWordSize);
__ CallCFunction(
compiler::Address(THR, kMsanUnpoisonRuntimeEntry.OffsetFromThread()));
// Outgoing arguments passed by register to the foreign function.
__ LoadImmediate(CallingConventions::kArg1Reg, InputCount());
__ CallCFunction(compiler::Address(
THR, kMsanUnpoisonParamRuntimeEntry.OffsetFromThread()));
__ PopRegisters(kVolatileRegisterSet);
}
#endif

We should also then cover the TypedData's as with Pointer.asTypedList we can also emit stores from that machine code.

Instructions to cover:

  • StoreIndexedInstr (loads/stores through Pointer, Struct access, FFI callbacks)
  • MemoryCopyInstr (.asTypedList().setRange())

Architectures to cover:

  • x64
  • Arm64
    RISC-V (but our current clang does not support RISC-V, so MSAN is not run on RISC-V currently)
@dcharkes dcharkes added area-vm Use area-vm for VM related issues, including code coverage, FFI, and the AOT and JIT backends. library-ffi labels May 15, 2023
@dcharkes dcharkes changed the title [vm/ffi] MSAN unpoison stores into Pointers [vm/ffi] MSAN unpoison stores into Pointers and TypedDatas May 15, 2023
@dcharkes
Copy link
Contributor Author

repro:

import 'dart:ffi';

main() {
  // final memory = calloc(8, 1).cast<Int8>(); // fine
  final memory = malloc(8).cast<Int8>(); // error
  final typedList1 = memory.asTypedList(8);
  final readVal = typedList1[0];
  print(readVal);
}

@Native<Pointer<Void> Function(IntPtr num, IntPtr size)>(isLeaf: true)
external Pointer<Void> calloc(int num, int size);

@Native<Pointer<Void> Function(IntPtr)>(isLeaf: true)
external Pointer<Void> malloc(int size);
tools/build.py -mrelease -ax64 --sanitizer=msan runtime ffi_test_functions && tools/test.py -n vm-msan-linux-release-x64 tests/ffi/regress_52399_test.dart
DART_CONFIGURATION=ReleaseMSANX64 out/ReleaseMSANX64/dart --sound-null-safety -Dtest_runner.configuration=vm-msan-linux-release-x64 --ignore-unrecognized-flags --packages=/usr/local/google/home/dacoharkes/dart-sdk/sdk/.dart_tool/package_config.json /usr/local/google/home/dacoharkes/dart-sdk/sdk/tests/ffi/regress_52399_test.dart

exit code:
-6

stderr:
==962527==WARNING: MemorySanitizer: use-of-uninitialized-value
/usr/local/google/home/dacoharkes/dart-sdk/sdk/buildtools/linux-x64/clang/bin/llvm-symbolizer: error: '/memfd:dart-code (deleted)': No such file or directory
    #0 0x55c262160ace in dart::DN_HelperTypedData_GetInt8(dart::Isolate*, dart::Thread*, dart::Zone*, dart::NativeArguments*) ../../out/ReleaseMSANX64/../../runtime/lib/typed_data.cc:248:1
    #1 0x55c262160ace in dart::BootstrapNatives::DN_TypedData_GetInt8(dart::Thread*, dart::Zone*, dart::NativeArguments*) ../../out/ReleaseMSANX64/../../runtime/lib/typed_data.cc:248:1
    #2 0x55c26235d929 in dart::NativeEntry::BootstrapNativeCallWrapper(_Dart_NativeArguments*, void (*)(_Dart_NativeArguments*)) ../../out/ReleaseMSANX64/../../runtime/vm/native_entry.cc:140:37
    #3 0x7f3b5cc82ba3  (/memfd:dart-code (deleted)+0x2ba3)
    #4 0x7f3b5a426281  (/memfd:dart-code (deleted)+0x26281)
    #5 0x7f3b5a4261c1  (/memfd:dart-code (deleted)+0x261c1)
    #6 0x7f3b5a4243aa  (/memfd:dart-code (deleted)+0x243aa)
    #7 0x7f3b5a4242b5  (/memfd:dart-code (deleted)+0x242b5)
    #8 0x7f3b5a4241ef  (/memfd:dart-code (deleted)+0x241ef)
    #9 0x7f3b5a4230b4  (/memfd:dart-code (deleted)+0x230b4)
    #10 0x7f3b5a422df2  (/memfd:dart-code (deleted)+0x22df2)
    #11 0x7f3b5a421eab  (/memfd:dart-code (deleted)+0x21eab)
    #12 0x7f3b5cc82f0b  (/memfd:dart-code (deleted)+0x2f0b)
    #13 0x55c262263c8b in dart::DartEntry::InvokeFunction(dart::Function const&, dart::Array const&, dart::Array const&) ../../out/ReleaseMSANX64/../../runtime/vm/dart_entry.cc:150:33
    #14 0x55c2622683d5 in dart::DartEntry::InvokeFunction(dart::Function const&, dart::Array const&) ../../out/ReleaseMSANX64/../../runtime/vm/dart_entry.cc:37:10
    #15 0x55c2622683d5 in dart::DartLibraryCalls::HandleMessage(long, dart::Instance const&) ../../out/ReleaseMSANX64/../../runtime/vm/dart_entry.cc:737:28
    #16 0x55c2622c7483 in dart::IsolateMessageHandler::HandleMessage(std::__2::unique_ptr<dart::Message, std::__2::default_delete<dart::Message>>) ../../out/ReleaseMSANX64/../../runtime/vm/isolate.cc:1291:15
    #17 0x55c262333996 in dart::MessageHandler::HandleMessages(dart::MonitorLocker*, bool, bool) ../../out/ReleaseMSANX64/../../runtime/vm/message_handler.cc:239:16
    #18 0x55c262334c6e in dart::MessageHandler::TaskCallback() ../../out/ReleaseMSANX64/../../runtime/vm/message_handler.cc:458:18
    #19 0x55c262655126 in dart::ThreadPool::WorkerLoop(dart::ThreadPool::Worker*) ../../out/ReleaseMSANX64/../../runtime/vm/thread_pool.cc:158:15
    #20 0x55c262655ee1 in dart::ThreadPool::Worker::Main(unsigned long) ../../out/ReleaseMSANX64/../../runtime/vm/thread_pool.cc:330:9
    #21 0x55c2624faeb7 in dart::ThreadStart(void*) ../../out/ReleaseMSANX64/../../runtime/vm/os_thread_linux.cc:154:5
    #22 0x7f3b5d65dfd3 in start_thread nptl/pthread_create.c:442:8

SUMMARY: MemorySanitizer: use-of-uninitialized-value ../../out/ReleaseMSANX64/../../runtime/lib/typed_data.cc:248:1 in dart::DN_HelperTypedData_GetInt8(dart::Isolate*, dart::Thread*, dart::Zone*, dart::NativeArguments*)
Exiting

--- Re-run this test:
python3 tools/test.py -n vm-msan-linux-release-x64 ffi/regress_52399_test
[00:01 | 100% | +    0 | -    1]

@dcharkes
Copy link
Contributor Author

DN_TypedData_GetInt8 only shows up in unoptimized code.

Unoptimized Compilation
==== file:///usr/local/google/home/dacoharkes/dart-sdk/sdk/tests/ffi/regress_52399_test.dart_::_repro52399 (RegularFunction)
B0[graph]:0
B1[function entry]:2
    CheckStackOverflow:8(stack=0, loop=0)
    DebugStepCheck:10()
    t0 <- LoadLocal(memory @1)
    t0 <- StaticCall:12( Int8Pointer|asTypedList<0> t0, t1)
    StoreLocal(typedList1 @-1, t0)
    t0 <- LoadLocal(typedList1 @-1)
    t0 <- InstanceCall:14( []<0>, t0, t1)
    StoreLocal(readVal @-2, t0)
    DebugStepCheck:16()
    Return:18(t0)
*** END CFG
*** BEGIN CFG
After AllocateRegisters
==== file:///usr/local/google/home/dacoharkes/dart-sdk/sdk/tests/ffi/regress_52399_test.dart_::_repro52399 (RegularFunction)
  0: B0[graph]:0 {
      v0 <- Constant(#null) T{Null?}
      v1 <- Constant(#<optimized out>) T{Sentinel~}
      v3 <- Constant(#8) [8, 8] T{_Smi}
      v5 <- Constant(#0) [0, 0] T{_Smi}
      v24 <- Constant(#TypeArguments: (Hae0ec1e) [Type: Pointer<Int8>]) T{TypeArguments}
      v25 <- Constant(#Pointer<Int8>) T{_OneByteString}
      v26 <- Constant(#TypeArguments: (H32c84ab6) [Type: int]) T{TypeArguments}
      v27 <- Constant(#length) T{_OneByteString}
      v28 <- Constant(#1) [1, 1] T{_Smi}
      v29 <- Constant(#_ImmutableList len:5) T{_ImmutableList}
      v46 <- Constant(#index) T{_OneByteString}
      v91 <- Constant(#Pointer address must be aligned to a multiple of the element size (1).) T{_OneByteString}
}
  2: B31[function entry]:2 {
      v2 <- Parameter(0) T{Pointer}
}
  4:     CheckStackOverflow:8(stack=0, loop=0)
  6:     MoveArgument(v24 T{TypeArguments}, SP+2)
  8:     MoveArgument(v2 T{Pointer}, SP+1)
 10:     MoveArgument(v25 T{_OneByteString}, SP+0)
 12:     StaticCall:62( checkNotNull<1> v24 T{TypeArguments}, v2 T{Pointer}, v25 T{_OneByteString})
 14:     MoveArgument(v26 T{TypeArguments}, SP+2)
 16:     MoveArgument(v3 T{_Smi}, SP+1)
 18:     MoveArgument(v27 T{_OneByteString}, SP+0)
 20:     StaticCall:64( checkNotNull<1> v26 T{TypeArguments}, v3 T{_Smi}, v27 T{_OneByteString})
 22:     MoveArgument(v2 T{Pointer}, SP+0)
 24:     v17 <- StaticCall:68( get:address<0> v2 T{Pointer}, recognized_kind = FfiGetAddress) [-9223372036854775808, 9223372036854775807] T{int}
 26:     CheckSmi:14(v17)
 28:     Branch if TestSmi(v17 T{_Smi}, v5) goto (34, 35)
 30: B34[target]:24
 32:     v81 <- AllocateObject:26(cls=ArgumentError) T{ArgumentError}
 34:     ParallelMove S-3 <- rax
 34:     MoveArgument(v81, SP+1)
 36:     MoveArgument(v91 T{_OneByteString}, SP+0)
 38:     StaticCall:32( ArgumentError.<0> v81, v91 T{_OneByteString})
 39:     ParallelMove rax <- S-3
 40:     Throw:34(v81)
 42: B35[target]:36
 44:     MoveArgument(v2 T{Pointer}, SP+1)
 46:     MoveArgument(v3 T{_Smi}, SP+0)
 48:     v18 <- StaticCall:72( _asExternalTypedDataInt8@8050071<0> v2 T{Pointer}, v3 T{_Smi}, recognized_kind = FfiAsExternalTypedDataInt8) T{Int8List}
 50:     ParallelMove S-4 <- rax
 50:     CheckClass:14(v18 T{Int8List} Cids[1: _ExternalInt8Array@7027147 etc.  cid 115])
 52:     v101 <- LoadField(v18 T{_ExternalInt8Array} . TypedDataBase.length {final}) [0, 4611686018427387903] T{_Smi}
 53:     ParallelMove S-3 <- rcx
 54:     Branch if RelationalOp:22(>=, v5, v101) T{bool} goto (20, 23)
 56: B20[target]:38
 58:     v41 <- AllocateObject:42(cls=IndexError) T{IndexError}
 60:     ParallelMove S-5 <- rax
 60:     MoveArgument(v41, SP+4)
 62:     MoveArgument(v5 T{_Smi}, SP+3)
 64:     MoveArgument(v101 T{_Smi}, SP+2)
 66:     MoveArgument(v18 T{_ExternalInt8Array}, SP+1)
 68:     MoveArgument(v46 T{_OneByteString}, SP+0)
 70:     StaticCall:46( IndexError.withLength<0> v41, v5 T{_Smi}, v101 T{_Smi}, v18 T{_ExternalInt8Array}, v46 T{_OneByteString})
 71:     ParallelMove rax <- S-5
 72:     Throw:48(v41)
 74: B23[target]:50
 75:     ParallelMove rax <- C
 76:     Return:18(v0)
*** END CFG

@dcharkes
Copy link
Contributor Author

From an offline discussion with @mraleph:

I think we should just instrument all pointer and typed data stores to call C helpers which execute the store if we are running with MSAN/TSAN/etc.
Feels less fragile then other alternatives.
We can also cheat a bit and not do a full blown C call because we know that these helpers are going to be very optimised

I've explored this a bit, and found two implementation strategies:

  • Implementing this in IL (architecture agnostic) for FFI stores kernel_to_il.cc and TypedData stores inliner.cc by doing a CCallInstr instead of a StoreIndexedInstr.
  • Implementing this in Machine code: StoreIndexedInstr::EmitNative and adding a bool to the instruction so that we can mark the call sites that we want to move over to doing calls.

I have made a prototype for the IL approach for FFI stores in: https://dart-review.googlesource.com/c/sdk/+/303360/5

This fixes the reproduction I pasted above.
However, it also seems to introduce new sanitizer failures in existing tests, which should be investigated.

Before I continue on this path, maybe we should consider the MC implementation instead. It will have to be duplicated for all archs likely (maybe we can use arch agnostic assembler commands), however, we don't have to bother constructing duplicate IL in various places.

@dcharkes
Copy link
Contributor Author

StoreIndexedInstr on x64: https://dart-review.googlesource.com/c/sdk/+/303360

copybara-service bot pushed a commit that referenced this issue Sep 19, 2023
TEST=ffi/function_callbacks_structs_by_value_generated_test
TEST=ffi/function_callbacks_structs_by_value_native_callable_generated_test
TEST=ffi/regress_52399_test.dart

Bug: #52399
Change-Id: Id16ccea5645d9b14a8f2726cb896b99266bba5a2
Cq-Include-Trybots: luci.dart.try:vm-msan-linux-release-x64-try,vm-aot-msan-linux-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/303360
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
copybara-service bot pushed a commit that referenced this issue Sep 22, 2023
TEST=ffi/function_structs_by_value_generated_ret_arg_native_test
TEST=MSAN SDK build, exercises this instruction a lot.

Bug: #52399
Change-Id: Id65a6c4e5500afd2a155d609f8e0144a157aa3b0
Cq-Include-Trybots: luci.dart.try:vm-msan-linux-release-x64-try,vm-aot-msan-linux-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/327201
Reviewed-by: Tess Strickland <sstrickl@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
copybara-service bot pushed a commit that referenced this issue Nov 28, 2023
TEST=local
Bug: #52399
Change-Id: I6af1954c70208534e87b6b87dd5cf62aaf595bd3
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/329582
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
@a-siva
Copy link
Contributor

a-siva commented Dec 7, 2023

@dcharkes I noticed you have submitted some PRs here, is this issue fixed and can be closed ?

@dcharkes
Copy link
Contributor Author

dcharkes commented Dec 8, 2023

The current results only shows 1 test failing on ubsan, and that test is failing on other configurations as well.

So, I believe we've covered everything.

@dcharkes dcharkes closed this as completed Dec 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-vm Use area-vm for VM related issues, including code coverage, FFI, and the AOT and JIT backends. library-ffi
Projects
None yet
Development

No branches or pull requests

2 participants