Strictly sanitize mmapped AppendVec file contents #7464

ryoqun · 2019-12-13T02:25:50Z

Problem

Currently, It's very easy to cause DoS with crafted AppendVec data file. That's because data_len is directly used to allocate the data_len number of u8[], and is used for the offset calculation without overflow check, for example.

Also, I've carefully audited the fields in the AppendVec data file this time. Most of fields including Pubkey, Hash and lamports can legally contain arbitrary values for its type domain. So there aren't much to sanitize them at the AppendVec layer. However there are only two exceptions: data_len and executable.

As mentioned before, data_len must be sensible u64 for memory allocation. This is obvious and simple.

And exeutable is a bit subtle. It's bool consuming 1 bit logically in Rust land, but it consumes 8 bits physically. That means the higher 7 bits are usually not touched, however we must sanitize those bits to be cleared when snapshot ingestion. Otherwise, it's undefined behavior so bogus checks for exeutable could be possible depending on some myriad of combination of runtime configuration (rust version, compiler optimization, machine architecture, OS varieties).

After all, we should be super careful; we're fearless and very rare people to dare to mmap completely untrusted (=not even semi-trusted) data directly with minimal sanitization... :p We're proudly performance-obsessed. :)

Summary of Changes

Small preparatory clean up two commits
The actual meat including some unsafe {}s in both production and test code (mandatory due to the need to prepare malicious (=crafted) bytes and to guard against it)
- data_len: Protect by the way of strict offset calculation sanitization. This PR doesn't explicitly impose limits on it; In combination with Sanitize AppendVec's file_size #7373, it'll effectively limit huge memory allocation because data_len in this PR won't be greater than AppendVec's file_size.
- executable: Simply forbid any bad value other than 0b0000_0000 and 0b0000_00001.

Part of #7167

ryoqun · 2019-12-13T02:27:23Z

runtime/src/append_vec.rs

@@ -389,8 +417,8 @@ impl Serialize for AppendVec {
        S: serde::ser::Serializer,
    {
        use serde::ser::Error;
-        let len = std::mem::size_of::<usize>() as u64;


These casts are odd...

ryoqun · 2019-12-13T02:29:59Z

runtime/src/append_vec.rs

+
+        if !self.sanitize_layout_and_length() {
+            return Err(std::io::Error::new(
+                std::io::ErrorKind::Other,


I know using those Errors is a bit off...

Yea.. I would prefer using either a custom Result type or maybe even something like io::Result::InvalidInput https://doc.rust-lang.org/std/io/enum.ErrorKind.html#variant.InvalidInput

ryoqun · 2019-12-13T02:32:43Z

runtime/src/append_vec.rs

+        // Yes, this really hannpens; see test_set_file_crafted_executable
+        let executable_bool: &bool = &self.account_meta.executable;
+        // UNSAFE: Force to interpret mmap-backed bool as u8 to ensure higher 7-bits are cleared correctly.
+        let executable_byte: &u8 = unsafe { &*(executable_bool as *const bool as *const u8) };


This unsafe is in production code path. But risk should have been minimized; it only reads a byte of memory with narrowest scoping.

ryoqun · 2019-12-13T02:37:34Z

runtime/src/append_vec.rs

@@ -13,11 +13,12 @@ use std::{
    sync::Mutex,
 };

-//Data is aligned at the next 64 byte offset. Without alignment loading the memory may


I'm fairly certain 64 byte offset is wrong description; it should be 8 byte offset or 64 bit offset if you prefer bits. Padding at 64 byte boundary would be too wasteful. I've never heard of such architecture. Also, the macro impl doesn't look like actualy aligning with 64 byte, too.

Yea 64-byte in the description is wrong, but some vector instructions like vmovapd can require 64-byte alignment for avx-512 moves:
https://www.felixcloutier.com/x86/movapd

Of course compilers will probably always emit the unaligned-tolerant versions of those instructions.

avx-512 moves

Oh, the mighty 512 bits! Yeah, 64-byte alignment will be warranted in some special cases! Thanks for the tip!

ryoqun · 2019-12-13T02:42:32Z

runtime/src/append_vec.rs

@@ -187,17 +199,39 @@ impl AppendVec {

        let map = unsafe { MmapMut::map_mut(&data)? };
        self.map = map;
+
+        if !self.sanitize_layout_and_length() {


This adds additional sanitization costs for the snapshot ingestion codepath. Its impact on the overall validator performance should be minimal because it's only done only once when starting a validator from snapshot.

This PR intentionally didn't added these checks for the actual AppendVec write codepath for the performance concerns and its dubious merits.

Also this PR didn't add these check for snapshot generation code path as well with the same reason.

runtime/src/append_vec.rs

ryoqun · 2019-12-13T02:48:12Z

runtime/src/append_vec.rs

            return None;
        }
-        let data = &self.map[offset..offset + size];
-        //Data is aligned at the next 64 byte offset. Without alignment loading the memory may


IMO, these comments are redundant at best; so removed them.

ryoqun · 2019-12-13T02:54:04Z

runtime/src/append_vec.rs

+
+        av.flush().unwrap();
+        let result = av.set_file(path);
+        assert_matches!(result, Err(ref message) if message.to_string() == *"incorrect layout/length");


Better assertion could be possible...

codecov · 2019-12-13T02:56:35Z

Codecov Report

Merging #7464 into master will decrease coverage by 9.8%.
The diff coverage is 79%.

@@           Coverage Diff            @@
##           master   #7464     +/-   ##
========================================
- Coverage    80.7%   70.8%   -9.9%     
========================================
  Files         244     245      +1     
  Lines       48682   55276   +6594     
========================================
- Hits        39291   39170    -121     
- Misses       9391   16106   +6715

ryoqun · 2019-12-13T03:00:04Z

runtime/src/append_vec.rs

+            let executable_bool: &bool = &account.account_meta.executable;
+            // we can not use assert_eq!...
+            // *executable_bool is true but its actual memory value is crafted_executable, not 1
+            assert!(*executable_bool != true);


dark side of unsafe (part 1) xD

runtime/src/append_vec.rs

ryoqun · 2019-12-13T03:01:19Z

runtime/src/append_vec.rs

+            assert_eq!(executable_bool, false);
+            // UNSAFE: Force to interpret mmap-backed bool as u8 to really read the actual memory content
+            let executable_byte: u8 = unsafe { std::mem::transmute::<bool, u8>(executable_bool) };
+            assert_eq!(executable_byte, 0); // Wow, not crafted_executable!


dark side of unsafe (part 2) xD

sakridge · 2019-12-13T03:51:32Z

runtime/src/append_vec.rs

+            // *executable_bool is true but its actual memory value is crafted_executable, not 1
+            assert!(*executable_bool != true);
+            // UNSAFE: Force to interpret mmap-backed bool as u8 to really read the actual memory content
+            let executable_byte: &u8 = unsafe { &*(executable_bool as *const bool as *const u8) };


this unsafe block/casting is repeated in the tests a few times, can we have a function that is assert_eq_bool(ptr, expected_bool_value);

Yeah, I was a bit annoyed the repeated unsafes... Thanks for suggestion! I've done the cleanup differentially, though. How does that look for you?: 6d62daa

runtime/src/append_vec.rs

failures: ---- append_vec::tests::test_set_file_crafted_executable stdout ---- thread 'append_vec::tests::test_set_file_crafted_executable' panicked at 'assertion failed: `(left == right)` left: `true`, right: `true`', runtime/src/append_vec.rs:683:13 stack backtrace:

runtime/src/append_vec.rs

ryoqun · 2019-12-16T10:30:10Z

@mvines @sakridge I've polished this up! Could you review again in your free time? I think this PR is ready for merge. :)

mvines

lgtm, @sakridge is a better reviewer for this change though so I defer approval to him 👑

ryoqun · 2019-12-17T17:41:39Z

lgtm, @sakridge is a better reviewer for this change though so I defer approval to him crown

Thank you very much!

@sakridge How does this look now?

sakridge

lgtm

ryoqun added 3 commits December 13, 2019 08:43

Clean up align_to_8byte!

8af4cf8

small clean up

bc956ca

Strictly sanitize mmapped AppendVec files

a054cab

ryoqun requested a review from sakridge December 13, 2019 02:25

ryoqun commented Dec 13, 2019

View reviewed changes

runtime/src/append_vec.rs Show resolved Hide resolved

ryoqun commented Dec 13, 2019

View reviewed changes

ryoqun changed the title ~~Sanitize append vec mmap~~ Strictly sanitize mmapped AppendVec file contents Dec 13, 2019

ryoqun commented Dec 13, 2019

View reviewed changes

mvines reviewed Dec 13, 2019

View reviewed changes

runtime/src/append_vec.rs Outdated Show resolved Hide resolved

ryoqun commented Dec 13, 2019

View reviewed changes

ryoqun added 2 commits December 13, 2019 12:14

Clean up

2b301a1

Fix typo

e234d35

sakridge reviewed Dec 13, 2019

View reviewed changes

mvines reviewed Dec 13, 2019

View reviewed changes

runtime/src/append_vec.rs Outdated Show resolved Hide resolved

runtime/src/append_vec.rs Outdated Show resolved Hide resolved

ryoqun mentioned this pull request Dec 13, 2019

Include rent_epoch and executable into account hash #7415

Merged

ryoqun added 6 commits December 13, 2019 14:22

Rename align_to_8byte => u64_align

b1700ed

Fix typo

e0aca2a

Clean up unsafe into methods of StoredAccount

6d62daa

Yet more clarification

80877f6

Promote a PR comment into a src comment

dadb2e0

sakridge reviewed Dec 13, 2019

View reviewed changes

runtime/src/append_vec.rs Outdated Show resolved Hide resolved

sakridge reviewed Dec 13, 2019

View reviewed changes

runtime/src/append_vec.rs Outdated Show resolved Hide resolved

ryoqun added 2 commits December 16, 2019 16:00

Fix typo...

5785872

Move ref_executable_byte out of tests impl

581ae43

ryoqun requested review from mvines and sakridge December 16, 2019 08:03

mvines reviewed Dec 16, 2019

View reviewed changes

sakridge approved these changes Dec 17, 2019

View reviewed changes

ryoqun merged commit 629a4b5 into solana-labs:master Dec 18, 2019

ryoqun mentioned this pull request Jan 7, 2020

Fix AppendVec test breakage... #7693

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strictly sanitize mmapped AppendVec file contents #7464

Strictly sanitize mmapped AppendVec file contents #7464

ryoqun commented Dec 13, 2019

ryoqun Dec 13, 2019

ryoqun Dec 13, 2019

sakridge Dec 13, 2019

ryoqun Dec 13, 2019

ryoqun Dec 13, 2019

sakridge Dec 13, 2019

sakridge Dec 13, 2019

ryoqun Dec 13, 2019

ryoqun Dec 13, 2019

ryoqun Dec 13, 2019 •

edited

ryoqun Dec 13, 2019

ryoqun Dec 13, 2019

codecov bot commented Dec 13, 2019 •

edited

ryoqun Dec 13, 2019

ryoqun Dec 13, 2019

sakridge Dec 13, 2019

ryoqun Dec 13, 2019

ryoqun commented Dec 16, 2019

mvines left a comment

ryoqun commented Dec 17, 2019

sakridge left a comment

Strictly sanitize mmapped AppendVec file contents #7464

Strictly sanitize mmapped AppendVec file contents #7464

Conversation

ryoqun commented Dec 13, 2019

Problem

Summary of Changes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryoqun Dec 13, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Dec 13, 2019 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryoqun commented Dec 16, 2019

mvines left a comment

Choose a reason for hiding this comment

ryoqun commented Dec 17, 2019

sakridge left a comment

Choose a reason for hiding this comment

ryoqun Dec 13, 2019 •

edited

codecov bot commented Dec 13, 2019 •

edited