New MPU Interface #1159

dverhaert · 2018-08-22T00:49:42Z

Pull Request Overview

This pull request introduces an updated MPU interface that is no longer Cortex-M4 specific. It follows up on #1126, being part of a plan to make Tock architecture-agnostic (#985).

An overview of the key differences:

The interface now has two Tock-specific functions for allocating and updating the MPU region for the app-owned portion of its RAM.
The interface now only supports specifying user permissions for regions (instead of also specifying supervisor permissions). Memory locations not within any allocated region are required to be inaccessible in user mode, and accessible in supervisor mode. A related change is that we no longer allocate an MPU region for the grant memory in process.rs.
Instead of reallocating all the MPU regions every context switch, regions are now only allocated (computed by the MPU) when they are created or updated. Configuration of the MPU registers is now the only thing that happens during context switches.

The main reasoning behind the first decision is that the app break grows over the lifetime of the process. As such, we have found it infeasible for the client to request an MPU region that is suitable for this purpose through a generic region request. The reasoning behind the second decision is that allowing simultaneous specification of both user and supervisor permissions necessitates exposing a large matrix of options that no single MPU is able to support. When decoupled, we are left with a small, logical set of user permissions that we can reasonably expect all MPUs to be able to support. Furthermore, it is more consistent with our current practice of only enabling the MPU for processes.

The new interface allows removal of code from process.rs that assumed the presence of the Cortex-M MPU, and in particular its constraints and features. For instance, in the Cortex-M, the size of an MPU region must to be a power of two and aligned to the start address, and in the case of region overlap, the permissions of the overlapped segment are those of the region with the highest region number.

For further background and discussion, see the relevant thread on tock-dev. If something remains unclear, feel free to leave a comment below.

Testing Strategy

This pull request was tested by running the mpu_stack_growth app in libtock-c, as well as a few others: one that walks over a process's RAM and flash regions, and several that try to make invalid accesses outside of these regions.

TODO or Help Wanted

Fix _start in userland. @alevy is working on a branch in libtock-c that fixes _start so that processes never access memory past their app break. This pull request is blocking on that branch being merged.
Loading processes. Ideally we would load processes in decreasing order of RAM size on boards that use the Cortex-M MPU. Doing so would eliminate the possibility of external fragmentation between processes on such boards.
Size of the mpu_regions array. This field in struct Process is currently hard-coded to be 6 in size because the Cortex-M supports 8 regions (and two of these are reserved for the flash and RAM regions). This is the last architecture-specific code remaining in process.rs. It currently seems infeasible to make this size chip-specific, seeing as Rust doesn't seem to support this yet.

Documentation Updated

Updated the relevant files in /docs, or no updates are required.

Formatting

Ran make formatall.

Co-authored-by: Danilo Verhaert <daniloverhaert@gmail.com>

phil-levis · 2018-08-22T01:15:36Z

Couple of question:

Can you walk through the intended call sequence for the API? Reading the API and Process::create, it sounds like allocate_region is intended for flash, while allocate_app_ region is intended for RAM. Or is allocate_region the general purpose form, and allocate_app_region is a special purpose form intended for process memory layout?
It would be useful to say how they are intended to be used and their constraints. E.g., would it be valid for a process to invoke allocate_app_region twice?
The notion of a "parent" region implies there is an MPU region for it, but this seems to instead just be a block of unallocated address space?

I personally don't like how now every MPU implementation is specifically bound to the current process memory layout; I think it is much better, in terms of maintenance and debugging, to separate the concerns between laying out memory in the manner that Tock expects and allocating regions. But I realize that (better in the long term) approaches is much more subtle and trick to get right, and so this approach is fine. We can always decompose the two later if experience shows it is needed (the current MPU trait becomes chip-independent, on top of a chip-specific implementation of another, lower-level trait).

cmcavity · 2018-08-22T20:13:33Z

Yes, allocate_region is the more general function (for flash, IPC, etc.), and allocate_app_memory_region is specifically for setting up process memory in RAM and allocating the MPU region for the app-owned part. It is not valid for a process to invoke allocate_app_memory_region more than once: the implementation must return an error in this case. Regarding the name "parent", yes we meant it to be "block of unallocated address space" -- do you think this name is ambiguous?

phil-levis · 2018-08-22T21:22:54Z

I think the name “parent” is amibiguous because it implies a tree. I.e., I allocate a region, then allocate child regions from it. I.e., it implies that it’s a region structure, not unallocated memory. Phil

…

On Aug 22, 2018, at 1:13 PM, Conor McAvity ***@***.***> wrote: Yes, allocate_region is the more general function (for flash, IPC, etc.), and allocate_app_memory_region is specifically for setting up process memory layout and allocating the region for app-owned memory. It is not valid for a process to invoke allocate_app_memory_region more than once: the implementation must return an error in this case. Regarding the name "parent", yes we meant it to be "block of unallocated address space" -- do you think this name is ambiguous? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1159 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEG3ayO0-A-lhGBJBysFPpFrYoEgIpogks5uTbt7gaJpZM4WG0Ko>.

Added documentation and slightly changed fn signatures in interface, added region overlap checks and refactored in cortex-m implementation

arch/cortex-m4/src/mpu.rs

kernel/src/process.rs

bradjc · 2018-08-29T21:08:35Z

kernel/src/process.rs

+        unallocated_memory_start: *const u8,
+        unallocated_memory_size: usize,
+        min_region_size: usize,
+    ) -> Option<(*const u8, usize)>;


I think it makes sense to add something like

struct MpuRegion { start_address: *const u8, length_bytes: usize, }

to be used in the return value to provide a helping hint on what the tuple is. This would also help with the mpu regions array in the process struct.

chips/cc26x2/src/chip.rs

bradjc · 2018-09-28T19:43:48Z

Also I just tried this pr on hail and the app (blink) immediately crashed. I did a little looking, and when I commented out subregion usage blink and c_hello work again.

dverhaert · 2018-09-28T20:09:25Z

Also I just tried this pr on hail and the app (blink) immediately crashed. I did a little looking, and when I commented out subregion usage blink and c_hello work again.

This is actually due to a bug in userland, where processes access memory past their app break in _start (one of the TODOs). Amit has fixed this in the fix_start_order branch of libtock-c, so if you run in that branch of libtock-c you shouldn't encounter it.

bradjc · 2018-09-28T20:17:24Z

This is actually due to a bug in userland, where processes access memory past their app break in _start (one of the TODOs). Amit has fixed this in the fix_start_order branch of libtock-c, so if you run in that branch of libtock-c you shouldn't encounter it.

Gotcha. What would it take to make current/old apps not fault? I ask because we had this idea that apps compiled at 1.0 should still work, and now they won't even though technically they shouldn't have worked before either...

phil-levis · 2018-09-28T20:33:29Z

Hoo boy. Let's not go down the Windows approach of masking fatal bugs in old applications...

dverhaert · 2018-09-28T22:33:45Z

This is actually due to a bug in userland, where processes access memory past their app break in _start (one of the TODOs). Amit has fixed this in the fix_start_order branch of libtock-c, so if you run in that branch of libtock-c you shouldn't encounter it.

Gotcha. What would it take to make current/old apps not fault? I ask because we had this idea that apps compiled at 1.0 should still work, and now they won't even though technically they shouldn't have worked before either...

In the new Cortex-M MPU implementation, we create a process's RAM as one MPU region and use subregions to subdivide this between the PAM and grant, where an enabled subregion represents access for a process. For the unused memory that is between the PAM and grant, we chose to disable subregions. I've just tested changing this to enabled subregions, and this would prevent 1.0 apps from faulting.

bradjc · 2018-10-03T20:47:25Z

Ok now that #1171 is merged, do you want to try updating this PR? a2c3347 should be a good starting point

…face-merge

bradjc · 2018-10-09T21:16:33Z

kernel/src/mem.rs

+                    match process.add_mpu_region(self.ptr() as *const u8, self.len(), self.len()) {
+                        Some(_) => true,
+                        None => false,
+                    }


I think this can just be process.add_mpu_region(self.ptr() as *const u8, self.len(), self.len()).is_ok(), but I'm not completely sure.

kernel/src/platform/mpu.rs

niklasad1 · 2018-10-09T23:38:22Z

kernel/src/platform/mpu.rs

+        config: &mut Self::MpuConfig,
+    ) -> Option<(*const u8, usize)> {
+        let memory_size = {
+            if min_memory_size < initial_app_memory_size + initial_kernel_memory_size {


simplify this code to, let memory_size = core::cmp::max(min_memory_size, initial_app_memory_size + initial_kernel_memory_size)

niklasad1 · 2018-10-09T23:55:53Z

arch/cortex-m4/src/mpu.rs

-            })
-        } else {
-            // Memory base not aligned to memory size
+        let region_num = match config.unused_region_number() {


let region_num = config.unused_region_number()?;

arch/cortex-m4/src/mpu.rs

niklasad1 · 2018-10-10T09:07:20Z

arch/cortex-m4/src/mpu.rs

+                // except for the bits whose index lie within
+                // [min_subregion, max_subregion]
+                //
+                // Note: Rust ranges are minimum inclusive, maximum exclusive, hence


I don't understand this,

Can you explain why you can't use Rust's exclusive range i.e, min_subregion..=max_subregion?

niklasad1 · 2018-10-10T10:11:11Z

arch/cortex-m4/src/mpu.rs

+                // max_subregion + 1.
+                let mask =
+                    (min_subregion..(max_subregion + 1)).fold(!0, |res, i| res & !(1 << i)) & 0xff;
+


I find this code very hard to read.
Why are you bitwise and_ing the fold result with 0xff? (seems needless to me)

I think you can me this a little be more readable by something like:

let mask = (start..end+1).fold(u32::max_value(), |res, i| { // clear bit-by-bit (1 ^ 1 == 0) res ^ (1 << i) });

niklasad1 · 2018-10-10T10:14:17Z

arch/cortex-m4/src/mpu.rs


-        regs.region_base_address.set(region.base_address());
+        let subregion_mask = (0..num_subregions_used).fold(!0, |res, i| res & !(1 << i)) & 0xff;


Really, create a function that calculates subregion mask. Third occurrence of this!

kernel/src/mem.rs

niklasad1

Looks generally good, some minor things to address 👍

bradjc

Ok take a look at the commit I added: 1e60298

It removes debug and panic statements, and relies on the return value. We probably aren't being expressive enough with our return values, but that is a problem for another day.

With those changes I'm happy with this PR.

thanks, rust

bradjc · 2018-10-17T19:02:58Z

bors r+

@alevy

1159: New MPU Interface r=bradjc a=dverhaert ### Pull Request Overview This pull request introduces an updated MPU interface that is no longer Cortex-M4 specific. It follows up on #1126, being part of a plan to make Tock architecture-agnostic (#985). An overview of the key differences: 1. The interface now has two Tock-specific functions for allocating and updating the MPU region for the app-owned portion of its RAM. 2. The interface now only supports specifying user permissions for regions (instead of also specifying supervisor permissions). Memory locations not within any allocated region are required to be inaccessible in user mode, and accessible in supervisor mode. A related change is that we no longer allocate an MPU region for the grant memory in process.rs. 3. Instead of reallocating all the MPU regions every context switch, regions are now only allocated (computed by the MPU) when they are created or updated. Configuration of the MPU registers is now the only thing that happens during context switches. The main reasoning behind the first decision is that the app break grows over the lifetime of the process. As such, we have found it infeasible for the client to request an MPU region that is suitable for this purpose through a generic region request. The reasoning behind the second decision is that allowing simultaneous specification of both user and supervisor permissions necessitates exposing a large matrix of options that no single MPU is able to support. When decoupled, we are left with a small, logical set of user permissions that we can reasonably expect all MPUs to be able to support. Furthermore, it is more consistent with our current practice of only enabling the MPU for processes. The new interface allows removal of code from process.rs that assumed the presence of the Cortex-M MPU, and in particular its constraints and features. For instance, in the Cortex-M, the size of an MPU region must to be a power of two and aligned to the start address, and in the case of region overlap, the permissions of the overlapped segment are those of the region with the highest region number. For further background and discussion, see the relevant [thread on tock-dev](https://groups.google.com/forum/#!topic/tock-dev/Fduxk_0xvUE). If something remains unclear, feel free to leave a comment below. ### Testing Strategy This pull request was tested by running the `mpu_stack_growth` app in libtock-c, as well as a few others: one that walks over a process's RAM and flash regions, and several that try to make invalid accesses outside of these regions. ### TODO or Help Wanted - [ ] **Fix `_start` in userland**. @alevy is working on a [branch](https://github.com/dverhaert/libtock-c/tree/fix_start_order) in libtock-c that fixes `_start` so that processes never access memory past their app break. This pull request is blocking on that branch being merged. - [ ] **Loading processes**. Ideally we would load processes in decreasing order of RAM size on boards that use the Cortex-M MPU. Doing so would eliminate the possibility of external fragmentation between processes on such boards. - [ ] **Size of the `mpu_regions` array**. This field in `struct Process` is currently hard-coded to be 6 in size because the Cortex-M supports 8 regions (and two of these are reserved for the flash and RAM regions). This is the last architecture-specific code remaining in process.rs. It currently seems infeasible to make this size chip-specific, seeing as [Rust doesn't seem to support this yet](rust-lang/rust#43408). ### Documentation Updated - [x] Updated the relevant files in `/docs`, or no updates are required. ### Formatting - [x] Ran `make formatall`. Co-authored-by: Conor McAvity <cmcavity@protonmail.com> Co-authored-by: Brad Campbell <bradjc5@gmail.com> Co-authored-by: Danilo Verhaert <daniloverhaert@gmail.com>

bors · 2018-10-17T19:11:48Z

Build succeeded

continuous-integration/travis-ci/push

New mpu interface

f79253a

Co-authored-by: Danilo Verhaert <daniloverhaert@gmail.com>

cmcavity added 2 commits August 22, 2018 18:16

Update mpu interface and cortex-m implementation

e74a7aa

Added documentation and slightly changed fn signatures in interface, added region overlap checks and refactored in cortex-m implementation

Refactor and fix cortex-m mpu implementation

b95dec1

bradjc reviewed Aug 29, 2018

View reviewed changes

cmcavity added 5 commits August 29, 2018 15:54

Update mpu documentation, add new struct to interface

0bd31dc

Remove some panics from process.rs

c4a3465

Clean up add_mpu_region

b646633

Merge branch 'master' into mpu-interface-merge

084b4e5

Fix overlap algorithm in cortex-m mpu

9f66bd2

cmcavity force-pushed the mpu-interface-merge branch from 6a6a235 to 9f66bd2 Compare September 18, 2018 22:46

phil-levis previously approved these changes Sep 26, 2018

View reviewed changes

Remove mut from Chip trait

1c34d78

bradjc requested changes Sep 28, 2018

View reviewed changes

chips/cc26x2/src/chip.rs Outdated Show resolved Hide resolved

bradjc mentioned this pull request Sep 28, 2018

kernel: Chip does not need to be mut #1171

Merged

2 tasks

bradjc added the P-Significant This is a substancial change that requires review from all core developers. label Sep 28, 2018

bradjc mentioned this pull request Sep 28, 2018

Tracking: Make Tock Architecture Agnostic #985

Closed

9 tasks

Handle edge case in cortex-m mpu

5131abb

Change initial PAM size to 3 KB

ee4ee1c

cmcavity dismissed phil-levis’s stale review via ee4ee1c October 4, 2018 18:46

cmcavity added 2 commits October 5, 2018 22:09

Merge remote-tracking branch 'upstream/new-mpu-bradjc' into mpu-inter…

b773f2f

…face-merge

Merge branch 'master' into mpu-interface-merge

6cf7f28

Make formatall

7350a07

cmcavity force-pushed the mpu-interface-merge branch from d137586 to 7350a07 Compare October 6, 2018 05:20

bradjc requested changes Oct 9, 2018

View reviewed changes

Refactor

8197349

niklasad1 reviewed Oct 9, 2018

View reviewed changes

kernel/src/platform/mpu.rs Show resolved Hide resolved

niklasad1 reviewed Oct 9, 2018

View reviewed changes

Use core::cmp::max

df317e4

niklasad1 reviewed Oct 9, 2018

View reviewed changes

niklasad1 reviewed Oct 10, 2018

View reviewed changes

arch/cortex-m4/src/mpu.rs Show resolved Hide resolved

niklasad1 reviewed Oct 10, 2018

View reviewed changes

kernel/src/mem.rs Show resolved Hide resolved

niklasad1 suggested changes Oct 10, 2018

View reviewed changes

cmcavity and others added 2 commits October 12, 2018 14:00

Simplify code

fcb6f58

new mpu: updates

1e60298

bradjc previously approved these changes Oct 16, 2018

View reviewed changes

niklasad1 previously approved these changes Oct 17, 2018

View reviewed changes

kernel: process: map_or -> and_then

f170fb4

thanks, rust

bradjc dismissed stale reviews from niklasad1 and themself via f170fb4 October 17, 2018 13:46

bradjc approved these changes Oct 17, 2018

View reviewed changes

niklasad1 approved these changes Oct 17, 2018

View reviewed changes

bors bot merged commit f170fb4 into tock:master Oct 17, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New MPU Interface #1159

New MPU Interface #1159

dverhaert commented Aug 22, 2018 •

edited

Loading

phil-levis commented Aug 22, 2018

cmcavity commented Aug 22, 2018 •

edited

Loading

phil-levis commented Aug 22, 2018 via email

bradjc Aug 29, 2018

bradjc commented Sep 28, 2018

dverhaert commented Sep 28, 2018

bradjc commented Sep 28, 2018

phil-levis commented Sep 28, 2018

dverhaert commented Sep 28, 2018

bradjc commented Oct 3, 2018

bradjc Oct 9, 2018

niklasad1 Oct 9, 2018

niklasad1 Oct 9, 2018 •

edited

Loading

niklasad1 Oct 10, 2018

niklasad1 Oct 10, 2018

niklasad1 Oct 10, 2018

niklasad1 left a comment

bradjc left a comment

bradjc commented Oct 17, 2018

bors bot commented Oct 17, 2018


		regs.region_base_address.set(region.base_address());
		let subregion_mask = (0..num_subregions_used).fold(!0, \|res, i\| res & !(1 << i)) & 0xff;

New MPU Interface #1159

New MPU Interface #1159

Conversation

dverhaert commented Aug 22, 2018 • edited Loading

Pull Request Overview

Testing Strategy

TODO or Help Wanted

Documentation Updated

Formatting

phil-levis commented Aug 22, 2018

cmcavity commented Aug 22, 2018 • edited Loading

phil-levis commented Aug 22, 2018 via email

bradjc Aug 29, 2018

Choose a reason for hiding this comment

bradjc commented Sep 28, 2018

dverhaert commented Sep 28, 2018

bradjc commented Sep 28, 2018

phil-levis commented Sep 28, 2018

dverhaert commented Sep 28, 2018

bradjc commented Oct 3, 2018

bradjc Oct 9, 2018

Choose a reason for hiding this comment

niklasad1 Oct 9, 2018

Choose a reason for hiding this comment

niklasad1 Oct 9, 2018 • edited Loading

Choose a reason for hiding this comment

niklasad1 Oct 10, 2018

Choose a reason for hiding this comment

niklasad1 Oct 10, 2018

Choose a reason for hiding this comment

niklasad1 Oct 10, 2018

Choose a reason for hiding this comment

niklasad1 left a comment

Choose a reason for hiding this comment

bradjc left a comment

Choose a reason for hiding this comment

bradjc commented Oct 17, 2018

bors bot commented Oct 17, 2018

Build succeeded

dverhaert commented Aug 22, 2018 •

edited

Loading

cmcavity commented Aug 22, 2018 •

edited

Loading

niklasad1 Oct 9, 2018 •

edited

Loading