X86 optimize context switch

[andrewboie]

The PR does the following:

Major changes:

• x86 now uses per-thread page tables, with a simple CR3 update on context switch or when trampolining to/from user mode
• memory domain architecture interface no longer makes decisions on whether the arch code needs to be notified, necessary when domain configuration is managed per-thread instead of common data or registers

Minor changes:

• thread->stack_obj is now set when z_new_thread() is called
• stack overflow guard pages on x86 are now read-only instead of non-present, making debugging easier
• added debug functions for dumping page tables at runtime
• added inline functions to get/set active page tables on x86
• fatal exceptions on x86 now report CR3 register (which points to the active page tables)
• memory domain documentation clarification
• some test case adjustments

Please read individual commit messages for more details.

This patch leaves the build-time page table generation alone, although this new code doesn't care how the master page tables were created and when we switch to runtime generated page tables at boot, this code can remain the same.

Addresses #15135 on x86
Fixes: #13003
Contains groundwork necessary for: #13441 #13074 #15223

[zephyrbot]

All checks are passing now.

Review history of this comment for details about previous failed status.
Note that some checks might have not completed yet.

[dbkinder]

No doc changes to review

[andrewboie]

@ioannisg @vonhust @ruuddw any objections to the two patches which touch the mem domain API and documentation changes? they are fairly trivial, mostly just moved the current thread check into the arch implementations

The relevant patches in this PR are

• "userspace: adjust arch memory domain interface"
• "userspace: clarify k_mem_partition_add()"

[ioannisg]

@ioannisg @vonhust @ruuddw any objections to the two patches which touch the mem domain API and documentation changes? they are fairly trivial, mostly just moved the current thread check into the arch implementations

The relevant patches in this PR are

* "userspace: adjust arch memory domain interface"

* "userspace: clarify k_mem_partition_add()"

@andrewboie sorry for being late on this.
I had a look in the mem_domain related changes, and they reflect your intention of moving the decisions to arch-specific code. So I think they are fine.

So, AFAICS, it is the arch-specific code which decides to re-program MPU (ARC, ARM), if the functions are called for the current thread. Does this mean that this principle is no longer needed, generally, e.g. for other architectures, such as x86?

In that case, I believe we lack some documentation in arm_core_mpu.c, arc_core_mpu.c, or in the corresponding API header, that clarifies why these checks are required. IMHO, maybe some notes on the function docs would suffice...

[andrewboie]

So, AFAICS, it is the arch-specific code which decides to re-program MPU (ARC, ARM), if the functions are called for the current thread. Does this mean that this principle is no longer needed, generally, e.g. for other architectures, such as x86?

I'm not quite sure what you mean, it depends on the arch code. On x86 it's any thread because there is arch-specific state that has to be managed for any thread that was in the domain being modified (in this case the per-thread pagetables)

So if I destroy a memory domain, I need to update all the per-thread page tables for the threads that were running in it, because they are no longer in a memory domain.

The key thing about per-thread state management is it removes having to compute anything at context switch, at the expense of more overhead when the memory domain is configured.

In that case, I believe we lack some documentation in arm_core_mpu.c, arc_core_mpu.c, or in the corresponding API header, that clarifies why these checks are required. IMHO, maybe some notes on the function docs would suffice...

The checks are required because the arch code was assuming that any changes were for the active thread and the MPU hardware gets immediately reprogrammed.

For 2.1 I'm prioritizing #13074 for at least ARMv7 MPU and these will get rewritten

[ruuddw]

Moving the check for 'MPU HW' reprogramming or not to the arch specific code is ok for me if it helps other arches. Not sure yet if it would help or be required for #13074 - managing such a 'TLB' would preferably be generic and not arch specific code. I can imagine a 3 layer approach instead of current 2 layers as well: logical/api mem domain management -> logical to physical mapping (and lazy programming/TLB management) -> physical MPU programming (as in current arch specific code).

[andrewboie]

Moving the check for 'MPU HW' reprogramming or not to the arch specific code is ok for me if it helps other arches. Not sure yet if it would help or be required for #13074 - managing such a 'TLB' would preferably be generic and not arch specific code.

That doesn't matter, it manages per-thread data so it will require this

[ioannisg]

I'm not quite sure what you mean, it depends on the arch code. On x86 it's any thread because there is arch-specific state that has to be managed for any thread that was in the domain being modified (in this case the per-thread pagetables)

So if I destroy a memory domain, I need to update all the per-thread page tables for the threads that were running in it, because they are no longer in a memory domain.

OK, so I understand that ARC and ARM only call the on-line reprogramming if it is the current thread, while in x86 it is any thread. Thanks for clarifying.

So, as I said above (maybe not clear enough) is that I would like us to document the reasoning why we only need to do on-line re-programming if it is the current thread in ARM Core MPU code. Earlier, these checks for current-thread were not present in arch-code, because they were in kernel/mem_domain, but since they are now in ARM core mpu, i think it would be nice to comment why we do them, to help explicitly the ARM developer understand the code better.