Software
The ArmBoy project is one of developing a hand held game system, yes, but also one of developing a robust and adaptable operating system for Cortex-M3/4 devices. With the possibility of porting to any ARMv7 architecture with very minor changes. The System is heavily abstracted and separated to maintain maximum flexibility in purpose and in platform. The system is composed of 5 major parts:
-
Kernel Core. The kernel core is the main part of the operating system. Its job is to manage system resources, CPU, SRAM and FLASH; handle dynamic modules, both user and kernel; and expose core peripheral functionality to user land applications. Perhaps more interesting is not what it dose but, what it does not do. The kernel core does not directly manage hardware (platform agnostic) and it does not include any application specific functionality. Ex: though the ArmBoy is a hand held gaming system the kernel core has no notion of this. Thus, the kernel core can easily be re-purposed to any other application without a firmware update.
-
Hardware Abstraction layer. The hardware abstraction layer is a library used by the kernel core so that it does not need to know how the hardware on which it runs works. This allows the kernel core to be easily recompiled for different hardware targets.
-
Kernel modules, such as the LCD Driver, File System Driver and User Input Driver. These modules and more, handle peripheral devices and give the system purpose (like being a hand held gaming system). For example the LCD driver works much like a modern graphics driver. It handles the screen hardware and exposes a rendering interface via system calls, that user land applications can then call indirectly through an API. This is very similar to, graphics driver + OpenGL / DX / Vulkan. Furthermore, all kernel modules are dynamic. That is they can be modified, loaded and unloaded at run time. For example, if an application wishes to read data from an IDE hard drive (default driver only supports SD cards), it need only request the kernel core to load an IDE hard drive kernel module (assuming one is installed ofc). Then seamlessly the user land application can start reading data off the drive and when done it can unload the module to save SRAM.
-
System API. The system api exposes kernel core, kernel module and utility functionality to user-land applications. Think Windows API or Unix API. This API is simply a convenience layer so that application programmers need not make ugly system calls all over there code. This API hides system calls for convenience and provides some bonus utility functions along the way, such as text rendering.
-
User-land applications. The final piece of the puzzle, user-land applications give the system direction. User-land applications utilize all the aforementioned infrastructure to produce games or anything you can imagine. As with kernel modules, user-land applications are dynamic and can be loaded both from the SD card and through the serial interface for debugging.
All these parts come together to form the ArmBoy Gaming System!
The below image shows the basic orginization of the ArmBoy system. User applications are un-privileged thus,
unable to directly communicate with hardware rather all such operations must move through system calls. Kernel
modules, kernel core and hardware abstraction layer execute in the privileged context. Finally the memory map
of the system is depicted.
The ArmBoy software stack is built on principles of reusable design, with layers by in large constrained to "talking" to the layer immediately bellow it. I do not state this as an absolute because the kernel modules are "part" of the kernel (linked against the kernel). That is to say they can call kernel functions and user kernel variables as easy as if they where part of the kernel. As such they may make direct access to the hardware abstraction layer. Another key division between layers is that between User Applications and Kernel modules. This relationship requires a different approach as User Applications are unprivileged while kernel modules are privileged, as such user land applications cannot directly call functions in kernel modules (if this where so then user applications would have to be privileged). Thus to allow user-land applications to invoke kernel module functionality the hardware system call functionality of the Cortex-M ARM chips is used. This gives User Applications an almost complete independence from operating system. As mentioned before a key tenant of this systems design is adaptability, this software stack reinforces that idea.
As stated earlier the memory map is dependent on system architecture. Thus the detailed information following will be hardware specific.
Sam3x8e Cortex-M3 Memory Layout. The sam3x8e chip as with many micro controllers sports both SRAM and FLASH memory. It has 96KB of SRAM and 512KB of Flash. The SRAM is divided in to three chunks. At the lowest address we find 6912 Bytes of kernel reserved memory. Following that we find 86319 Bytes of free dynamic memory for use by applications or kernel modules. Finally at the top of SRAM we have 5072 Bytes for the kernel stack, which remember is shared by kernel modules. Now for the Flash. On the sam3x8e flash memory is divided in to two "planes" of 256 KB each. For the most use cases the existence of two planes has no effect, after all they appear as one continuous block of memory. There is however one key restriction to be aware of, if you are writing to a flash plane you cannot read from it at the same time. This presents a problem, where by the kernel code that is writing to the flash is its self running on the flash. Thus when writing the flash the CPU cannot fetch the next instruction of the program because reading the flash is not allowed while writing it. To get around this problem the first data plane (low 256KB) is reserved entirely for the kernel core (even though it uses only 25-30% of it), while the second plane (high 256KB) is used exclusively for kernel module storage.
The ArmBoy uses a context switching mechanism detailed in countless pieces of computer science literature. That being involuntary interrupt based CPU time sharing. Due to the well documented nature I will not focus on the idea of context switching rather, how it is done on a Cortex-M ARM system (my only other experience being AVR based systems). When context switching on a Cortex-M system one must keep a few things in mind, modes of execution, nested interrupts and stack layout.
On Cortex-M based systems there are two execution modes and two privilege levels. The modes of execution are Thread and Handler, while the privilege modes are privileged and unprivileged. The processor enters Thread execution mode on reset (power up) or on exception return, assuming no other pending exception. The other mode, Handler mode is active when entering an interrupt handler. Thread mode can be either privileged or unprivileged while Handler mode must be privileged. Finally the processor has two stack pointers, the master stack pointer (MSP) and the process stack pointer (PSP). These stack pointers are closely tied to execution mode and privilege level. Where either the MSP or PSP can be used in Thread mode. MSP if privileged code and PSP if unprivileged code. On the other hand, Handler mode must use the MSP.
Possible states
- Privileged Thread mode using MSP
- Unprivileged Thread mode using PSP
- Handler mode using MSP
They ArmBoy system jumps between these three modes when context switching. The primary way in which one switches states is by returning to a special address on exception return.
| instruction | result |
|---|---|
bx 0xFFFFFFFD |
return from interrupt in to unprivileged thread mode using PSP |
bx 0xFFFFFFF9 |
return from interrupt in to privileged thread mode using MSP |
bx 0xFFFFFFF1 |
return from interrupt int to privileged handler mode using MSP |
the last of these instructions must only be issued if there is another pending interrupt
The NVIC is a peripheral device common to all Cortex-M devices that allows the programmer to configure and handle system interrupts. Operations include, disabling and enabling specific interrupts; setting interrupt priority and checking current interrupt state. Not all interrupts however are created equal. While most interrupts can have there priority changed some "system" interrupts cannot. Unfortunately one such interrupt is the, SysTick_IRQ an interrupt of grate importance to context switching.
Another key feature of the NVIC which may be strange to AVR programmers is the notion of, nested interrupts and priority. In AVR interrupt handling the one key rule I remember is disable interrupts during handling of an interrupt. This is not the case on the ARM system. On an Cortex-M system, interrupts can be preempted by interrupts of a higher priority (lower number) and interrupts can queue them self's for execution if the current interrupt is of higher priority then themselves. Furthermore, it seems that if an interrupt pends for to great a time it will be lost. This interrupt system introduces considerable uncertainty in to the path of execution. This uncertainty must be managed when context switching.
ARM Cortex-M devices (with FPU disabled where applicable) push this frame to the stack on interrupt.
/****** Stack *********//*
xPSR
PC
LR
R12
R3
R2
R1
R0
*//********************/
Following the above stack items I add these just to be safe.
/** Extra Stack Values ***//*
R11
R10
R9
R8
R7
R6
R5
R4
*//********************/
The above diagram depicts the contexts switching process. In it there are three different colors of arrows,
each of which depicts a different execution path. One path for entering application code and two for returning to kernel code. The return to kernel code requires to different paths depending on the state of execution the processor is in when the involuntary context switch is triggered. This is needed because when switching back to the kernel the applications stack must be preserved and the kernel stack must be restored but, if the processor was executing an interrupt handler when a context switch is triggered the kernels stack will be filled with stack frames form the interrupt handler (because interrupts use the kernel stack). Thus the context switching code will save a some what corrupted set of application registers and leave the kernel in some sort of state where by its stack is completely corrupted with interrupt handler stack frames. To solve this problem the PendSV IRQ is used. The PendSV interrupt is set to minimum priority (255) so that when control is inside said interrupt one can be sure that. (1) control has directly entered this interrupt from application code, insuring register integrity. (2) the kerels stack is completely free of interrupt handler stack frames.
Context Switching implementation (MACRO definitions available in cSwitch.h)
void SYS_TICK_IRQ(void){
toUser = false;
triggerPendSV();
}
__attribute__ ((naked)) void PENDSV_IRQ(void) {
if(toUser){ // <-- danger! if compiler decides to use registers other than r0-r3 hear we are fucked.
SAVE_STATE();
RESTORE_STATE_PSP();
IRQ_RET_PSP();
}
else{
SAVE_STATE_PSP();
READ_PSP((uint32_t)currentPd->stackPtr);
RESTORE_STATE();
IRQ_RET_MSP();
}
}
__attribute__ ((naked)) void __launchProcess(uint32_t * sp){
asm("push {lr}\n");
MSP_TO_PSP((uint32_t)sp); // "MSP_TO_PSP(sp)" is misleading. should really read SET_PSP(sp).
triggerPendSV();
asm("pop {pc}\n");
}
void launchProcess(uint32_t * sp){
toUser = true;
startSysTick();
__launchProcess(sp);
//return here in CONTEXT_SWITCH_INTERVAL ms
stopSysTick();
}
}
-
Green: these arrows depict the path taken when context switching from the kernel to a user mode application. To perform this switch, the kernel first initializes the application stack to appear as if the application was interrupted (this setup it only required for first switch, subsequent switches do not require this step). Then the kernel triggers a PenSV interrupt. The handler of this interrupt performs a returns to unprivileged thread code through the uses of the
bx 0xFFFFFFFDinstruction discussed in, Execution modes on Cortex-M systems. At this point the switch to application code is complete. -
Blue: These arrows show the path taken on return to kernel code. This path of execution is triggered by a hardware timer that the kernel configured before switching in to user code. In the current kernel such a timer is set to trigger every 150ms (long quanta I know). When the timer triggers a SysTick IRQ interrupt is raised. At this point the execution path diverges. If SysTick IRQ is triggered while the processor is executing application code then all is well and the switching mechanism can continue as normal. However, if the processor is currently handling another interrupt such as a SysCall IRQ execution moves to the realm of the red arrows (see below). Now let us assume that when SysTick IRQ is triggerd the processor is in application code. In such a case the user code is interrupted and the handler code for SysTick IRQ triggers a PendSV IRQ and returns. Technically SysTick IRQ returns to application code but because a PendSV interrupt is pending, user code is again immediately interrupted and control is shifted to the PendSV handler. In the PendSV a
bx 0xFFFFFFF9instruction is executed to return to kernel code. -
Red: These arrows show the special case path invoked when the SysTick IRQ is triggered during the handling of another exception (most likely a system call). First SysTick IRQ is triggered and executes as normal. however instead of continuing the switch to the kernel after SysTick IRQ finishes, execution returns to the IRQ that was being serviced before SysTick IRQ was raised. Said interrupt is given as much time as necessary to finish. When it and all other pending interrupts finish control is shifted to the PensSV handler and execution continues as normal.
Dynamic loading is perhaps the most complex part of the system. There are two types of dynamic objects, kernel modules and user applications. User applications are intended to load and operate as independently from the OS as possible while kernel modules integrate as closely as possible with the OS (they must be compiled for the exact kernel core binary running on the system).
How does one execute a dynamic binary? Its not as easy as it at first appears, especially on Cortex-M style devices as they have no virtual address support (no MMU at all really). Due to this we must understand the format of a binary file and construct our own dynamically loadable format.
A binary file is much more than a list of machine code. It contains, machine code, program data, meta data, debugging data and all the information necessary to dynamically load the binary. You can explore this for your self by using objdump on an executable. For example here is a dump of the tetris.elf executable.
arm-none-eabi-objdump -h tetris.elf
tetris.elf: file format elf32-littlearm
Sections:
Idx Name Size VMA LMA File off Algn
0 .text 00001af0 00000000 00000000 00000054 2**2
CONTENTS, ALLOC, LOAD, CODE
1 .got.plt 0000000c 00001af0 00001af0 00001b44 2**2
CONTENTS, ALLOC, LOAD, DATA
2 .got 00000004 00001afc 00001afc 00001b50 2**2
CONTENTS, ALLOC, LOAD, DATA
3 .data 00000000 00001b00 00001b00 00001b54 2**0
CONTENTS, ALLOC, LOAD, DATA
4 .bss 00000064 00001b00 00001b00 00001b54 2**2
ALLOC
5 .comment 0000007e 00000000 00000000 00001b54 2**0
CONTENTS, READONLY
6 .ARM.attributes 00000031 00000000 00000000 00001bd2 2**0
CONTENTS, READONLY
7 .debug_frame 00000124 00000000 00000000 00001c04 2**2
CONTENTS, READONLY, DEBUGGING
As you can see a binary is really composed of multiple sections. The important sections are:
- 1 .text: This section contains the actual machine code of the application. In it you will see the code you wrote along with the code of any statically linked libraries.
- 2 .data: This section contains any static data in your program. For example if you had,
printf("Hello World")in your program, then the string "Hello World" would appear in the .data section. A quick side note on security, because people can sniff around in the .data section of binaries, never ever store sensitive information as static program data. Ex,char * mySecretKey = "A352-BD52-GKB3-GEY9";will be visible in the .data section of your binary. - 3 .bss: This section is a sort of place holder for zeroed out memory. This memory will be used by global objects. For example,
char stuff[100];(outside function) will produce a section in the .bss of 100 Bytes of zero. Note the bss is a virtual section, so it does not really contain zeros rather it just contains the size of a continuous region of memory to be zeroed and allocated by the OS. - 4 .got: This is a key section for dynamic loading / relocation of binaries. this section is essentially a jump table. That is, it contains a list of addresses, where each address is the location of some piece of data. If we look in the .got section of the tetris file using,
arm-none-eabi-objdump -d -j .got tetris.elfwe get:
00001afc <__got_start__>:
1afc: 00001b00 .word 0x00001b00
The key thing to focus on is the number 00001b00. This is the location of some piece of data. For this particular piece of data we can see that it is located at the start of the .data section (VMA address of .data is 00001b00).
You can then issue commands such as arm-none-eabi-objdump -d -j .text tetris.elf to get the disassembly for
the .text section (the section that contains the machine code).
00000910 <main>:
910: e92d 43f0 stmdb sp!, {r4, r5, r6, r7, r8, r9, lr}
914: 4b49 ldr r3, [pc, #292] ; (a3c <main+0x12c>)
916: f5ad 5d04 sub.w sp, sp, #8448 ; 0x2100
91a: b083 sub sp, #12
91c: 4848 ldr r0, [pc, #288] ; (a40 <main+0x130>)
91e: f10d 0860 add.w r8, sp, #96 ; 0x60
922: ae05 add r6, sp, #20
924: 9305 str r3, [sp, #20]
926: ab96 add r3, sp, #600 ; 0x258
928: aa07 add r2, sp, #28
92a: 6073 str r3, [r6, #4]
92c: 4641 mov r1, r8
92e: f60d 0348 addw r3, sp, #2120 ; 0x848
932: 4478 add r0, pc
934: 920c str r2, [sp, #48] ; 0x30
936: 9308 str r3, [sp, #32]
938: f000 fc9a bl 1270 <openFile>
93c: b130 cbz r0, 94c <main+0x3c>
93e: 4841 ldr r0, [pc, #260] ; (a44 <main+0x134>)
940: af3e add r7, sp, #248 ; 0xf8
942: 4639 mov r1, r7
944: 4478 add r0, pc
946: f000 fc93 bl 1270 <openFile>
94a: b988 cbnz r0, 970 <main+0x60>
94c: 2164 movs r1, #100 ; 0x64
94e: 2532 movs r5, #50 ; 0x32
950: f04f 1428 mov.w r4, #2621480 ; 0x280028
...
...
...
The information of standard binary files will be cut down and used in a custom binary format for the ArmBoy system.
All modern computers have a hardware component called an, Memory Management Unit (MMU). Such a unit is absent on Cortex-M systems and most embedded systems for that matter. The functionality of the MMU that would help with dynamic loading is its ability to translate memory accesses on the fly. For example, when the CPU requests memory address 0x0000001 the MMU can translate the request, without the CPUs knowledge, to the physical address 0x14154f0. This creates the idea of Virtual Memory. Virtual Memory is the idea that the CPU "thinks" its operating on physical memory addresses but, it is really using translated memory address. Let us now think back to the binary format. Throughout the assembly we see memory specific operations, like loading registers with information from the .data section (address 00001b00). This is fine if the binary is run at memory address 0x0 but, if it where to be offset from zero by any amount (as it would in a OS environment) all the accesses would be come incorrect (because all sections moved but the access remain the same). To correct this issue, modern computers use the MMU to create a virtual address space. Such as space causes the program to "think" it is executing in a continues region of memory starting at 0x0 even though it is in all likely hood made up of fragmented memory regions throughout the physical address space. As the Cortex-M chips do not have an MMU a different and more complex approach must be taken.
The ArmBoy executable format contains the, .text, .data, .bss and .got sections of a normal ELF executable plus, .module_name, .binary_layout and .jump_vector sections. The first section, .module_name is simply a string with the name of the module (60 chars max). It is only for kernel modules and is not present in user land binaries. The second of these sections describes the layout of the binary file. It is defined as:
.section .binary_layout
.globl __binary_layout
__binary_layout:
.long __text_start__ // <--- offset of .text section from start of file
.long __text_end__
.long __got_start__ // <--- offset of .got section from start of file
.long __got_end__
.long __data_start__ // <--- offset of .data section from start of file
.long __data_end__
.long __bss_start__ // <--- I think you can guess
.long __bss_end__
.long 0x400 /* !!! HEAP SIZE !!! <--- Edit as you need */
.long 0x400 /* !!! STACK SIZE !!! <--- Edit as you need */
.size __binary_layout, . - __binary_layout
Immediately following that section is the .jump_vector list / section. The .jump_vector is a fixed size list (40) of jump vectors (function pointers) that binaries use to expose there functionality (yes yes I know fixed size bad... I'll fix it one day). Kernel modules uses these to expose there functionality while user applications just use the first vector to point to there main function. The jump table is defined as:
.section .jump_vector
.align 2
.globl __jump_vector
__jump_vector:
/* jump vectors. replace Default_Handler with you handler as required*/
.long Default_Handler // 0
.long Default_Handler
.long Default_Handler
.long Default_Handler
.long Default_Handler
.long Default_Handler
.long Default_Handler
.long Default_Handler
.long Default_Handler
.long Default_Handler // 10
.long Default_Handler // 11
.long Default_Handler
...
...
...
.long Default_Handler
.long Default_Handler
.long Default_Handler // 40
.size __jump_vector, . - __jump_vector
To expose a function simply replace Default_Handler with the name of the function you whish to expose.
to be clear ArmBoy binaries are arranged as
| kernel module | user application |
|---|---|
| .module_name | NA |
| .binary_layout | .binary_layout |
| .jump_vector | .jump_vector |
| .text | .text |
| .got | .got |
| .data | .data |
| .bss | .bss |
There are two different processes for loading executables, one for kernel modules and another for user applications.
Kernel modules reside in the flash memory of the micro controller. A kernel module can be uploaded to said flash by using the ArmBoy Flasher utility, ./abFlasher <binary file> <serial port>. The steps for loading a module from flash are:
- Read the .binary_layout section of the binary. This tells the OS where the various parts of the binary are located and there size.
- Based on step 1. allocate memory in the dynamic memory area big enough to hold the .data, .bss, .got and heap of the kernel module.
- copy the .got, .data and .bss sections, in that order, in to the newly allocated memory region. Additionally zero out the .bss section if present.
- update the .got table to reflect the new location of binary data. mainly that, .data and .bss reside in SRAM while .text is located in the FLASH.
for(uint32_t i = 0; i < (fh->got_end - fh->got_start)/4; i ++){
if(*(wGot + i) < FUNCTION_BASE_ADDR){
if(*(wGot + i) < fh->text_end){
//address is pointing in to .text section update to point at new FLASH address
*(wGot + i) = *(wGot + i) + (uint32_t)textAddr;
}
else{
//address is pointing to .data/.bss section update to point at new SRAM address
*(wGot + i) = (*(wGot + i) - fh->got_start) + (uint32_t)gotAddress;
}
}
else {
// FUNCTIONS at address > FUNCTION_BASE_ADDR are functions inside the kernel core. as such they
// need no relocation
}
}
- when calling any function in this kernel module set r9 (known as the static base) to the address of the .got table in SRAM.
The memory layout of a fully loaded kernel module. Arrows indicate pointers. Ex jump_vector points in to the .text section and the .text section points to the .got
Loading user applications is very similar to loading kernel modules, though there are some differences. The first and most obvious difference is that user applications are not stored in flash memory. Rather, user applications are stored on disk (SD) or uploaded through the programming/debug port. Regardless of storage media the steps are the same:
- Allocate dynamic memory large enough to hold the entire binary. Copy the binary from the storage medium to the allocated memory.
- Read the .binary_layout section of the binary. This tells the OS where the various parts of the binary are located and there size.
- Unlike kernel modules, no individual sections need to be relocated.
- update .got table to reflect the new location of the binary (SRAM). using same code as show in step 4. of "loading kernel modules".
- no need to set r9 (static base) because user applications are compiled with relative addressing instead of r9 addressing.
The memory layout of a fully loaded user application. Arrows indicate pointers. Ex jump_vector points in to the .text section and the .text section points to the .got