From 580d0d9c76c94e1914cee2523a43c6b6fef95bf6 Mon Sep 17 00:00:00 2001 From: Ivan Gualandri Date: Mon, 16 Sep 2024 09:56:42 +0100 Subject: [PATCH 01/13] overview: update from stivale to limine protocol --- 01_Build_Process/01_Overview.md | 27 +++++++++++++++------------ 06_Userspace/05_Example_ABI.md | 6 +----- 2 files changed, 16 insertions(+), 17 deletions(-) diff --git a/01_Build_Process/01_Overview.md b/01_Build_Process/01_Overview.md index eaaa3d6..e654508 100644 --- a/01_Build_Process/01_Overview.md +++ b/01_Build_Process/01_Overview.md @@ -6,7 +6,7 @@ In this part we are going to explore all the tools and steps that are needed in In this chapter we will have a global overview of the build process, touching briefly what are the steps involved, and the tools that are going to be used. -Then in the [Boots Protocols and Bootloaders](02_Boot_Protocols.md) chapter we will explore in detail how to boot a kernel, and describe two options that can be used for the boot process: Multiboot and Stivale. +Then in the [Boots Protocols and Bootloaders](02_Boot_Protocols.md) chapter we will explore in detail how to boot a kernel, and describe two options that can be used for the boot process: Multiboot and Limine. The [Makefiles](04_Gnu_Makefiles.md) chapter will explain how to build a process, even if initially is just a bunch of file and it can be done manually, it soon grow more complex, and having the process automated will be more than useful, we will use _Makefile_ for our build script. @@ -14,7 +14,6 @@ One of the most _obscure_, that is always present while building any software, b Finally the kernel is built but not ready to run yet, we need to copy it into a bootable media, the [Generating A Bootable Iso](06_Generating_Iso.md) chapter will show how to create a bootable iso of our kernel, and finally being able to launch it and see the results of our hard work. - For the rest of this part a basic knowledge of compiling these languages is assumed. ## Freestanding Environment @@ -49,15 +48,17 @@ However, each GCC toolchain will use the platform-specific headers by default, w Compiling GCC from source doesn't take too long on a modern CPU (~10 minutes for a complete build on a 7th gen intel mobile cpu, 4 cores), however there are also prebuilt binaries online from places like bootlin, see the useful links appendix for . ## Setting up a build environment + Setting up a proper build environment can be broken down into a few steps: -- Setup a cross compilation toolchain. +- Setup a cross compilation toolchain (refer to the [Appendix E: Cross Platform Building](../99_Appendices/E_Cross_Compilers.md). - Install an emulator. - Install any additional tools. - Setup the bootloader of choice. - Run a hello world to check everything works. ### Getting a Cross Compiler + The easiest approach here is to simply use clang. Clang is designed to be a cross compiler, and so any install of clang can compile to any supported platform. To compile for another platform simply invoke clang as normally would, additionally passing `--target=xyz`, where xyz is the target triplet for the target platform. @@ -80,6 +81,7 @@ The following sections will use the common shorthands to keep things simple: If using clang be sure to remember to pass `--target=xyz` with each command. This is not necessary with GCC. ### Building C Source Files + Now that we have a toolchain setup we can test it all works by compiling a C file. Create a C source file, its contents don't matter here as we wont be running it, just telling it compiles. @@ -212,9 +214,9 @@ Getting access to these debug symbols is dependent on the boot protocol used: Multiboot 2 provides the Elf-Symbols (section 3.6.7 of the spec) tag to the kernel which provides the elf section headers and the location of the string table. Using these is described below in the stivale section. -### Stivale 2 +### Limine Protocol -Stivale2 uses a similar and slightly more complex (but more powerful) mechanism of providing the entire kernel file in memory. This means we're not limited to just using elf files, and can access debug symbols from a kernel in any format. This is because we have the file base address and length and have to do the parsing by ourselves. +Limine uses a similar and slightly more complex (but more powerful) mechanism of providing the entire kernel file in memory. ### ELFs Ahead, Beware! @@ -239,18 +241,19 @@ Languages built around the C model will usually perform some kind of name mangli ### Locating The Symbol Table -We'll need to access the data stored in the string table quite frequently for looking up symbols, so let's calculate that and store it in the variable `char* strtab_data`. For both protocols it's assumed that we have found the tag returned by the bootloader that contains the location of the elf file/elf symbols. +We'll need to access the data stored in the string table quite frequently for looking up symbols, so let's calculate that and store it in the variable `char* strtab_data`. For both protocols it's assumed that we have found the tag returned by the bootloader that contains the location of the elf file/elf symbols (for more details, see the [Boot Protocols](./02_Boot_Protocols.md) chapter. ```c //multiboot 2 multiboot_tag_elf_sections* sym_tag; const char* strtab_data = sym_tag->sections[sym_tag->shndx].sh_offset; -//stivale 2 -stivale2_struct_tag_kernel_file* file_tag; -Elf64_Ehdr* hdr = (Elf64_Ehdr*)file_tag->kernel_file; -Elf64_Shdr* shdrs = (Elf64_Shdr*)(file_tag->kernel_file + hdr->shoff); -const char* strtab_data = shdrs[hdr->e_shstrndx].sh_offset; +//Limine Protocol +struct limine_kernel_file_response file_response; +struct limine_file kernel_file = file_response->kernel_file; +Elf64_Ehdr* hdr = (Elf64_Ehdr*) kernel_file->address; +Elf64_Shdr* shdrs = (Elf64_Shdr*) (kernel_file->address + hdr->shoff); +const char* strtab_data = shdrs[hdr->e_shstrndx].sh_offset + kernel_file->address ``` To find the symbol table, iterate through the section headers until one with the name `.symtab` is found. @@ -288,4 +291,4 @@ void print_symbol(uint64_t addr) } ``` -A quick note about getting the symbol table data address: On multiboot `sym_tab->sh_offset` will be the physical address of the data, while stivale2 will return the original value, which is an offset from the beginning of the file. This means for stivale 2 we would add `file_tag->kernel_base` to this address to get its location in memory. +A quick note about getting the symbol table data address: On multiboot `sym_tab->sh_offset` will be the physical address of the data, while limine protocol will return the original value, which is an offset from the beginning of the file. This means for limine protocol we need to add `kernel_file->address` to this address to get its location in memory. diff --git a/06_Userspace/05_Example_ABI.md b/06_Userspace/05_Example_ABI.md index 29a36da..5f34f9a 100644 --- a/06_Userspace/05_Example_ABI.md +++ b/06_Userspace/05_Example_ABI.md @@ -68,8 +68,4 @@ Now the question is what syscalls should we start to implement? As per usual thi * A way to map, unmap, and modify protections of virtual memory. * A way to terminate the current thread, since the wrapper function used in the scheduler chapter only works for kernel threads. -The list above acts just as a starting point, but the idea is that we want to expose most of the kernel can do, for example we probably want syscalls to create/terminate tasks and thread, syscalls to access files on different filesystems, accessing devices. - - - - +The list above acts just as a starting point, but the idea is that we want to expose most of the kernel can do, for example we probably want syscalls to create/terminate tasks and thread, syscalls to access files on different filesystems, accessing devices. From dafe0f1af480e1b74cb9b243a7a1070d1717b250 Mon Sep 17 00:00:00 2001 From: Ivan Gualandri Date: Sat, 5 Oct 2024 00:14:07 +0100 Subject: [PATCH 02/13] Replace stivale 2 protocol (cont) --- 01_Build_Process/02_Boot_Protocols.md | 49 +++++++++++++++++---------- 01_Build_Process/05_Generating_Iso.md | 2 +- 09_Loading_Elf/01_Elf_Theory.md | 2 +- 3 files changed, 33 insertions(+), 20 deletions(-) diff --git a/01_Build_Process/02_Boot_Protocols.md b/01_Build_Process/02_Boot_Protocols.md index f6ef9dd..6b7c2b5 100644 --- a/01_Build_Process/02_Boot_Protocols.md +++ b/01_Build_Process/02_Boot_Protocols.md @@ -2,19 +2,19 @@ A boot protocol defines the machine state when the kernel is given control by the bootloader. It also makes several services available to the kernel, like a memory map of the machine, a framebuffer and sometimes other utilities like uart or kernel debug symbols. -This chapter covers 2 protocols _Sultiboot 2_ and _Stivale 2_: +This chapter covers 2 protocols _Multiboot 2_ and _Limine Protocol_: * _Multiboot 2_ supercedes multiboot 1, both of which are the native protocols of grub. Meaning that anywhere grub is installed, a multiboot kernel can be loaded. making testing easy on most linux machines. _Multiboot 2_ is quite an old, but very robust protocol. -* _Stivale 2_ (also superceding stivale 1) is the native protocol of the limine bootloader. Limine and stivale were designed many years after multiboot 2 as an attempt to make hobbyist OS development easier. _Stivale 2_ is a more complex spec to read through, but it leaves the machine in a more known state prior to handing off to the kernel. +* _Limine Protocol_ (it was preceded by Stivale 1 and 2) is the native protocol of the Limine bootloader. Limine and Stivale protocols were designed many years after Multiboot 2 as an attempt to make hobbyist OS development easier. _Limine Protocol_ is a more complex spec to read through, but it leaves the machine in a more known state prior to handing off to the kernel. -Recently limine has added a new protocol (the limine boot protocol) which is not covered here. It's based on stivale2, with mainly architectural changes, but similar concepts behind it. If familiar with the concepts of stivale 2, the limine protocol is easy enough to understand. +The Limine protocol is based on Stivale2 (it was covered in earlier version of this book), with mainly architectural changes, but similar concepts behind it. If familiar with the concepts of stivale 2, the limine protocol is easy enough to understand. All the referenced specifications and documents are provided as links at the start of this chapter/in the readme. ## What about the earlier versions? -Both protocols have their earlier versions (_multiboot 1 & stivale 1_), but these are not worth bothering with. Their newer versions are objectively better and available in all the same places. Multiboot 1 is quite a simple protocol, and a lot of tutorials and articles online like to use it because of that: however its not worth the limited feature set we get for the short term gains. The only thing multiboot 1 is useful for is booting in qemu via the `-kernel` flag, as qemu can only process mb1 kernels like that. This option leaves a lot to be desired in the `x86` emulation, so there are better ways to do that. +Multiboot protocol has an earlier verison (_Multiboot 1_), while the limine prorocol was preceded by a different protocol, the _Stivale 1/2_. but in both cases. they are not worth bothering with. Their newer versions are objectively better and available in all the same places. Multiboot 1 is quite a simple protocol, and a lot of tutorials and articles online like to use it because of that: however its not worth the limited feature set we get for the short term gains. The only thing `multiboot 1` is useful for is booting in qemu via the `-kernel` flag, as qemu can only process mb1 kernels like that. This option leaves a lot to be desired in the `x86` emulation, so there are better ways to do that. ## Why A Bootloader At All? @@ -203,33 +203,46 @@ The interface between a higher level language like C and assembly (or another hi *Authors Note: If you're unsure of why we load a stack before jumping to compiled code in the kernel, it's simply required by all modern languages and compilers. The stack (which operates like the data structure of the same name) is a place to store local data that doesn't in the registers of a platform. This means local variables in a function or parts of a complex calculation. It's become so universal that it has also adopted other uses over time, like passing function arguments (sometimes) and being used by hardware to inform the kernel of things (the iret frame on x86).* -## Stivale 2 +## Limine Protocol -Stivale 2 is a much newer protocol, designed for people making hobby operating systems. It sets up a number of things to make a new kernel developer's life easy. -While multiboot 2 is about providing just enough to get the kernel going, keeping things simple for the bootloader, stivale2 creates more work for the bootloader (like initializing other cores, launching kernels in long mode with a pre-defined page map), which leads to the kernel ending up in a more comfortable development environment. The downsides of this approach are that the bootloader may need to be more complex to handle the extra features, and certain restrictions are placed on the kernel. Like the alignment of sections, since the bootloader needs to set up paging for the kernel. +The _Limine Protocol_ that has replaced the _Stivale_ protocol, is following the same philosophy, and is designed for people making hobby operating systems, it sets up a number of things to make a new kernel developer's life easy. +While _Multiboot 2_ is about providing just enough to get the kernel going, keeping things simple for the bootloader, _Limine_ creates more work for the bootloader (like initializing other cores, launching kernels in long mode with a pre-defined page map), which leads to the kernel ending up in a more comfortable development environment. The downsides of this approach are that the bootloader may need to be more complex to handle the extra features, and certain restrictions are placed on the kernel. Like the alignment of sections, since the bootloader needs to set up paging for the kernel. -To use stivale2, we'll need a copy of the limine bootloader. A link to it and the stivale2 specification are available at the start of this chapter. There is also a C header file containing all the structs and magic numbers used by the protocol. A link to a barebones example is also provided. +To use this protocol, we'll need a copy of the _Limine bootloader_. A link to it and the specification are available in the appendices. There is also a C header file containing all the structs and magic numbers used by the protocol. A link to a barebones example is also provided. -It operates in a similar way to multiboot 2, by using a linked list of tags, although this time in both directions (kernel -> bootloader and bootloader -> kernel). Tags from the kernel to the bootloader are called `header_tag`s, and ones returned from the bootloader are called `struct_tag`s. -Stivale 2 has a number of major differences to multiboot 2 though: +It is centered around the concept of `request/response`. For every information that we need from the bootloader, we provide a `request` structure, and it will return us with a `response`. + +Limine has a number of major differences to multiboot 2 though: - The kernel starts in 64-bit long mode, by default. No need for a protected mode stub to setup up some initial paging. - The kernel starts with the first 4GB of memory and any usable regions of memory identity mapped. -- Stivale 2 also sets up a _higher half direct map_, or _hhdm_. This is the same identity map as the lower half, but it starts at the `hhdm_offset` returned in a struct tag when the kernel runs. The idea is that as long we ensure all the pointers are in the higher half, we can zero the bottom half of the page tables and easily be ready for userspace programs. No need to move code/data around. +- Limine protocol also sets up a _higher half direct map_, or _hhdm_. This is the same identity map as the lower half, but it starts at the `hhdm_offset` returned in a struct tag when the kernel runs. The idea is that as long we ensure all the pointers are in the higher half, we can zero the bottom half of the page tables and easily be ready for userspace programs. No need to move code/data around. - A well-defined GDT is provided. -- Unlike _mb2_, a distinction is made between usable memory and the memory used by the bootloader, kernel/modules, and framebuffer. These are separate types in the memory, and don't intersect. Meaning usable memory regions can be used immediately. +- Unlike _Multiboot2_, a distinction is made between usable memory and the memory used by the bootloader, kernel/modules, and framebuffer. These are separate types in the memory, and don't intersect. Meaning usable memory regions can be used immediately. -To get the next tag in the chain, it's as simple as: +A `request` always has three members at the beginning of the structure: +```c +struct limine_example_request { + uint64_t id[4]; + uint64_t revision; + struct limine_example_response *response; + // the members that follow depends on the request type +}; ``` -stivale2_tag* next_tag = (stivale2_tag*)current_tag->next; -if (next_tag == NULL) - //we reached the end of the list. -``` +Where the fields are: + +* `id` is a magic number that the bootloader uses to find and identify the requests within the executable. It is 8 byte aligned. For every type of request there can be only one. If there are multiple requests with the same id the bootloader will refuse to start. +* `revision` is the revision of the request that the kernel provides. This number is bumped each time a new member is added to it. It starts from 0. It's backward compatible, that means if the bootloader does not support the revision of the request, it will be processed as if were the highest revision supported. +* `response` this will contain the response by limine, this field is filled by the bootloader at load time. If there was an error processing the request, or the request was not supported, the field is left as it is, so for example if it was set to `NULL`, it will stay this way. + +All the other fields depends on the type of the request. + +The response instead, has only one mandatory field, it's the `revision` field, that like in the request, it marks the revision of the response field there is no coupling between response and request `revision` number. ### Fancy Features -Stivale 2 also provides some more advanced features: +Limine also provides some more advanced features: - It can enable 5 level paging, if requested. - It boots up AP (all other) cores in the system, and provides an easy interface to run code on them. diff --git a/01_Build_Process/05_Generating_Iso.md b/01_Build_Process/05_Generating_Iso.md index 622c1d5..ea14335 100644 --- a/01_Build_Process/05_Generating_Iso.md +++ b/01_Build_Process/05_Generating_Iso.md @@ -44,7 +44,7 @@ menuentry "My Operating System" { Note the last command `boot`, this tells grub to complete the boot process and actually run our kernel. -## Limine (Stivale 2, multiboot 2) +## Limine (Limine Protocol, Multiboot 2) The process for limine is a little more manual: we must build the iso ourselves, and then use the provided tool to install limine onto the iso we created. To get started we'll want to create a working directory to use as the root of our iso. In this case we'll use `disk/`. Next we'll need to clone the latest binary branch of limine (using `git clone https://github.com/limine-bootloader/limine.git --branch=v3.0-branch-binary --depth=1`) and copy 'limine.sys', 'limine-cd.bin', and 'limine-cd-efi.bin' into `disk/limine/`, creating that directory if it doesn't exist. diff --git a/09_Loading_Elf/01_Elf_Theory.md b/09_Loading_Elf/01_Elf_Theory.md index 644fea2..d1e95ec 100644 --- a/09_Loading_Elf/01_Elf_Theory.md +++ b/09_Loading_Elf/01_Elf_Theory.md @@ -119,7 +119,7 @@ Before looking at some example code our VMM will need a new function that tries void* vmm_alloc_at(uintptr_t addr, size_t length, size_t flags); ``` -Alternatively the in `vmm_alloc` function we can make use of the `flag` parameter, and add a new flag like `VM_FLAG_AT_ADDR` that indicates the VMM should use the extra arg as the virtual address. Bear in mind that if we're loading a program into another address space we will need a way to copy the phdr contents into that address space. The specifics of this don't matter too much, as long as there is a way to do it. +Alternatively in the `vmm_alloc` function we can make use of the `flag` parameter, and add a new flag like `VM_FLAG_AT_ADDR` that indicates the VMM should use the extra arg as the virtual address. Bear in mind that if we're loading a program into another address space we will need a way to copy the phdr contents into that address space. The specifics of this don't matter too much, as long as there is a way to do it. The reason we need to use a specific address is that the code and data contained in the ELF are compiled and linked assuming that they're at that address. There might be code that jumps to a fixed address or data that is expected to be at a certain address. If we don't copy the program header where it expects to be, the program may break. From 826fa48f6ee10ad03b74e08692e88794e8dd6ff2 Mon Sep 17 00:00:00 2001 From: Ivan Gualandri Date: Tue, 8 Oct 2024 23:10:10 +0100 Subject: [PATCH 03/13] Stivale2 replacement, cont... --- 01_Build_Process/02_Boot_Protocols.md | 28 ++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/01_Build_Process/02_Boot_Protocols.md b/01_Build_Process/02_Boot_Protocols.md index 6b7c2b5..360ebcc 100644 --- a/01_Build_Process/02_Boot_Protocols.md +++ b/01_Build_Process/02_Boot_Protocols.md @@ -1,4 +1,4 @@ -# Boot Protocols + # Boot Protocols A boot protocol defines the machine state when the kernel is given control by the bootloader. It also makes several services available to the kernel, like a memory map of the machine, a framebuffer and sometimes other utilities like uart or kernel debug symbols. @@ -238,7 +238,7 @@ Where the fields are: All the other fields depends on the type of the request. -The response instead, has only one mandatory field, it's the `revision` field, that like in the request, it marks the revision of the response field there is no coupling between response and request `revision` number. +The response instead, has only one mandatory field, it's the `revision` field, that like in the request, it marks the revision of the response field. Note that there is no coupling between response and request `revision` number. ### Fancy Features @@ -248,26 +248,36 @@ Limine also provides some more advanced features: - It boots up AP (all other) cores in the system, and provides an easy interface to run code on them. - It supports KASLR, loading our kernel at a random offset each time. - It can also provide things like EDID blobs, address of the PXE server (if booted this way), and a device tree blob on some platforms. -- A fully ANSI-compliant terminal is provided. This does require the kernel to make certain promises about memory layout and the GDT, but it's a very useful debug tool or basic shell in the early stages. The limine bootloader not only supports x86, but tentatively supports aarch64 as well (uefi is required). There is also a stivale2-compatible bootloader called Sabaton, providing broader support for ARM platforms. -### Creating a Stivale2 Header -The limine bootloader provides a `stivale2.h` file which contains a number of nice definitions for us, otherwise everything else here can be placed inside of a c/c++ file. +### Creating a Limine Header +The limine bootloader provides a `limine.h` file which contains a number of nice definitions for us, otherwise everything else here can be placed inside of a c/c++ file. *Authors Note: I like to place my limine header tags in a separate file, for organisation purposes, but as long as they appear in the final binary, they can be anywhere. You can also implement this in assembly if you really want.* -First of all, we'll need an extra section in our linker script, this is how the bootloader knows our kernel can be booted via stivale2: +First of all, we'll need an extra section in our linker script, this is where all the limine requests will be placed: ``` -.stivale2hdr : +.requests : { - KEEP(*(.stivale2hdr)) -} + KEEP(*(.requests_start_marker)) + KEEP(*(.requests)) + KEEP(*(.requests_end_marker)) +} :requests ``` If not familiar with the `KEEP()` command in linker scripts, it tells the linker to keep that section even if it's not referenced by anything. Useful in this case, since the only reference will be the bootloader, which the linker can't know about at link-time. +First thing we want to do is set the base revision of the protocol. Latest version as time of writing is `2`, this can be done with the following line of code: + +```c +__attribute__((used, section(".requests"))) +static volatile LIMINE_BASE_REVISION(2); +``` + +If not familiar with the `__attribute__(())` syntax, it's a compiler extension (both clang and GCC support it) that allows us to do certain things our language wouldn't normally allow. This attribute specified that this variable should go into the `.requests` section, as is required by the Limine spec. + Next we'll need to create space for our stack (stivale2 requires us to provide our own) and define the stivale2 header, like so: ```c From 12a3cba6d05b8a3beb09fd0fbaa7698bc03e7303 Mon Sep 17 00:00:00 2001 From: Ivan Gualandri Date: Thu, 10 Oct 2024 20:14:44 +0100 Subject: [PATCH 04/13] replace stivale2 cont. --- 01_Build_Process/02_Boot_Protocols.md | 85 ++++++++------------------- 1 file changed, 25 insertions(+), 60 deletions(-) diff --git a/01_Build_Process/02_Boot_Protocols.md b/01_Build_Process/02_Boot_Protocols.md index 360ebcc..8436d2b 100644 --- a/01_Build_Process/02_Boot_Protocols.md +++ b/01_Build_Process/02_Boot_Protocols.md @@ -252,6 +252,7 @@ Limine also provides some more advanced features: The limine bootloader not only supports x86, but tentatively supports aarch64 as well (uefi is required). There is also a stivale2-compatible bootloader called Sabaton, providing broader support for ARM platforms. ### Creating a Limine Header + The limine bootloader provides a `limine.h` file which contains a number of nice definitions for us, otherwise everything else here can be placed inside of a c/c++ file. *Authors Note: I like to place my limine header tags in a separate file, for organisation purposes, but as long as they appear in the final binary, they can be anywhere. You can also implement this in assembly if you really want.* @@ -276,65 +277,39 @@ __attribute__((used, section(".requests"))) static volatile LIMINE_BASE_REVISION(2); ``` -If not familiar with the `__attribute__(())` syntax, it's a compiler extension (both clang and GCC support it) that allows us to do certain things our language wouldn't normally allow. This attribute specified that this variable should go into the `.requests` section, as is required by the Limine spec. - -Next we'll need to create space for our stack (stivale2 requires us to provide our own) and define the stivale2 header, like so: +If not familiar with the `__attribute__(())` syntax, it's a compiler extension (both clang and GCC support it) that allows us to do certain things our language wouldn't normally allow. This attribute specified that this variable should go into the `.requests` section, as is required by the Limine spec and it is marked as used. -```c -#include +The main requirement of any limine protocol variable, request is that, is that the compiler keep them as they are, without any optimization optimizing them, for this reason they are declared as `volatile` and they should be accessed at least once, or marked as `used`. -//8K for the initial stack, a reasonable default -static uint8_t init_stack[0x2000]; +In this protocol, `requests` can be placed anywhere, and there is no need to use any type of list or place them in a specific memory area. For example, let's imagine that we want to get the framebuffer information from the bootloader. In this case we need to declare the `struct limine_framebuffer_request` type while creating a variable for our request: -__attribute__((section(".stivale2hdr))) -static stivale2_header stivale2_hdr = -{ - .entry_point = 0, - .stack = (uintptr_t)init_stack + 0x2000, - .flags = (1 << 1) | (1 << 2) | (1 << 3) | (1 << 4), - .tags = (uintptr_t)&framebuffer_tag +```c +__attribute__((used, section(".requests"))) +static volatile struct limine_framebuffer_request { + .id = LIMINE_FRAMEBUFFER_REQUEST, + .revision = 0 }; ``` -If not familiar with the `__attribute__(())` syntax, it's a compiler extension (both clang and GCC support it) that allows us to do certain things our language wouldn't normally allow. This attribute specified that this variable should go into the `.stivale2hdr` section, as is required by the stivale2 spec. - -Next we set some fields in the stivale2 header: +The requests types are all declared in the `limine.h` header. -- `entry_point`: Is used to override the ELF's entry point address. Set this to zero to use the regular entry function we set in the linker script. -- `stack`: Self explanatory, used to set the stack the kernel code will start with. -- `flags`: A bitfield of flags. Bit 1 asks the bootloader to return higher half addresses to us for tags, modules and other things. Bit 2 asked the bootloader to make use of the nx-bit and write-enable bits in the page tables when loading the kernel. Bit 3 is recommended and enables the bootloader to load us at any physical address as long as the virtual address is the same. Bit 4 is required to be set, as it disables a legacy feature. -- `tags`: A pointer to the first stivale2 tag in the linked list of requests. +For any other information that we need to get from the bootloader, we are going to create a similar request using the correct request type. -In the example above we actually set the first tag to a framebuffer request, so lets see what that would look like: +The last detail is to add the kernel start function (declared in the `ENTRY()` section in the linker script): ```c -static stivale2_header_tag_framebuffer framebuffer_tag = -{ - .tag = - { - .identifier = STIVALE2_HEADER_TAG_FRAMEBUFFER, - .next = 0, - }, - .framebuffer_width = 0, - .framebuffer_height = 0, - .framebuffer_bpp = 0 -}; +void kernel_start(void); ``` -The `framebuffer_*` fields can be used to ask for a specific kind of framebuffer, but leaving them to zero tells the bootloader we want to best possible available. The `next` field can be used to point to the next header tag, if we had another one we wanted. The full list of tags is available in the stivale2 specification (see the useful links appendix). +Since all the bootloader information are provided via `static` variables, the kernel start function doesn't require any particular signature. -The last detail is to change the signature of our kernel entry function to: - -```c -void kernel_start(stivale2_struct* stivale2_data); -``` - -This struct points to a list of tags, each containing details about the machine we're booted on. These are called struct tags (bootloader -> kernel) as opposed to the tags we defined before (header tags: kernel -> bootloader). To get info about a specific feature, simply walk the linked list of tags, the next tag's address is available in the `tag->next` field. The end of the list is indicated by a nullptr. ## Finding Bootloader Tags + Since both multiboot 2 and stivale 2 return their info in linked lists, a brief example of how to traverse these lists is given below. These functions provide a nice abstraction to search the list for a specific tag, rather than manually searching each time. ### Multiboot 2 + Multiboot 2 gives us a pointer to the multiboot info struct, which contains 2x 32-bit fields. These can be safely ignored, as the list is null-terminated (a tag with a type 0, and size of 8). The first tag is at 8 bytes after the start of the mbi. All the structures and defines used here are available in the header provided by the multiboot specification (check the bottom section, in the example kernel), including the `MULTIBOOT_TAG_TYPE_xyz` defines (where xyz is a feature described by a tag). For example the memory map is `MULTIBOOT_TAG_TYPE_MMAP`, and framebuffer is `MULTIBOOT_TAG_TYPE_FRAMEBUFFER`. ```c @@ -361,27 +336,17 @@ void* multiboot2_find_tag(uint32_t type) Lets talk about the last three lines of the loop, where we set the `tag` variable to the next value. The multiboot 2 spec says that tags should always be 8-byte aligned. While this is not a problem most of the time, it is *possible* we could get a misaligned pointer by simply adding `size` bytes to the current pointer. So to be on safe side, and spec-compliant, we'll align the value up to the nearest 8 bytes. -### Stivale 2 -Stivale 2 gives us a pointer to a header at the start of the list, and then each item (including this header) contains a `next` pointer to the next item, and an `id` item with a unique 64-bit identifier for that tag. All the structures and defines are available in the standard `stivale2.h`. We'll know we've reached the end of the list when the `next` pointer is `NULL`. +### Limine -```c -//given to the kernel entry function -stivale2_struct* s2_struct; +Accessing the bootloader response is very simple, and it doesn't require iterating any list. We just need to read the content of the `.response` field of the request structure: -//returns null if tag could not be found -void* stivale2_find_tag(uint64_t id) -{ - stivale2_tag* tag = s2_struct->next; - - while (tag != NULL) - { - if (tag->id == id) - return tag; - tag = tag->next; - } - - return NULL; +```c +//somewhere in the code +if ( framebuffer_request->response != NULL) { + // Do something with the response } ``` -The above function can be used with the defines in `stivale2.h`, which follow the format `STIVALE2_STRUCT_TAG_xyz_ID`, where `xyz` represents the feature that is described in the tag. For example, the framebuffer would be `STIVALE2_STRUCT_TAG_FRAMEBUFFER_ID` and the memory map is `STIVALE2_STRUCT_TAG_MEMMAP_ID`. It's a little verbose, but easy to search for. +Is important to note, for every type of request the `response` field have a different type, in this case it is a pointer to a `struct limine_framebuffer_response`, for more info on all the available requests, and repsonses refer to the protocl documentation. + +The `framebuffer_*` fields can be used to ask for a specific kind of framebuffer, but leaving them to zero tells the bootloader we want to best possible available. The `next` field can be used to point to the next header tag, if we had another one we wanted. The full list of tags is available in the stivale2 specification (see the useful links appendix). From 608b91aa6b7af1f74622a8c5a777915edc1019f6 Mon Sep 17 00:00:00 2001 From: Ivan Gualandri Date: Thu, 10 Oct 2024 21:15:36 +0100 Subject: [PATCH 05/13] replace stivale2 cont. --- 01_Build_Process/02_Boot_Protocols.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/01_Build_Process/02_Boot_Protocols.md b/01_Build_Process/02_Boot_Protocols.md index 8436d2b..caef153 100644 --- a/01_Build_Process/02_Boot_Protocols.md +++ b/01_Build_Process/02_Boot_Protocols.md @@ -293,7 +293,19 @@ static volatile struct limine_framebuffer_request { The requests types are all declared in the `limine.h` header. -For any other information that we need to get from the bootloader, we are going to create a similar request using the correct request type. +Any type of information that needs to be requested to the bootloader, will have it's own `request` variable, with it's type. + +Finally the requests start and end markers needs to be declared: + +```c +__attribute__((used, section(".requests_start_marker"))) +static volatile LIMINE_REQUESTS_START_MARKER; + +__attribute__((used, section(".requests_end_marker"))) +static volatile LIMINE_REQUESTS_END_MARKER; +``` + +Again they can be placed anywhere in the code, since their position will be decided by the section they belongs to. The last detail is to add the kernel start function (declared in the `ENTRY()` section in the linker script): From 8cfef6f2e4745fa988e14bcea8be67dbafb5a720 Mon Sep 17 00:00:00 2001 From: Ivan Gualandri Date: Sun, 13 Oct 2024 00:35:58 +0100 Subject: [PATCH 06/13] stivale replacement (cont) --- 01_Build_Process/05_Generating_Iso.md | 22 +++++++++++++--------- 99_Appendices/H_Useful_Resources.md | 4 ++-- 2 files changed, 15 insertions(+), 11 deletions(-) diff --git a/01_Build_Process/05_Generating_Iso.md b/01_Build_Process/05_Generating_Iso.md index c951d01..b52c8d3 100644 --- a/01_Build_Process/05_Generating_Iso.md +++ b/01_Build_Process/05_Generating_Iso.md @@ -49,7 +49,7 @@ The process for limine is a little more manual: we must build the iso ourselves, To get started we'll want to create a working directory to use as the root of our iso. In this case we'll use `disk/`. Next we'll need to clone the latest binary branch of limine (using `git clone https://github.com/limine-bootloader/limine.git --branch=v3.0-branch-binary --depth=1`) and copy 'limine.sys', 'limine-cd.bin', and 'limine-cd-efi.bin' into `disk/limine/`, creating that directory if it doesn't exist. -Now we can copy our `limine.cfg` and kernel into the working directory. The `limine.cfg` file needs to be in one of a few locations in order to be found, the best place is the root directory. +Now we can copy our `limine.conf` and kernel into the working directory. The `limine.conf` file needs to be in one of a few locations in order to be found, the best place is the root directory. Now we're ready to run the following xorriso command: @@ -65,20 +65,24 @@ That's a lot of flags! Because of how flexible the iso format is, we need to be Now we have an iso with our kernel, config and bootloader files on it. We just need to install limine into the boot partitions, we can do this using `limine-deploy` like so: ```sh -limine-deploy my_iso.iso +./limine/limine bios-install $(IMAGE_NAME).iso ``` -That took a little more work than grub, but this can (and should) be automated as part of the build system. If can't find `limine-deploy` inside the cloned limine directory, we may need to run `make -C limine` from there for it to be build. +That took a little more work than grub, but this can (and should) be automated as part of the build system. If can't find `limine` inside the cloned limine directory, we may need to run `make -C limine` from there for it to be build. -### Limine.cfg +### Limine.conf Similar to grub, limine also uses a config file. This config file has its own documentation, which is available in the limine repository. -The `limine.cfg` file lists each boot entry as a title, followed by a series of key-value pairs. To boot our example from above using stivale 2, our config might look like the following: +The `limine.conf` file lists each boot entry as a title, followed by a series of key-value pairs. To boot our example from above using stivale 2, our config might look like the following: ``` -:My Operating System -PROTOCOL=stivale2 -KERNEL_PATH=boot:///boot/kernel.elf +# The entry name that will be displayed in the boot menu. +/My Operating System + # We use the Limine boot protocol. + protocol: limine + + # Path to the kernel to boot. boot():/ represents the partition on which limine.conf is located. + kernel_path: boot():/boot/kernel.elf ``` -One thing to note is how the kernel's path is specified. It uses the URI format, which begins with a protocol and then a protocol-specific path. In our case we're using the boot disk (`boot://`), and then an absolute path (`/boot/kernel.elf`), which is why we have the triple slash there. A common beginner mistake is only put 2 slashes there, which makes the path relative, and limine will fail to boot. +One thing to note is how the kernel's path is specified. It uses the URI-like format, which begins with a protocol and then a protocol-specific path. In our case we're using the boot disk (`boot():`), and then an absolute path (`/boot/kernel.elf`) diff --git a/99_Appendices/H_Useful_Resources.md b/99_Appendices/H_Useful_Resources.md index 0559a5f..062b195 100644 --- a/99_Appendices/H_Useful_Resources.md +++ b/99_Appendices/H_Useful_Resources.md @@ -7,8 +7,8 @@ This appendix is a collection of links we found useful developing our own kernel - Grub and grub.cfg documentation: [https://www.gnu.org/software/grub/manual/grub/grub.html](https://www.gnu.org/software/grub/manual/grub/grub.html) - Multiboot 2 Specification: [https://www.gnu.org/software/grub/manual/multiboot2/multiboot.html](https://www.gnu.org/software/grub/manual/multiboot2/multiboot.html) - Limine documentation:[https://github.com/limine-bootloader/limine](https://github.com/limine-bootloader/limine) -- Stivale 2 Specification: [https://github.com/stivale/stivale/blob/master/STIVALE2.md](https://github.com/stivale/stivale/blob/master/STIVALE2.md) -- Stivale 2 Barebones: [https://github.com/stivale/stivale2-barebones/](https://github.com/stivale/stivale2-barebones/) +- The Limine Protocol: [https://github.com/limine-bootloader/limine/blob/v8.x/PROTOCOL.md](https://github.com/limine-bootloader/limine/blob/v8.x/PROTOCOL.md) +- Limine C Template: [https://github.com/limine-bootloader/limine-c-template/tree/trunk](https://github.com/limine-bootloader/limine-c-template/tree/trunk) - Sabaton - ARM Stivale 2 Bootloader: [https://github.com/FlorenceOS/Sabaton](https://github.com/FlorenceOS/Sabaton) - Xorriso Documentation: [https://linux.die.net/man/1/xorriso](https://linux.die.net/man/1/xorriso) - GNU Make Documentation: [https://www.gnu.org/software/make/manual/make.html](https://www.gnu.org/software/make/manual/make.html) From 5981f8438f1ca29b9f13e8c7bf92160b79eae55c Mon Sep 17 00:00:00 2001 From: Ivan Gualandri Date: Sun, 13 Oct 2024 20:02:18 +0100 Subject: [PATCH 07/13] stivale replacement cont... --- 01_Build_Process/05_Generating_Iso.md | 2 +- 01_Build_Process/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/01_Build_Process/05_Generating_Iso.md b/01_Build_Process/05_Generating_Iso.md index b52c8d3..7942d19 100644 --- a/01_Build_Process/05_Generating_Iso.md +++ b/01_Build_Process/05_Generating_Iso.md @@ -62,7 +62,7 @@ xorriso -as mkisofs -b limine-cd.bin -no-emul-boot \ That's a lot of flags! Because of how flexible the iso format is, we need to be quite specific when building one. If curious about how crazy things can get, take a look at the flags grub-mkrescue generates behind the scenes. -Now we have an iso with our kernel, config and bootloader files on it. We just need to install limine into the boot partitions, we can do this using `limine-deploy` like so: +Now we have an iso with our kernel, config and bootloader files on it that can boot if using uefi. If we want to make it bootable also using the `bios` an extra step is required, we need to install limine into the boot partitions, it can be done using the `limine` command like so: ```sh ./limine/limine bios-install $(IMAGE_NAME).iso diff --git a/01_Build_Process/README.md b/01_Build_Process/README.md index b4cc14b..c6931a3 100644 --- a/01_Build_Process/README.md +++ b/01_Build_Process/README.md @@ -5,7 +5,7 @@ This part is going to cover all the tools and steps that are needed in order to Below the list of chapters for this part: - [General Overview](01_Overview.md): This chapter will serve as a high level overview of the overall building process, explaining why some steps are needed, and what is their purpose. It also give an introduction to many of the concepts that will be developed in the in the following chapters. -- [Boot Protocols & Bootloaders](02_Boot_Protocols.md): This chapter will explore two different solutions for booting our kernel: _Multiboot2_ and _Stivale_, explaining how they should be used and configured in order to boot our kernel, and what are their difference. +- [Boot Protocols & Bootloaders](02_Boot_Protocols.md): This chapter will explore two different solutions for booting our kernel: _Multiboot2_ and _Limine_, explaining how they should be used and configured in order to boot our kernel, and what are their difference. - [Makefiles](03_Gnu_Makefiles.md): The building script, we are going to use to build our kernel: Makefile. - [Linker Scripts](04_Linker_Scripts.md): This chapter will introduce one of the probably most _obscure_ part of the building process, especially for beginners, it will explain what are the linker scripts, why they are important, and how to write one. - [Generating A Bootable Iso](05_Generating_Iso.md): After building our kernel we want to run it too (yeah like the cake...). In order to do that we need a bootable iso, as only the kernel file is not enough. This chapter will show how to create a bootable iso and start to test it on emulators/real hardware. From 5ddba08780a15bb5ab010c8b42f66a8bff1848e6 Mon Sep 17 00:00:00 2001 From: Ivan Gualandri Date: Sun, 13 Oct 2024 20:05:34 +0100 Subject: [PATCH 08/13] Update updates --- 99_Appendices/J_Updates.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/99_Appendices/J_Updates.md b/99_Appendices/J_Updates.md index 90cbfab..5b0635c 100644 --- a/99_Appendices/J_Updates.md +++ b/99_Appendices/J_Updates.md @@ -58,3 +58,10 @@ Fifth book release * Added missing flags on page table/dirs entries * Rewrite and rearrangement of syscall/sysret section +### Revision 5 + +Relase date: TBD +Sixty Book Release + +* _Stivale 2_ protocol sections have been replaced with Limine protocl, since _stivale2_ has been deprecated. + From 5f9998bdc57f20245da82f78bde76b70e4fcce28 Mon Sep 17 00:00:00 2001 From: Ivan Gualandri Date: Mon, 14 Oct 2024 00:21:11 +0100 Subject: [PATCH 09/13] Fix typo --- 99_Appendices/J_Updates.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/99_Appendices/J_Updates.md b/99_Appendices/J_Updates.md index 5b0635c..71d41b5 100644 --- a/99_Appendices/J_Updates.md +++ b/99_Appendices/J_Updates.md @@ -61,7 +61,7 @@ Fifth book release ### Revision 5 Relase date: TBD -Sixty Book Release +Sixth Book Release * _Stivale 2_ protocol sections have been replaced with Limine protocl, since _stivale2_ has been deprecated. From ca2229c8906d8a6f9602cdb544d1c43a56ee5513 Mon Sep 17 00:00:00 2001 From: Ivan Gualandri Date: Sat, 19 Oct 2024 11:20:46 +0100 Subject: [PATCH 10/13] Changes requested --- 01_Build_Process/05_Generating_Iso.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/01_Build_Process/05_Generating_Iso.md b/01_Build_Process/05_Generating_Iso.md index 7942d19..e2075d5 100644 --- a/01_Build_Process/05_Generating_Iso.md +++ b/01_Build_Process/05_Generating_Iso.md @@ -47,7 +47,7 @@ Note the last command `boot`, this tells grub to complete the boot process and a ## Limine (Limine Protocol, Multiboot 2) The process for limine is a little more manual: we must build the iso ourselves, and then use the provided tool to install limine onto the iso we created. -To get started we'll want to create a working directory to use as the root of our iso. In this case we'll use `disk/`. Next we'll need to clone the latest binary branch of limine (using `git clone https://github.com/limine-bootloader/limine.git --branch=v3.0-branch-binary --depth=1`) and copy 'limine.sys', 'limine-cd.bin', and 'limine-cd-efi.bin' into `disk/limine/`, creating that directory if it doesn't exist. +To get started we'll want to create a working directory to use as the root of our iso. In this case we'll use `disk/`. Next we'll need to clone the latest binary branch of limine that at time of writing is at version `8.x` (using `git clone https://github.com/limine-bootloader/limine.git --branch=v8.x-binary --depth=1`) and copy 'limine.sys', 'limine-cd.bin', and 'limine-cd-efi.bin' into `disk/limine/`, creating that directory if it doesn't exist. Now we can copy our `limine.conf` and kernel into the working directory. The `limine.conf` file needs to be in one of a few locations in order to be found, the best place is the root directory. @@ -65,7 +65,7 @@ That's a lot of flags! Because of how flexible the iso format is, we need to be Now we have an iso with our kernel, config and bootloader files on it that can boot if using uefi. If we want to make it bootable also using the `bios` an extra step is required, we need to install limine into the boot partitions, it can be done using the `limine` command like so: ```sh -./limine/limine bios-install $(IMAGE_NAME).iso +./limine/limine bios-install my_iso.iso ``` That took a little more work than grub, but this can (and should) be automated as part of the build system. If can't find `limine` inside the cloned limine directory, we may need to run `make -C limine` from there for it to be build. From 0dc42c79de4b81ead58ff773eac9f8161fc4139b Mon Sep 17 00:00:00 2001 From: Ivan Gualandri Date: Sun, 20 Oct 2024 15:00:04 +0100 Subject: [PATCH 11/13] Changes requested --- 01_Build_Process/02_Boot_Protocols.md | 8 +------- 99_Appendices/J_Updates.md | 2 +- 2 files changed, 2 insertions(+), 8 deletions(-) diff --git a/01_Build_Process/02_Boot_Protocols.md b/01_Build_Process/02_Boot_Protocols.md index caef153..94ed433 100644 --- a/01_Build_Process/02_Boot_Protocols.md +++ b/01_Build_Process/02_Boot_Protocols.md @@ -309,10 +309,6 @@ Again they can be placed anywhere in the code, since their position will be deci The last detail is to add the kernel start function (declared in the `ENTRY()` section in the linker script): -```c -void kernel_start(void); -``` - Since all the bootloader information are provided via `static` variables, the kernel start function doesn't require any particular signature. @@ -359,6 +355,4 @@ if ( framebuffer_request->response != NULL) { } ``` -Is important to note, for every type of request the `response` field have a different type, in this case it is a pointer to a `struct limine_framebuffer_response`, for more info on all the available requests, and repsonses refer to the protocl documentation. - -The `framebuffer_*` fields can be used to ask for a specific kind of framebuffer, but leaving them to zero tells the bootloader we want to best possible available. The `next` field can be used to point to the next header tag, if we had another one we wanted. The full list of tags is available in the stivale2 specification (see the useful links appendix). +Is important to note, for every type of request the `response` field have a different type, in this case it is a pointer to a `struct limine_framebuffer_response`, for more info on all the available requests, and repsonses refer to the protocol documentation. diff --git a/99_Appendices/J_Updates.md b/99_Appendices/J_Updates.md index 71d41b5..16f7a34 100644 --- a/99_Appendices/J_Updates.md +++ b/99_Appendices/J_Updates.md @@ -63,5 +63,5 @@ Fifth book release Relase date: TBD Sixth Book Release -* _Stivale 2_ protocol sections have been replaced with Limine protocl, since _stivale2_ has been deprecated. +* _Stivale 2_ protocol sections have been replaced with Limine protocol, since _stivale2_ has been deprecated. From f8a34f506166ba679c0ce36de662037908b534aa Mon Sep 17 00:00:00 2001 From: Ivan Gualandri Date: Tue, 22 Oct 2024 19:14:39 +0100 Subject: [PATCH 12/13] Minor changes --- 01_Build_Process/02_Boot_Protocols.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/01_Build_Process/02_Boot_Protocols.md b/01_Build_Process/02_Boot_Protocols.md index 94ed433..c02dd38 100644 --- a/01_Build_Process/02_Boot_Protocols.md +++ b/01_Build_Process/02_Boot_Protocols.md @@ -293,7 +293,7 @@ static volatile struct limine_framebuffer_request { The requests types are all declared in the `limine.h` header. -Any type of information that needs to be requested to the bootloader, will have it's own `request` variable, with it's type. +Any type of information that needs to be requested to the bootloader, will have it's own `request` variable, with it's own type. Finally the requests start and end markers needs to be declared: @@ -307,7 +307,7 @@ static volatile LIMINE_REQUESTS_END_MARKER; Again they can be placed anywhere in the code, since their position will be decided by the section they belongs to. -The last detail is to add the kernel start function (declared in the `ENTRY()` section in the linker script): +The last detail is to add the kernel start function (declared in the `ENTRY()` section in the linker script). Since all the bootloader information are provided via `static` variables, the kernel start function doesn't require any particular signature. From c02ee96a931ea6d34882a5c239551623a1c0858f Mon Sep 17 00:00:00 2001 From: Ivan Gualandri Date: Tue, 22 Oct 2024 19:53:58 +0100 Subject: [PATCH 13/13] Minor fix --- 01_Build_Process/02_Boot_Protocols.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/01_Build_Process/02_Boot_Protocols.md b/01_Build_Process/02_Boot_Protocols.md index c02dd38..b2fe4cf 100644 --- a/01_Build_Process/02_Boot_Protocols.md +++ b/01_Build_Process/02_Boot_Protocols.md @@ -277,7 +277,7 @@ __attribute__((used, section(".requests"))) static volatile LIMINE_BASE_REVISION(2); ``` -If not familiar with the `__attribute__(())` syntax, it's a compiler extension (both clang and GCC support it) that allows us to do certain things our language wouldn't normally allow. This attribute specified that this variable should go into the `.requests` section, as is required by the Limine spec and it is marked as used. +If not familiar with the `__attribute__(())` syntax, it's a compiler extension (both clang and GCC support it) that allows us to do certain things our language wouldn't normally allow. This attribute specified that this variable should go into the `.requests` section, as is required by the Limine spec and it is marked as `used`. The main requirement of any limine protocol variable, request is that, is that the compiler keep them as they are, without any optimization optimizing them, for this reason they are declared as `volatile` and they should be accessed at least once, or marked as `used`.