Chapter 8: Loading an ELF file

.pandoc/decorators/appendix_section.tex

@@ -0,0 +1,3 @@
+\part*{Appendices}
+\addcontentsline{toc}{part}{Appendices}
+\appendix

.pandoc/decorators/part_architecture.tex

@@ -0,0 +1 @@
+\part{Architecture And Basic Driers}

.pandoc/decorators/part_build.tex

@@ -0,0 +1 @@
+\part{Build Process}

.pandoc/decorators/part_conclusion.tex

@@ -0,0 +1 @@
+\part{Conclusion}

.pandoc/decorators/part_elfs.tex

@@ -0,0 +1 @@
+\part{Loading ELFs}

.pandoc/decorators/part_ipc.tex

@@ -0,0 +1 @@
+\part{Inter Process Communication} 

.pandoc/decorators/part_memory.tex

@@ -0,0 +1 @@
+\part{Memory Management}

.pandoc/decorators/part_scheduling.tex

@@ -0,0 +1 @@
+\part{Scheduling and Processes}

.pandoc/decorators/part_userspace.tex

@@ -0,0 +1 @@
+\part{Userspace}

.pandoc/decorators/part_vfs.tex

@@ -0,0 +1 @@
+\part{The Virtual File System}

.pandoc/decorators/part_video_output.tex

@@ -0,0 +1 @@
+\part{Video Output}

.pandoc/pandoc.yaml

@@ -5,7 +5,7 @@ author:
   - Dean T.
 header-includes:
  - \usepackage{fvextra}
- - \usepackage{appendix}
+ - \usepackage[page,toc,titletoc,title]{appendix}
  - \DefineVerbatimEnvironment{Highlighting}{Verbatim}{breaklines,commandchars=\\\{\}}
 book: true
 ---

00_Introduction/02_AssumedKnowledge.md

@@ -8,6 +8,6 @@ As such, below is a list of the recommended prior experience before continuing w
 
 - Intermediate understanding of the C programming language. Mastery is not required, but you should be very familiar with the ins and outs of the language, especially pointers and pointer arithmetic.
 - You should be comfortable compiling and debugging code in userspace. GDB is recommended as several emulators provide a GDB server you can use to step through your kernel.
-- Knowledge and experience using common data structures like intrusive linked lists.
+- Knowledge and experience using common data structures like intrusive linked lists. While we may use array notation at several points to help visualize what's going on, you won't want to place arbitrary limits on your kernel by using fixed size arrays.
 
 If you feel confident in your knowledge of the above, please read on! If not, don't be discouraged. There are plenty of resources available for learning, and you can always come back later.

01_Build_Process/01_README.md

@@ -1,11 +1,13 @@
 # Kernel Build Process & Booting
-This section covers some of the very first topics you'll need to know when building a kernel. Why you need a custom linker script, and how to write one, setting up a build system (we use make) and finally which boot protocol and bootloader to use.
 
-- [General Overview](02_Overview.md)
-- [Boot Protocols & Bootloaders](03_Boot_Protocols.md)
-- [Makefiles](04_Gnu_Makefiles.md)
-- [Linker Scripts](05_Linker_Scripts.md)
-- [Generating A Bootable Iso](06_Generating_Iso.md)
+An OS like any other project needs to be built, and packaged in a special way in order to be "booted". 
+This part will cover all the steps needed to have an initial set of building script for our os, and also explore some of the bootloader that can be used to load our kernel. 
+
+- [General Overview](02_Overview.md) This chapter will serve as a high level overview of the building process, introducing some of the basic concepts and tools that will be used in the following chapters and showing two possible compiler options
+- [Boot Protocols & Bootloaders](03_Boot_Protocols.md) Here we will explore two possible solutions for booting our kernel: Multiboot2 and Stivale, explaining how they must be used and configured in order to boot our kernel
+- [Makefiles](04_Gnu_Makefiles.md) The building script, we are going to use: Makefile. 
+- [Linker Scripts](05_Linker_Scripts.md) Probably one of the most _obscure_ parts of the building process, especially for beginners, this chapter explains what are the linker scripts, why they are important, and how to write one.
+- [Generating A Bootable Iso](06_Generating_Iso.md) After building our kernel we want to run it too (yeah like the cake...). In order to do that we need a bootable iso, as only the kernel file is not enough. This chapter will show how to create a bootable iso and start to test it on emulators/real hardware.
 
 ## Useful Links
 

01_Build_Process/02_Overview.md

@@ -132,20 +132,23 @@ For an explanation of the above linker flags used:
 One other linker option to keep in mind is `-M`, which displays the link map that was generated. This is a description of how and where the linker allocated everything in the final file. It can be seen as a manual symbol table.
 
 ### Building with Makefiles
+
 Now compiling and building one file isn't so bad, but the same process for multiple files can quickly get out of hand. This is especially true when you only want to build files that have been modified, and use previously compiled versions of other files.
 
 Make is a common tool used for building many pieces of software due to how easy and commmon `make` is. Specifically GNU make. GNU make is also chosen as it comes installed by default in many linux distros, and is almost always available if it's not already installed.
 
 There are other make-like tools out there (xmake, nmake) but these are less popular, and therefore less standardized. For the lowest common denominator we'll stick with the original GNU make, which is discussed later on in this chapter, in it's own section.
 
 ## Quick Addendum: Easily Generating a Bootable Iso
+
 There are more details to this, however most bootloaders will provide a tool that lets you create a bootable iso, with the kernel, the bootloader itself and any other files you might want. For grub this is `grub-mkrescue` and limine provides `limine-install` for version 2.x or `limine-deploy` for version 3.x.
 
 While the process of generating an iso is straightforward enough when using something like xorisso, the process of installing a bootloader into that iso is usually bootloader dependent. This is covered more in detail in it's own section.
 
 If you're just here for a quick reference, grub uses `grub-mkrescue` and a grub.cfg file, limine reqiures you to build the iso yourself with a limine.cfg on it, and then run `limine-deploy`.
 
 ## Testing with An Emulator
+
 Now we have an iso with our bootloader and kernel installed onto it, how do we test this? Well there's a number of emulators out there, with varying levels of performance and debug utility. Generally the more debug functionality an emulator provides, the slower it will run. A brief comparison of some common x86 emulators is provided below.
 
 - Qemu is great middle ground between debugging and speed. By default your OS will run using software virtualization (qemu's implementation is called tcg), but you can optionally enable kvm with the `--enable-kvm` flag for hardware-assisted virtualization. Qemu also provides a wide range of supported platforms.
@@ -177,6 +180,7 @@ There are a few other qemu flags you might want to be aware of:
 - `-no-shutdown` some configurations of qemu will shutdown if `-no-reboot` is specified, instead of pausing the VM. This flag forces qemu to stay open, but paused.
 
 ## Building and Using Debugging Symbols
+
 You'll never know when you need to debug your kernel, especially when running in a virtualized environment. Having debug symbols included in your kernel will increase the file size, but can be useful. If you want to remove them from an already compiled kernel the `strip` program can be used to strip excess info from a file.
 
 Including debug info in the kernel is the same as any other program, simply compile with the `-g` flag. 
@@ -186,12 +190,15 @@ There are different versions of DWARF (the debugging format used by elf files),
 Getting access to these debug symbols is dependent on the boot protocol used:
 
 ### Multiboot 2
+
 Multiboot 2 provides the Elf-Symbols (section 3.6.7 of the spec) tag to the kernel which provides the elf section headers and the location of the string table. Using these is described below in the stivale section.
 
 ### Stivale 2
+
 Stivale2 uses a similar and slightly more complex (but more powerful) mechanism of providing the entire kernel file in memory. This means you're not limited to just using elf files, and can access debug symbols from a kernel in any format. This is because you have the file base address and length and have to do the parsing yourself.
 
 ### ELFs Ahead, Beware!
+
 This section is included to show how elf symbols could be loaded and parsed, but it is not a tutorial on the elf format itself. If you're unfamiliar with the format, give the elf64 specification a read! It's quite straightforward, and written very plainly. This section makes refernce to a number a of structures and fields from the specification.
 
 With that warning out of the way, let's look at the two fields from the elf header we're interested in. If you're using the multiboot 2 info, you will be given these fields directly. For stivale 2, you will need to parse the elf header yourself. We're interested in `e_shoff` (the section header offset) and `e_shstrndx` (the section header string index).
@@ -212,6 +219,7 @@ Now to get the name of a section, you'll need to find the matching symbol entry,
 Languages built around the C model will usually perform some kind of name mangling to enable features like function overloading, namespaces and so on. This is a whole topic on it's own. Name mangling can be through of as a translation that takes place, to allow things like function overloading and templates to work in the C naming model.
 
 ### Locating The Symbol Table
+
 We'll need to access the data stored in the string table quite frequently for looking up symbols, so let's calculate that and store it in the variable `char* strtab_data`. For both protocols it's assumed that you have found the tag returned by the bootloader that contains the location of the elf file/elf symbols.
 
 ```c

02_Architecture/01_README.md

@@ -1,4 +1,4 @@
-# Architecture
+# Architecture and Drivers
 
 Before going beyond a basic "hello world" and implementing the first real parts of our kernel, there are some key concepts about how the CPU operates that we have to understand. What is an interrupt, and how do we handle it? What does it mean to mask them? What is the GDT and what is it's purpose?
 
@@ -51,3 +51,7 @@ When an unexpected event happens, the cpu will immediately stop the current code
 
 The interrupted code is usually never aware that an interrupt even occured, and should continue on as normal.
 
+## Drivers
+
+Not device drivers for graphic cards, network interfaces, and other hardware, but on early stages of development we will need some basic drivers to implement some of the future features, for example we will need to have at least one supported Timer to implement the scheduler, we will most likely want to add a basic support for a keyboard in order to implement a cli, these topics will be covered in this section, along with other architecture specific drivers required by the CPU.
+

02_Architecture/02_Hello_World.md

@@ -0,0 +1,119 @@
+# Hello World
+
+During the development of our kernel we will need to debug a lot, and checking a lot of values, but so far our kernel is not capable of doing anything, and having proper video output with scrolling, fonts etc, can take some time, so we need a quick way of getting some text out from our kernel, not necessarily on the screen. 
+
+This is where the serial logging came to an aid, we will use the serial port to output our text and numbers. 
+
+Many emulators has an option to redirect serial data to a file, if you are using QEmu (for more information about it refer to the Appendices section) you need to start it passing the parameter *-s filename*:
+
+```bash
+qemu -S filename.log -cdrom yourosiso
+```
+
+This will save the serial output on the file called `filename.log`, if we want the serial output directly on the screen, we can use `stdio` instead.
+
+## Printing to Serial
+
+We will use the `inb` and `outb` instruction to communicate with the serial port. But the first thing our kernel should do is do is being able to write to serial ports. To do that we need: 
+
+* for simiplicity and readability two C functions that will make use of the inb/outb asm instructions (luckily they are asm functions so making their c version is very easy)
+* initialization of serial communication
+* and at least an instruction to send characters and strings to the serial. 
+
+The first step is pretty strightforward, using inline assembly we will create two "one-line" functions for inb and outb: 
+
+```C
+extern inline unsigned char inportb (int portnum)
+{
+  unsigned char data=0;
+  __asm__ __volatile__ ("inb %%dx, %%al" : "=a" (data) : "d" (portnum));
+  return data;
+}
+
+extern inline void outportb (int portnum, unsigned char data)
+{
+  __asm__ __volatile__ ("outb %%al, %%dx" :: "a" (data),"d" (portnum));
+}
+
+```
+
+Where `portnum` is the number of port where we are sending our data (usually is 0x3f8 or 0xe9), and the data is the `char` we want to send in output. 
+
+### Initialization
+
+The second part is pretty simple, we just need to send few configuration command for initializing the serial communication, the code below is copied from https://wiki.osdev.org/Serial_Ports#Initialization:
+
+```C
+#define PORT 0x3f8          // COM1
+
+static int init_serial() {
+   outb(PORT + 1, 0x00);    // Disable all interrupts
+   outb(PORT + 3, 0x80);    // Enable DLAB (set baud rate divisor)
+   outb(PORT + 0, 0x03);    // Set divisor to 3 (lo byte) 38400 baud
+   outb(PORT + 1, 0x00);    //                  (hi byte)
+   outb(PORT + 3, 0x03);    // 8 bits, no parity, one stop bit
+   outb(PORT + 2, 0xC7);    // Enable FIFO, clear them, with 14-byte threshold
+   outb(PORT + 4, 0x0B);    // IRQs enabled, RTS/DSR set
+   outb(PORT + 4, 0x1E);    // Set in loopback mode, test the serial chip
+   outb(PORT + 0, 0xAE);    // Test serial chip (send byte 0xAE and check if serial returns same byte)
+
+   // Check if serial is faulty (i.e: not same byte as sent)
+   if(inb(PORT + 0) != 0xAE) {
+      return 1;
+   }
+
+   // If serial is not faulty set it in normal operation mode
+   // (not-loopback with IRQs enabled and OUT#1 and OUT#2 bits enabled)
+   outb(PORT + 4, 0x0F);
+   return 0;
+}
+```
+
+Notice that usually the com1 port is mapped to address: *0x3f8*. The function above is setting just default values for serial communication. An alternative that does not require any initialization is to use the port `0xe9`, this is also know as the _debugcon_ or the _port e9 hack_ and it still use the `inportb` and `outportb` functions as they are, but is often faster because is a special port that sends data directly to the emulator console output. 
+
+### Sending a string
+
+Last thing to do is to create functions to print string/numbers on the serial. The idea is pretty simple, the current functions we created are handling single bytes/char, what we want is to send strings, so a good idea is to start with a function like: 
+
+```c
+void log_to_serial (char *string) {
+    // Left as exercise
+}
+```
+
+The input parameter for this function is a string, so what it will do is looping through the variable `string` and printing each character until the symbol `\0` (End Of String) is found.
+
+This is the first function that we want to implement. 
+
+### Printing Digits
+
+Once we are able to print strings is time to print digits. The basic idea is simple, we read every single digit that compose the number, and print the corresponding character, luckily enough the digits symbols are consecutive in the ascii map, so for example: 
+
+```c
+'0' + 1 // will contain the symbol '1'
+'0' + 5 // will contain the symbol '5'
+```
+
+How to get the single digits will depend on what base we are using (the most common are base 8, 10 and 16), let's assume we want for now just print decimals (base 10). 
+
+To get decimal strings we will use a property of division by 10: _The remainder of any integer number divided by 10 is always the same as the least significant digit. _
+
+As an example consider the number 1235:  $1235/10=123.5$ and $1235 \mod 10=5$, remember that in C (and other programming languages) a division between integers will ignore any decimal digit, so this means that $1235=123$. And what if now we divide 123 by 10? yes we get 3 as remainder, below the full list of divisions for the number 1235:
+
+* $1235/10 = 123$ and $1235 \mod 10 = 5$
+* $123/10 = 12$ and $123 \mod 10 = 3$
+* $12/10 = 1$ and $12 \mod10 = 2$
+* $1/10 = 0$  and $1 \mod 10 = 1$
+
+And as you can see we got all the digits in reverse order, so now the only thing we need to do is reverse the them. The implementation of this function should be now pretty straightforward, and it will be left as exercise. 
+
+Printing other format like Hex or Octal is little bit different, but the base idea of getting the single number and converting it into a character is similar. The only tricky thing with the hex number is that now we have symbols for numbers between 10 and 15 that are characters, and they are before the digits simbol in the ascii map, but once that is know is going to be just an if statement in our function. 
+
+### Troubleshooting
+
+If the output to serial is not working, there is no output in the log, try to remove the line that set the serial as loopback: 
+
+```C
+outb(PORT + 4, 0x1E);    // Set in loopback mode, test the serial chip 
+```
+