-
Notifications
You must be signed in to change notification settings - Fork 1
Home
Software Systems, Olin College, Spring 2011
Authors: Jialiya Huang, Paul Booth, Tim Ryan
Jiauliyan OS is a bare-bones, ANSI C compliant, extensible environment for x86 machines. Unlike current operating systems which rely on rich client applications, Jiauliyan OS is built with a goal of being easily interfaced with web APIs in order to achieve the same functionality. Starting with basic OS functionality (working keyboard, serial, and vga drivers and kernel mode code) we added in an interpreted language, ANSI C-compliant Lua, and then develop higher-level functionality. In this way, we have a simple academic system which one could build upon.
We also sought to demonstrate how modern computers have once again reverted to the "dumb terminals" with the advent of the modern web. Using an emulated HTTP connection over our serial (which in the future, could be expanded to support actual hardware-level network cards) we interact with web APIs to offload computation-intensive processes or database access to web services. Thus we can perform productive work fitting a "full OS" like sending/receiving messages, email, or editing documents, without implementing much more than a JSON library. Specifically, we chose not to implement a web browser at all, instead we created a dummy server that would send parsed output to our OS over serial.
The job of the OS was then just to 1) recognize commands issued by the user, 2) send server requests for those commands, 3) handle output from the server, and 4) show them to the user. We ended up implementing our own "applications" for each service, which are elaborated upon in the features section.
Jiauliyan is an OS compiled for x86 processors ran on the QEMU emulator. QEMU is a "processor emulator that relies on dynamic binary translation", that "emulates a full computer system, including a processor and various peripherals [... and] can boot many guest operating systems". Although QEMU supports multiple architectures, our code is x86-specific since x86 is the most widely available architecture today.
We relied heavily on OS development guides to bootstrap our development, in particular, on [Brans Kernel Development Code] ^1 to help us bootstrap vga and keyboard code, the global descriptor table (GDT), and the interrupt descriptor table (IDT), interrupt requests (IRQ), and the interrupt service routine (ISR), each x86 specific. From this base we relied on information from ^2 to develop serial drivers. From this point on, most of our code was self-directed or relying on a variety of sources.
Anticipating that architectures today are more numerous and accessible than ever, Jiauliyan is designed to not rely on a specific architecture. Instead, our OS can be repurposed by implementing a small set of low-level functions for each platform, upon which the rest of our architecture is built. We provide functionality for "streams" and "bytearrays" which are not POSIX-compliant, thus allowing us to free ourselves from the shackles of 1970's programming and provide a more modern system design. Our custom libc implementation is built on this platform, as almost a compatibility layer for existing code.
In order to support Lua and provide a base level of compatibility with existing code, we reimplemented the C Standard Library using our custom library as a base. We relied heavily on existing resources for the more mundane (and less academic) portions of the library, while implementing our own virtual filesystem, stream formatting API, and memory functions as educational exercises.
Resources we relied on heavily to check our code was the BSD-licensed Rhombus OS ^3, another hobby-developed OS designed to be POSIX-compliant, but which also reimplemented libc. From this we double-checked crucial functionality, as well as borrowed printf() code and the Math library. We also relied on ^4 and ^5 to get certain parts of our string library and numerous predefined headers and constants. We based our memory manager on the definitive C reference, K&R ^6, as it presented an easy and elegant memory manager which could be implemented without too many high-level optimizations.
As stated, our OS relies on a Python HTTP server to power its communication to the web. Serial output was drop-dead simple to implement in the OS, whereas network cards have so many differing implementations that it would be hard to develop a baseline, much less a TCP/IP driver, in the amount of time provided for this class. Going forward this point could be address to add native network drivers.
Jiauliyan was compiled using GCC with all supporting code libraries disabled.
Jiauliyan initializes processor-level interrupt tables and peripherals and immediately boots into a Lua script. The relevant code directories are as follows:
- src/sys --- x86-specific code
- src/kernel --- Jiauliyan OS code, including streams API, serial/timer/VGA/keyboard code, and kernel source
- src/libc --- Jialiyan's implementation of libc
- lua-5.1/src --- Lua programming language source code
- src/scripts --- Jiauliyan's Lua code, namely "os.lua"
- server/* --- Python HTTP server that Jiauliyan relies on for web communication
We devised a few simple build scripts (in lieu of makefiles) to compile our code. clean empties the bin directory, build compiles and links all available code in the src directory, and run loads the kernel image directly into QEMU. We used an external resource to setup our build environment
sudo apt-get install qemu
We chose to use QEMU over other emulators such as Bochs because of QEMU's comparatively larger development community and it's ability to do keyboard and serial emulation. Without those two, we could not have implemented any further features.
The code in src/sys was heavily influenced by [Brans Kernel Development Code] ^1, especially start.s which included the assembly functionality for the exception mapping in the IDT, the segment register pointers for the GDT, the exception handling in the ISR, and the keyboard and serial interrupts in the IRQ. Furthermore, the tutorial also provided us a way to
write the keyboard handler that generates an IRQ.
The Lua source code is a complete package downloaded from the Lua website and redistributed here. Lua 5.1 as described by Wikipedia:
Lua is a lightweight multi-paradigm programming language designed as a scripting language with extensible semantics as a primary goal. Lua has a relatively simple C API compared to other scripting languages.
Lua is a staple scripting language for embedded systems, though it is also available as a standalone programming environment for Linux. Here, we configure Lua to compile in ANSI C mode for maximum portability, and link it to our build scripts.
Jiauliyan OS's functionality is based entirely around HTTP requests. Jiauliyan has defined message-passing between it and the Python server using a message-encoding schema. The only defined interaction here is "httpreq", which allows Lua to communicate to its serial driver a URL, HTTP method, headers, and optional body to send with a request, then synchronously block or asynchronously provide a callback for it.
In order to tweet on Identi.ca we relied on their API in order to send updates. We also implemented the JSON library in order to decode the return data to determine if the result was sucessfully tweeted or not. Since Identi.ca relies on Basic Authentication for security, we also implemented a Lua function to add the Basic Auth header to every update we send.
In order to tweet, a command like the following is issued:
send-tweet [tweet]
In order to see the timeline, a command like the following is issued:
get-tweets
The results are then parsed through the JSON library and displayed to the user.
Our OS genuinely lacked finesse with just ASCII text and a few colors, so we decided to utilize our HTTP service to add image-to-ASCII art conversion. Given a command like
image [url]
Jiauliyan OS will return an image rendered using ASCII characters and the limited 16-color character set.
[INSERT BALLER CAT IMAGE HERE]
This is done by relaying the url to a custom built web server which grabs the image, resizes it to fit in the display size of Jiauliyan OS (80x20), and sends back information about each pixel including: what character is displayed, the foreground color, and the background color.
One of the biggest challenges was the lack of debugging facilities. Due to our lack of foresight, we were unable to use gdb since we didn't build it in Jiauliyan OS. Thus, our primary method of debugging was restricted to printed error statements and selectively commenting out chunks of code.
Getting code to run on the bare hardware was an interesting challenge. There were many aspects of the Global Descriptor Table and Interrupt Descriptor Table that were complex to understand and implement, such as having to create our own GDT so that we don't get a triple fault, communicating to the registers about our new GDT, and how exceptions are handled for the IDT. It was also the first time we had to link C to assembly code. There also wasn't a way for us to test if our GDT and IDT were working, besides getting weird booting issues if they were not. We then had to deal with processor specific code in order to get VGA output, keyboard input, and serial input interrupt handlers working.
Emulating the standard C library was another hurdle we had to overcome. In order to implement Lua, we had to build out the underlying C back end that Lua relied on. We looked through the C functions that were called in the standard Lua library, and built those up by either writing our own implementations or we referred to ^4 and ^5 for more complex functions.
As an example of the sort of problems that we had to deal with, the backspace functionality ended up being solved through a combination lua and C. In C, we handle the keyboard interrupt, and we can test when a key is pressed. With the scancode, we can tell when a backspace key is pressed. Since we are using a buffer, the backspace character still goes into the buffer, which will be used when a program wants keyboard input (including lua scripts). We decided that instead of altering the buffer contents, we would add the backspace like any other character, in case something in lua wants to be able to receive backspace characters. However, in the command-line UI, a user expects the cursor to move back, and the last character to be erased. We accomplish this by moving the current pointer backwards one character by sending a backspace character, writing an empty space (which will move the cursor back to the original position), then moving back again through another backspace. This makes the UI look and feel good when a user presses backspace in Jiauliyan. In the lua code, when we interpret the keyboard input into a command that doesn't want to handle the backspaces, we recursively remove backspace and preceding character pairs so the actual command typed is returned. This gives us the best of all the worlds, being able to handle the backspace if we want, having the input feel natural, and allowing normal commands to receive the intended input.
Actual bugs: Finish ANSI C compliance, better debugging support; Add in networking functionality; we don't implement any memory protection, which should be okay as long as all code is running in Lua, but...
We did not truly develop a type of "kernel": Jiauliyan is truly single-tasking, having no scheduler, memory manager, or processor-level code execution privileges. Instead, we gained a very deep understanding of the C Standard Library, processor ports, legacy chipsets (such as keyboard controllers which for legacy reasons control all sorts of unrelated peripherals) interrupts, and what tasks an OS must handle before it even once boots user code. From this step we could integrate scheduling as part of our timer process, implement semaphores and mutexes as described in class, and add a small extension to Lua to add true preemptive multitasking^7.
More than simply implementing a custom kernel, Jiauliyan demonstrates the potential for new paradigms of computing. Web Apps have found success by requiring no installation and little overhead in order to run code. Chrome OS is an example of an operating system which takes this to a logical extreme, making the browser the sole functionality of the computer and through which all work is performed. But why stop there? Jiauliyan uses web APIs to provide "native" applications with cloud backends; it may just as well download and cache all functions of the OS from the web, downloading new Lua scripts as necessary and delivering updates by updating a remote server.
With this kind of integration, we can imagine there not being a need for "installing" libraries or applications; all functionality that is needed is requested and downloaded on demand, cached locally, and executed in a safe environment. The operating system of the future will not be a fixed binary of operations, it will be incredibly minimal and infinitely extensible.
- [The C Library Reference Guide][http://www.acm.uiuc.edu/webmonkeys/book/c_guide/] --- An excellent reference for the C Standard Library prototypes.