Merge upstream (test)

.editorconfig

@@ -7,6 +7,8 @@ end_of_line = lf
 charset = utf-8
 trim_trailing_whitespace = true
 insert_final_newline = true
+draw_white_space = all
+trim_trailing_white_space_on_save = true
 
 [Makefile]
 indent_style = tab

.gitignore

@@ -12,6 +12,7 @@
 *.vmdk
 *.tmp
 *.d
+*.pyc
 nbproject
 
 src/bootloader

.gitmodules

@@ -1,10 +1,10 @@
 
 [submodule "mod/rapidjson"]
 	path = mod/rapidjson
-	url = git@github.com:fwsGonzo/rapidjson.git
+	url = https://github.com/fwsGonzo/rapidjson.git
 [submodule "mod/GSL"]
 	path = mod/GSL
-	url = git@github.com:Microsoft/GSL.git
+	url = https://github.com/Microsoft/GSL.git
 [submodule "test/lest"]
 	path = test/lest
-	url = git@github.com:martinmoene/lest.git
+	url = https://github.com/martinmoene/lest.git

CONTRIBUTE.md

@@ -1,14 +1,14 @@
 # Contributing to IncludeOS
 
-Feel free to [clone, edit and send pull-request](https://help.github.com/articles/using-pull-requests). 
+Feel free to [clone, edit and send pull-request](https://help.github.com/articles/using-pull-requests).
 
-* Send any and all pull-requests to the dev-branch. It's ok if it comes from your master branch.
-* Do exactly one thing pr. pull-request. This makes it possible to quickly see and understand what you've done. 
+* Send any and all pull-requests to the [dev-branch](https://github.com/hioa-cs/IncludeOS/tree/dev). It's ok if it comes from your master branch.
+* Do "one thing" pr. pull-request. This makes it possible to quickly see and understand what you've done.
 * Please don't redo the folder-structure - if you have suggestions for this, just post an issue explaining the benefits of your suggested structure.
-* Everything you commit will be under the same license as this repo (Undecided so far) and HiOA will retain the right to publish your commits under a different license.
+* Everything you commit will be under the same license as this repo and the copyright holders will retain the right to publish your commits under a different license.
 
 ### Consider making a standalone module!
-We're working on a github-based package manager, much like [npm](https://www.npmjs.com/). Most new funcitonality from us, such as HTTP and a REST-framework, will probably come out like separate packages - each with their own repository. This will help keep the IncludeOS core small, and easier to maintain. Clearly, we also want to gather everything in one place, and our upcoming package manager will be doing that. Meanwhile: If you do want make a module - just make it a separate github-repo, and let us know about it. We'll link to it from here, until the package manager is ready.
+We're working on a github-based package manager, much like [npm](https://www.npmjs.com/). Most new functionality from us, such as HTTP and a REST-framework, will probably come out like separate packages - each with their own repository. This will help keep the IncludeOS core small, and easier to maintain. Clearly, we also want to gather everything in one place, and our upcoming package manager will be doing that. Meanwhile: If you do want make a module - just make it a separate github-repo, and let us know about it. We'll link to it from here, until the package manager is ready.
 
 ## Issue tracker
 Post any issues not already mentioned, in the [issue tracker](https://github.com/hioa-cs/IncludeOS/issues). You can also post questions, not answered in the [FAQ](https://github.com/hioa-cs/IncludeOS/wiki/FAQ).

README.md

@@ -15,26 +15,31 @@ IncludeOS is free software, with "no warranties or restrictions of any kind".
 
 
 ## Build status
-We're working towards automating everything with our Jenkins CI server. Unfortunately he was recently hacked and is being cloned from DNA recovered from a mosqito in resin plus pig stem cells as we speak. He'll be back on his feet and just like before (ish) pretty soon.
+We're working towards automating everything with our Jenkins CI server. The tests performed to generate these badges are taken from the tests folder. More tests are added regularly, so to see which tests have been completed to generate the results click the corresponding badge. 
+
+|        | Build from bundle                                                                                                                                             | Build from source |
+|--------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------|
+| Master | [![Build Status](https://jenkins.includeos.org/buildStatus/icon?job=shield_master_bundle_pipe)](https://jenkins.includeos.org/job/shield_master_bundle_pipe/) | Coming soon       |
+| Dev    | [![Build Status](https://jenkins.includeos.org/buildStatus/icon?job=shield_dev_bundle_pipe)](https://jenkins.includeos.org/job/shield_dev_bundle_pipe/)       | Coming soon       |
 
 ### Key features
 * **Extreme memory footprint**: A minimal bootable image, including bootloader, operating system components and a complete C++ standard library is currently 693K when optimized for size.
 * **KVM and VirtualBox support** with full virtualization, using [x86 hardware virtualization](https://en.wikipedia.org/wiki/X86_virtualization) whenever available (it is on most modern x86 CPU's). In principle IncludeOS should run on any x86 hardware platform, even on a physical x86 computer, given appropriate drivers. Officially, we develop for- and test on [Linux KVM](http://www.linux-kvm.org/page/Main_Page), which power the [OpenStack IaaS cloud](https://www.openstack.org/), and [VirtualBox](https://www.virtualbox.org), which means that you can run your IncludeOS service on both Linux, Microsoft Windows and Apple OS X. 
 * **C++11/14 support**
-    * Full C++11/14 language support with [clang](http://clang.llvm.org) v3.6 and later.
-    * Standard C++ library** (STL) [libc++](http://libcxx.llvm.org) from [LLVM](http://llvm.org/)
+    * Full C++11/14 language support with [clang](http://clang.llvm.org) v3.8 and later.
+    * Standard C++ library (STL) [libc++](http://libcxx.llvm.org) from [LLVM](http://llvm.org/)
     * Exceptions and stack unwinding (currently using [libgcc](https://gcc.gnu.org/onlinedocs/gccint/Libgcc.html))
     * *Note:* Certain language features, such as threads and filestreams are currently missing backend support. 
 * **Standard C library** using [newlib](https://sourceware.org/newlib/) from [Red Hat](http://www.redhat.com/)
 * **Virtio Network driver** with DMA. [Virtio](https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=virtio) provides a highly efficient and widely supported I/O virtualization. Like most implementations IncludeOS currently uses "legacy mode", but we're working towards the new [Virtio 1.0 OASIS standard](http://docs.oasis-open.org/virtio/virtio/v1.0/virtio-v1.0.html)
 * **A highly modular TCP/IP-stack** written from scratch, still under heavy development.
-    * TCP: Just enough to serve HTTP
-    * UDP: Enough to support a high performance DNS service
-    * DHCP: Basic support, tested on VirtualBox and KVM
-    * ICMP: Enough to answer ping, no control messages yet
+    * TCP: Not all the RFC's are impelemented yet, but it's stable and usable, including congestion control and common options.
+    * UDP: More or less complete, enough to support a high performance DNS service
+    * DHCP: Basic support, tested on VirtualBox, KVM and OpenStack
+    * ICMP: Enough to answer ping, no integration with the rest of the stack yet.
     * ARP
     * Ethernet
-    * IPv6 support under active development
+    * IPv6 support under development
 
 A longer list of features and limitations is on the [wiki feature list](https://github.com/hioa-cs/IncludeOS/wiki/Features)
 
@@ -65,14 +70,13 @@ You can now log in to the vagrant build environment and build and run a test ser
       $ ./test.sh
 ```
 
-
 ## Prerequisites for building IncludeOS VM's
-  * **Ubuntu 14.04 LTS x86_64**, Vanilla, either on a physical or virtual machine (A virtualbox VM works fine)
+  * **Ubuntu 16.04 or 14.04 LTS, x86_64**, either on a physical or virtual machine (A virtualbox VM works fine)
      * For the full source build, you'll need at least 1024 MB memory
-     * In order to support VGA graphics inside a VM, we recommend a lightweight GUI, such as  [lubuntu](http://lubuntu.net/blog/lubuntu-1404-trusty-tahr-released) which runs great inside a virtual machine.
+     * In order to support VGA graphics inside a VM, we recommend a lightweight GUI, such as  [lubuntu](https://help.ubuntu.com/community/Lubuntu/GetLubuntu) which runs great inside a virtual machine.
          * *NOTE:* Graphics is by no means necessary, as all IncludeOS output by default will be routed to the serial port, and in Qemu,
      * The install scripts may very well work on other flavours on Linux, but we haven't tried. Please let us know if you do.
-     * **Building on a Mac:** we have done a successful build from bundle, directly on a Mac. It's a work in progress, but see [./etc/install_osx.sh](./etc/install_osx.sh) for details.
+     * **Building on a Mac:** you can build IncludeOS (from bundle only) directly on a Mac by running [./etc/install_osx.sh](./etc/install_osx.sh).
   * You'll need `git` to clone from github.
 
 Once you have a system with the prereqs (virtual or not), you can choose a full build from source, or a fast build from binaries:
@@ -84,7 +88,7 @@ Once you have a system with the prereqs (virtual or not), you can choose a full
     $ ./etc/install_from_bundle.sh
 
 **The script will:**
-* Install the required dependencies: `curl make clang-3.6 nasm bridge-utils qemu`
+* Install the required dependencies: `curl make clang-3.8 nasm bridge-utils qemu`
 * Download the latest binary release bundle from github, using the github API.
 * Unzip the bundle to `$INCLUDEOS_INSTALL_LOC` - which you can set in advance, or which defaults to `$HOME`
 * Create a network bridge called `include0`, for tap-networking
@@ -102,7 +106,7 @@ About a minute or two (On a 4-core virtualbox Ubuntu VM, runing on a 2015 MacBoo
 
 **The script will:**
 * Install all the tools required for building IncludeOS, and all libraries it depends on:
-  * `build-essential make nasm texinfo clang-3.6 cmake ninja-build subversion zlib1g-dev libtinfo-dev`
+  * `build-essential make nasm texinfo clang-3.8 cmake ninja-build subversion zlib1g-dev libtinfo-dev`
 * Build a GCC cross compiler along the lines of the [osdev howto](http://wiki.osdev.org/GCC_Cross-Compiler) which we really only need to build `libgcc` and `newlib`.
 * Build [Redhat's newlib](https://sourceware.org/newlib/) using the cross compiler, and install it according to `./etc/build_newlib.sh`. The script will also install it to the mentioned location.
 * Build a 32-bit version of [LLVM's libc++](http://libcxx.llvm.org/) tailored for IncludeOS. 

api/async

@@ -0,0 +1,8 @@
+#pragma once
+#ifndef API_ASYNC_HEADER
+#define API_ASYNC_HEADER
+
+// hmm...
+#include "utility/async.hpp"
+
+#endif

api/common

@@ -7,9 +7,9 @@
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
 // You may obtain a copy of the License at
-// 
+//
 //     http://www.apache.org/licenses/LICENSE-2.0
-// 
+//
 // Unless required by applicable law or agreed to in writing, software
 // distributed under the License is distributed on an "AS IS" BASIS,
 // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@@ -38,25 +38,13 @@
 #error "This project needs to be compiled with an ix86-elf compiler"
 #endif
 
-#ifndef __includeOS__
-#define __includeOS__
-#endif
-
-/* Unused parameters (necessary for syscall warnings) */
+/* Unused parameters */
 #ifdef __GNUC__
 #  define UNUSED(x) UNUSED_ ## x __attribute__((__unused__))
 #else
 #  define UNUSED(x) UNUSED_ ## x
 #endif
 
-/* Hardware @todo Make this more C++'ish and proper. */
-#define MAX_NICS    4
-#define MAX_DISKS   4 
-#define MAX_SERIALS 4
-
-/* LIMITS */
-//#define SBRK_MAX 0x10000
-
 /* BOCHS Break point */
 #define BREAK __asm__ volatile("xchg %bx,%bx");
 
@@ -65,9 +53,47 @@
 #include "warn"
 #include "info"
 
-// From sanos, pdir.h
-#define PAGESHIFT      12
-#define BTOP(x) ((unsigned long)(x) >> PAGESHIFT)
-#define PAGESIZE 4096
+#include <gsl.h>
 
-#endif
+/* Define our own termination policy for contract   */
+/* violation to generate relevant information about */
+/* the reason for termination.                      */
+#if defined(OS_TERMINATE_ON_CONTRACT_VIOLATION)
+
+#include <cstdlib> // Needed for std::exit
+#include <iostream> // Needed for std::err
+#define OS_STRINGIFY(x) #x
+
+/* Function used to set termination behaviour */
+inline void expects_and_ensures_termination() { std::exit(EXIT_FAILURE); }
+
+#undef Expects
+#undef Ensures
+
+#define Expects(cond)                                       \
+  do {                                                      \
+    if (not (cond)) {                                       \
+      std::cerr << "OS: Precondition (" OS_STRINGIFY(cond)  \
+                << ") failed...\n"  << "\t...in function: " \
+                << __func__ << " @" << __FILE__ << ":"      \
+                << __LINE__ << '\n';                        \
+      std::set_terminate(expects_and_ensures_termination);  \
+      std::terminate();                                     \
+    }                                                       \
+  } while(0)
+
+#define Ensures(cond)                                       \
+  do {                                                      \
+    if (not (cond)) {                                       \
+      std::cerr << "OS: Postcondition (" OS_STRINGIFY(cond) \
+                << ") failed...\n"  << "\t...in function: " \
+                << __func__ << " @" << __FILE__ << ":"      \
+                << __LINE__ << '\n';                        \
+      std::set_terminate(expects_and_ensures_termination);  \
+      std::terminate();                                     \
+    }                                                       \
+  } while(0)
+
+#endif //< defined(OS_TERMINATE_ON_CONTRACT_VIOLATION)
+
+#endif //< INCLUDEOS_COMMON_HEADER

api/fs/common.hpp

@@ -20,16 +20,80 @@
 #define FS_COMMON_HPP
 
 #include <memory>
+#include <string>
 
 namespace fs {
 
-typedef std::shared_ptr<uint8_t> buffer_t;
+  typedef std::shared_ptr<uint8_t> buffer_t;
 
-// TODO: transform this into a class with a bool operator
-using error_t = bool;
-
-/** @var no_error: Always returns boolean false when used in expressions */
-extern error_t no_error;
+  struct error_t
+  {
+    enum token_t {
+      NO_ERR = 0,
+      E_IO, // general I/O error
+      E_MNT,
+
+      E_NOENT,
+      E_NOTDIR,
+    };
+
+    error_t(token_t tk, const std::string& rsn)
+      : token_(tk), reason_(rsn) {}
+
+    // error code to string
+    std::string token() const noexcept;
+    // show explanation for error
+    std::string reason() const noexcept {
+      return reason_;
+    }
+
+    // returns "description": "reason"
+    std::string to_string() const noexcept {
+      return token() + ": " + reason();
+    }
+
+    // returns true when it's an error
+    operator bool () const noexcept {
+      return token_ != NO_ERR;
+    }
+
+  private:
+    token_t     token_;
+    std::string reason_;
+  };
+
+  struct Buffer
+  {
+    Buffer(error_t e, buffer_t b, size_t l)
+      : err(e), buffer(b), len(l) {}
+
+    // returns true if this buffer is valid
+    bool is_valid() const noexcept {
+      return buffer != nullptr;
+    }
+    operator bool () const noexcept {
+      return is_valid();
+    }
+
+    uint8_t* data() {
+      return buffer.get();
+    }
+    size_t   size() const noexcept {
+      return len;
+    }
+
+    // create a std::string from the stored buffer and return it
+    std::string to_string() const noexcept {
+      return std::string((char*) buffer.get(), size());
+    }
+
+    error_t  err;
+    buffer_t buffer;
+    uint64_t len;
+  };
+
+  /** @var no_error: Always returns boolean false when used in expressions */
+  extern error_t no_error;
 
 } //< namespace fs