tinyVP - tiny embedded MIPS virtualization platform, v0.96
===========================================================

  tinyVP is a very small and lean hypervisor using MIPS R5 hardware VZ option
for embedded systems (IoT in mind). It provides a complete isolation of
guest OSes from each other and hypervisor itself.

  tinyVP is a real and efficient emulating hypervisor, guest OS can run
baremetal without (I hope) any modification. Para-virtualized devices are not
needed but can be added if needed in future.

  It requires VZ option and so far runs on PIC32MZ-EF SoC based on MIPS M5150
CPU, and tested on Digilent chipKIT WiFire board, Microchip DM320007 and Olimex
PIC32-EMZ64.
  It can be adopted to another VZ-compatible core (Baikal-T1/P5600, for example)
or ported to newest MIPS R6 architecture.

  This is a reference implementation which is intentionally kept small and can
be extended and used as a base for specific embedded application requirements.
I wrote it for better understanding of MIPS CPU "VZ" option.

  Currently, tinyVP uses around 35KB code and less than 24KB RAM. It can be even
less if "printf" is removed.


Major design notes
==================

  All data in hypervisor are static, to ensure a better reliability and
security. Guest code is placed in read-only flash memory, data in SoC RAM.
All guests are isolated from other guests and root via memory management and
VZ protection.

  About 25% of code is a configuration tool ran on desktop - a couple of
Python scripts which helps allocating memory for guests, creates root level page
tables for guest virtualization, fills multiple tables to support interrupt
controller and route interrupts, assign different shared device emulators to
guest OSes and does some sanity check. Configuration is done statically,
at build build phase. The result of build is .SREC file which can be flashed
into embedded device by some means like 'pic32prog' utility.

  Embedded systems (and PIC32MZ-EF) have a small number of TLBs,
so tinyVP uses variable page sizes for efficient guests memory mapping and
configuration tool is mandatory to calculate a correct TLB pairs and still
minimize a map tree.
  Besides that, many embedded systems (and PIC32MZ-EF) have multiple device
registers blocks per page, so some effort is done to protect guest access
to incorrect devices. Access to correct devices is emulated by tinyVP for these
pages.

  There are two mandatory device emulators in tinyVP, others can be easily added.
It is a shared UART console and Interrupt Controller. Both devices can be
used by any guests simultaneously. SoC port control can be shared too but that
driver is not written yet. System configuration and clocks registers can be
write protected (actually, write is ignored and that is defined during
configuration).

  Both guest interrupt models are supported - no interrupt in guest (use WAIT)
and a normal interrupt handling with EIC interrupt controller. However,
to provide a multiple guests interrupts with VZ incompatible interrupt
controller like PIC32MZ-EF SoC, tinyVP works with non-elevated level of CPU
interrupt priority. It may cause an excessive or "stray" interrupts for guest
due to mismatch of CPU interrupt timing but it is a price of virtualization on
SoC with non-VZ compatible interrupt controller. (I think about changing it for
a single guest configuration)

  Tight memory requirement in PIC32MZ-EF SoC (only 512KB are available) enforces
yet another decision - a single hypervisor stack is used for any exception/IRQ.
Shadow register sets are used to increase interrupt handling performance and
each guest has a separate SRS.

  Root multithreading is implemented. Each thread should have a separate stack
and can be placed in some shadow register set too (or share SRS0).


Security Considerations
=======================

  tinyVP is an isolating hypervisor. It means that with right configuration
there is no ways for one guest to intervene or crash another guest or tinyVP.
However, there are some points which need to be considered:

1. Common UART console. It is emulated and each guest input/output is isolated.
But other side of communication line (terminal) can be tricked to send some
commands to tinyVP, depending on terminal. Ansi-compatible terminal can't do
that, guest input switch symbol is ASCII EOT but who knows.

2. Other sharable devices, including a shared memory. No comments here, as no
code in tinyVP.

3. Access to system-critical devices, like SoC/board configuration, clocks,
watchdog or deadman timers. Of course, if you give a guest a write access to it
then you should think about it twice.

4. tinyVP startup code doesn't setup any boot-related EJTAG restriction or access
restriction or whatever. It is done intentionally, to prevent an accident loss
of PIC32MZEF SoC due to programming error in initial development. So, security
is provided only after tinyVP starts and not against EJTAG.
   But nothing in tinyVP prevents setting it in production to prevent EJTAG-ing.


LICENSING and COPYING
=====================

Most of source code is labeled by MIT-compatible license (taken from LittleKernel)
However, branch.c was taken from Linux kernel and has GPLv2.
So, the whole product is covered by GPLv2 due to virus nature of GPL,
at least up to change of branch.c and small similar header files.
Please see file COPYING for details.


Build and run two-guests test example on Digilent chipKIT "WiFire" board
========================================================================

  Two guests configuration is written in "config.file". One guest works with
interrupt disabled and uses MIPS WAIT CPU instruction for timed delays.
Another guest has a small interrupt handler to proceed with time delay and
console IRQs. Both guests control LEDs, LD1 is for guest1 and LD2-3-4 are
for guest2, and print some symbols on console to mark a progress.
  Root also has a background process which just prints periodically a current
system time. Some input symbols into root are interpreted as some commands.

    0.  Ensure these tools are installed on the build machine.
	 * Python 2.7.8 - 2.7.X
	 * SRecord tools (srec_cat, etc.; in package "srecord" on Linux)
	 * "MIPS Linux "MTI" variant toolchain; available from:

	  https://www.mips.com/develop/tools/compilers/
	or
	  http://codescape.mips.com/components/toolchain/2017.10-05/downloads.html

    1.  Build guests ELF files

	Come to guest subdirectories and do "make" in both
	You can start a repeated build from this point if you change guest code

    2.  Extract segment maps and sizes from Guest ELF binaries, into .map files

	    make maps BOARD=DigilentChipKitWiFire CONFIG=config.file

	Actual RAM requirements are already put into config.file because size
	can't be extracted from ELF binary. But code size is reliable.

    3.  Build the single .SREC file with tinyVP + guests

	    make build BOARD=DigilentChipKitWiFire CONFIG=config.file

	You can start a repeated build from this point if you change tinyVP only


    4.  Upload the result "a.merged.srec" into Digilent chipKIT "Wi-Fire"
	board connected via USB-2-serial:

	    ./pic32prog -d /dev/ttyUSB0 a.merged.srec

	(You may need to press reset button to call a board bootloader)

    5.  Start a console on 115200 baud

	    minicom ttyUSB0

	and look the blinking LEDs and observe an output on console terminal
	from both VMs. (VM1 controls LED1 and VM2 controls LED2-3-4).
	(You may need to press reset button to call a board bootloader here too)

    6.  Press <CR> on console to clear an input UART buffer

	    <CR>

	Change an output symbol in guest1, type:

	    ^E1=

	Symbol should be changed to '=' at next guest out.
	Change the similar one in guest2 to '+':

	    ^E2+

	Return back to root input:

	    ^E0

	Also, the root can act on a simple input:

	    1s          - stop VM1
	    1c          - continue VM1
	    2r          - restart VM2
	    1q          - print VM1 status
	    T           - print TLB array
	    q           - Query, prints various VM states
	    I           - Interrupt controller printout: IRQ requests, IRQ masks and
			    virtual IRQ requests + masks
	    d           - switch ON "debug" mode
	    e           - print exception trace in debug mode
	    i           - print interrupt trace in debug mode
	    t           - print time handling trace (COUNT/COMPARE/IRQ) in debug mode

	(you need to switch to root input before commands, see above)

Note: to switch a test to another board you may need to modify and build guests
which are designed to run with Digilent board LEDs, see an appropriate
directories. tinyVP itself has no any attachment to board specific besides
using UART4 as a console, only PIC32MZ-EF SoC is needed.

Note2: You can try two another boards with:

	Microchip DM320007 (+ MEB-II aka Multimedia Extension Board II):

	    guest:  pic32mz-DM320007
	    make:   CONFIG=config-Microchip-DM320007+MEBII.file
		    BOARD=Microchip-DM320007+MEBII

	Olimex PIC32-EMZ64:

	    guest:  pic32mz-Olimex
	    make:   CONFIG=config-OlimexPic32EMZ64.file
		    BOARD=OlimexPic32EMZ64

or you can try a DMA guest example (memory-2-memory):

	Digilent chipKIT WiFire:

	    guest:  pic32mz-dma-guest
	    make:   CONFIG=config-dma.file
		    BOARD=DigilentChipKitWiFire

Contact
===================================================================

    Leonid Yegoshin aka

    lyegoshin (at gmail)