Sync with upstream

.travis.yml

@@ -0,0 +1,24 @@
+---
+sudo: required
+dist: trusty
+language: python
+python:
+  - "3.3"
+  - "3.4"
+  - "3.5"
+before_install:
+  - sudo apt-get update -q
+  - sudo apt-get -y install libvirt-bin libvirt-dev
+install:
+  - pip install tox
+script:
+  - tox -e lint
+  - tox -e testxunit
+notifications:
+  irc:
+    on_success: change
+    on_failure: always
+    skip_join: true
+    channels:
+      - "irc.freenode.org#virtapi"
+  email: false

README.md

@@ -16,6 +16,7 @@ Monkeying around with virtual machines and pxe configs.
     - [HTTP PXE](#http-pxe)
     - [HTTP VM](#http-vm)
     - [HTTP installimage](#http-installimage)
+    - [HTTP DHCP](#http-dhcp)
 + [Issues](#issues)
 + [Copyright](#copyright)
 + [Name origin](#name-origin)
@@ -24,10 +25,26 @@ Monkeying around with virtual machines and pxe configs.
 
 ## Setup
 
-Create `marmoset.conf` before using marmoset! See `Configuration` for details.
+Clone the repo into /opt, then copy the service file into the systemd directory and reload systemd to recognize the file:
+```bash
+cd /opt
+git clone https://github.com/virtapi/marmoset.git
+cd marmoset
+cp ext/marmoset.service /etc/systemd/system/
+systemctl daemon-reload
+```
 Copy the `marmoset.conf.example` to `marmoset.conf` and adjust the settings to your needs.
 Checkout the Comments in the file our our [Configuration](#configuration) section.
 
+Now we need to setup a virtualenv and install the required python packages (remove libvirt from the requirements.txt and `pkg-config libvirt gcc` from the list of packages to install if you don't want to manage VMs with marmoset):
+```bash
+pacman -Syu python-virtualenv pkg-config libvirt gcc
+virtualenv prod
+source prod/bin/activate
+pip install --upgrade pip
+pip install -r requirements.txt
+```
+
 ---
 
 ## Requirements
@@ -49,6 +66,7 @@ Please checkout our [requirements.txt](requirements.txt) for a complete and auth
 * wheel
 
 In addition to these python packages, you also need Python 3. This project originally started with Python 3.3, we are currently developing and testing on 3.5 but we plan to support Python 3.3 and 3.4 as well.
+We also need pkg-config and libvirt, which you need to install via your system-wide package manager (these aren't python packages).
 
 ---
 
@@ -128,9 +146,11 @@ Start it like this:
 
     $ ./marmoset.py server
 
-Or with gunicorn:
+Or use our systemd service:
+
+    $ systemctl start marmoset
 
-    $ gunicorn marmoset.app:app
+A third solution is to use nginx + uwsgi to power the app. This is the recommended way if you expect a high amount of requests.
 
 ### HTTP PXE
 
@@ -381,7 +401,7 @@ Remove a VM:
 This endpoint is meant to work together with our [installimage](https://github.com/virtapi/installimage). We identify each dataset by its MAC address and store the key:value config pairs for the installimage.
 
 #### List Entries
-    curl -u admin:secret http://localhost:8080/v1/installimage
+    curl -u admin:secret http://localhost:5000/v1/installimage
 
 ```json
 [
@@ -414,7 +434,7 @@ This endpoint is meant to work together with our [installimage](https://github.c
 ```
 
 #### List a single Entry
-		curl -u admin:secret http://localhost:8080/v1/installimage/b8:ac:6f:97:7e:77
+		curl -u admin:secret http://localhost:5000/v1/installimage/b8:ac:6f:97:7e:77
 
 ```json
 {
@@ -434,7 +454,7 @@ This endpoint is meant to work together with our [installimage](https://github.c
 ```
 
 #### List a single Entry in the installimage format
-		curl -u admin:secret http://localhost:8080/v1/installimage/b8:ac:6f:97:7e:77/config
+		curl -u admin:secret http://localhost:5000/v1/installimage/b8:ac:6f:97:7e:77/config
 
 ```
 PART / ext4 all
@@ -449,7 +469,7 @@ HOSTNAME CentOS-71-64-minimal
 ```
 
 #### Create a record
-		curl -u admin:secret --data "drive1=/dev/sda&bootloader=grub&hostname=CentOS-71-64-minimal&PART=/ ext4 all&image=/root/.installimage/../images/CentOS-71-64-minimal.tar.gz" http://localhost:8080/v1/installimage/b8:ac:6f:97:7e:77
+		curl -u admin:secret --data "drive1=/dev/sda&bootloader=grub&hostname=CentOS-71-64-minimal&PART=/ ext4 all&image=/root/.installimage/../images/CentOS-71-64-minimal.tar.gz" http://localhost:5000/v1/installimage/b8:ac:6f:97:7e:77
 
 Returns the created record:
 ```json
@@ -470,7 +490,7 @@ Returns the created record:
 ```
 
 #### Delete a Record
-		curl -u admin:secret -X DELETE http://localhost:8080/v1/installimage/b8:ac:6f:97:7e:77
+		curl -u admin:secret -X DELETE http://localhost:5000/v1/installimage/b8:ac:6f:97:7e:77
 
 Errormessage if you want to delete or list a nonexistent entry:
 ```json
@@ -479,8 +499,156 @@ Errormessage if you want to delete or list a nonexistent entry:
 }
 ```
 
+### HTTP DHCP
+This endpoint allows us the throw static IP/MAC combinations into a ldap database. Currently tested is only the [openldap](http://www.openldap.org/) backend. This database is connected to an isc-dhcpd. We can identify an object by its IP or MAC address.
+
+#### List Entries
+		curl -u admin:secret  http://localhost:5000/v1/dhcp
+```json
+[
+    {
+        "additional_statements": {},
+        "dhcp_hostname": "odin.fritz.box",
+        "gateway": "192.168.10.1",
+        "ip_address": "192.168.10.5",
+        "mac": "00:00:00:00:00:00",
+        "networkmask": "255.255.255.0"
+    },
+    {
+        "additional_statements": {},
+        "dhcp_hostname": "odin.fritz.box",
+        "gateway": "10.3.7.1",
+        "ip_address": "10.3.7.41",
+        "mac": "00:00:00:00:00:00",
+        "networkmask": "255.255.255.0"
+    }
+]
+```
+
+#### List One Entry based on MAC
+		curl -u admin:secret  http://localhost:5000/v1/dhcp/mac/00:00:00:00:00:00
+```json
+{
+		"additional_statements": {},
+		"dhcp_hostname": "odin.fritz.box",
+		"gateway": "10.3.7.1",
+		"ip_address": "10.3.7.41",
+		"mac": "00:00:00:00:00:00",
+		"networkmask": "255.255.255.0"
+}
+```
+
+or:
+	curl -u admin:secret http://localhost:5000/v1/dhcp/mac/23.45.67.8
+```
+"please provide a valid mac address"
+```
+
+#### List one Entry based on IP
+		curl -u admin:secret  http://localhost:5000/v1/dhcp/ipv4/10.3.7.41
+```json
+{
+		"additional_statements": {},
+		"dhcp_hostname": "odin.fritz.box",
+		"gateway": "10.3.7.1",
+		"ip_address": "10.3.7.41",
+		"mac": "00:00:00:00:00:00",
+		"networkmask": "255.255.255.0"
+}
+```
+
+or:
+		curl -u admin:secret http://localhost:5000/v1/dhcp/ipv4/23.45.67.888
+```
+"please provide a valid ipv4 address"
+```
+
+#### Create a new Entry:
+this will return the new created entry:
+		curl -u admin:secret --data 'ip_address=10.3.7.41&mac=b8:ac:6f:97:7e:77&gateway=10.3.7.1&networkmask=255.255.255.0' http://localhost:5000/v1/dhcp
+```json
+{
+    "additional_statements": {},
+    "dhcp_hostname": "example.com",
+    "gateway": "10.3.7.1",
+    "ip_address": "10.3.7.41",
+    "mac": "b8:ac:6f:97:7e:77",
+    "networkmask": "255.255.255.0"
+}
+```
+
+an update works the same way, just submit the command again and change new updated params, again the API will respond with the updated entry.
+
+#### Delete an Entry:
+    curl -u admin:secret -X DELETE http://localhost:5000/v1/dhcp/ipv4/10.3.7.41
+
+Deleting a nonexistent entry:
+    curl -u admin:secret -X DELETE http://localhost:5000/v1/dhcp/ipv4/10.3.7.41
+
+will return:
+```json
+{
+  "message": "The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again. You have requested this URI [/v1/dhcp/ipv4/10.3.7.41] but did you mean /v1/dhcp/ipv4/<ipv4> ?"
+}
+```
+
 ---
 
+## Setup LDAP + isc-dhcpd
+We will do all this in a clean Arch nspawn container (login for a new container is always root without password):
+```bash
+mkdir marmoset_container
+sudo pacman -Syu arch-install-scripts
+sudo pacstrap -c -d marmoset_container
+sudo systemd-nspawn -b -D marmoset_container/
+pacman -Syu git openldap
+```
+
+Installing the isc-dhcpd is currently a bit tricky because the default package is not linked against ldap. You can get the sources and replace the PKGBUILD with [mine](https://p.bastelfreak.de/U6f5/) or use the package that I [built](https://p.bastelfreak.de/C6An/). The needed data can be found [here](https://p.bastelfreak.de/t5XS/). Please adjust the params `suffix` and `rootdn` for your needs (in /etc/openldap/slapd.conf), we use `dc=example,dc=com` and `cn=root,dc=example,dc=com` in our example.
+
+```bash
+curl -JO https://p.bastelfreak.de/C6An/
+pacman -U dhcp*.pkg.tar.xz
+cd /etc/openldap/schema
+curl -JO https://raw.githubusercontent.com/dcantrell/ldap-for-dhcp/master/dhcp.schema
+cd ..
+echo 'include /etc/openldap/schema/dhcp.schema' >> slapd.conf
+echo 'index dhcpHWAddress eq' >> slapd.conf
+echo 'index dhcpClassData eq' >> slapd.conf
+cp DB_CONFIG.example /var/lib/openldap/openldap-data/DB_CONFIG
+curl https://p.bastelfreak.de/j63 > initial_data.ldif
+systemctl start slapd
+ldapadd -x -W -D 'cn=root,dc=example,dc=com' -f initial_data.ldif -c
+```
+
+Last step, you need to add the following settings to your `/etc/dhcpd.conf` before you start the daemon:
+```
+ldap-server "localhost";
+ldap-port 389;
+ldap-username "cn=root, dc=example, dc=com";
+ldap-password "secret";
+ldap-base-dn "dc=example, dc=com";
+ldap-method dynamic;
+ldap-debug-file "/var/log/dhcp-ldap-startup.log";
+```
+
+```bash
+systemctl start dhcpd4.service
+```
+
+We also provide a prepacked nspawn container. It has a working openldap + DHCP server,
+the openldap is configured to start at boot. Systemd is so awesome that it supports downloading the tar,
+so no fiddeling with curl/wget. machinectl will throw the image into a btrfs subvol and requires you to run /var/lib/machines on btrfs:
+```bash
+pacman -Syu btrfs-progs
+modprobe loop
+machinectl --verify=no pull-tar https://bastelfreak.de/marmoset_container.tar marmoset_container
+machinectl start marmoset_container
+machinectl login marmoset_container
+```
+
+If you don't run btrfs you can still download the tar to /var/lib/machines, extract it by hand and then continue with the machinectl commands (or start it oldschool like with systemd-nspawn).
+
 ## Issues
 
 Find this code at [the git repo](https://www.github.com/virtapi/marmoset/). Find the original code at [the git repo](https://www.aibor.de/cgit/marmoset/).

ext/marmoset.service

@@ -7,3 +7,6 @@ ExecStart=/opt/marmoset/marmoset.py server
 User=marmoset
 Group=marmoset
 PrivateTmp=true
+
+[Install]
+WantedBy=multi-user.target

marmoset.conf.example

@@ -5,7 +5,9 @@
 [Modules]
 PXE: True
 VM: True
-INSTALLIMAGE: True
+Webserver: True
+Installimage: True
+DHCP: True
 
 [Webserver]
 # Settings for HTTP Basic Auth
@@ -18,7 +20,7 @@ Port: 5000
 # Domain that is used by clients to address the server. It is used to
 # build absolute urls. If unset, it defaults to 'localhost'.
 # take a look here about the syntax: http://stackoverflow.com/a/23312820
-ServerName:
+ServerName: localhost:5000
 Debug: False
 
 [Libvirt]
@@ -34,9 +36,13 @@ Network:
 # storage name is given.
 Storage:
 
+# this will be used for storing PXE options
 [PXEConfig]
 ConfigDirectory: /srv/tftp/pxelinux.cfg/
 
+# this will be used for storing installimage options
+[Config]
+ConfigDirectory: /srv/tftp/config/
 # The PXELabel section has no defaults, as its entries depend on
 # the PXE configuration. The syntax is `label: callbackmethod`.
 # If the entry doesn't need any callback, just let callbackmethod
@@ -47,3 +53,13 @@ ConfigDirectory: /srv/tftp/pxelinux.cfg/
 #freebsd: createpwhashfile
 #openbsd:
 
+[DHCPConfig]
+# list of all statements that we allow
+# see https://github.com/virtapi/marmoset/issues/7#issuecomment-175262281
+# and see https://github.com/virtapi/marmoset/issues/7#issuecomment-175275522
+additional_statements = option domain-name-servers,option domain-name,max-lease-time,default-lease-time,option ntp-servers
+ldap_server = localhost
+ldap_port = 389
+ldap_bind_dn = cn=root,dc=example,dc=com
+ldap_passwort = secret
+ldap_client_base_dn = cn=static,cn=DHCP Service Config,dc=example,dc=com

marmoset.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python3
+#!/usr/bin/env python3
 
 import marmoset
 

marmoset/__init__.py

@@ -1,6 +1,9 @@
-from . import config, cli
+from marmoset import config
+from marmoset import cli
+from marmoset import validation
 
-def run(config_file = None):
+
+def run(config_file=None):
     cfg = config.load(config_file)
     cli.parse(cfg)
 

marmoset/app.py

@@ -1,4 +1,6 @@
 from . import config, webserver
 
-app = webserver.app(config.load())
+config = config.load()
+
+app = webserver.app(config)
 

marmoset/config.py

@@ -1,11 +1,17 @@
 from os import path
-import configparser, warnings
+import configparser
+import warnings
+import socket
 
 PATH = path.join(path.dirname(__file__), '../marmoset.conf')
 
 def default():
     config = configparser.ConfigParser()
 
+    config['Common'] = dict(
+        FQDN=socket.getfqdn()
+    )
+
     config['Modules'] = dict(
         Webserver = 'True',
         PXE = 'True',

marmoset/dhcp/__init__.py

@@ -0,0 +1,2 @@
+from .dhcp_config import DhcpConfig
+from .isc_dhcp_ldap_config import ISCDhcpLdapConfig