Skip to content
Permalink
Browse files

Remote logging based on Graylog (#11)

  • Loading branch information...
frzb committed Feb 13, 2019
1 parent 6bd7013 commit 374f54e38063b9697555e47f2b94bc173e7b7775
1 .env
@@ -0,0 +1,6 @@
boot/vmlinuz
boot/initramfs
boot/undionly.kpxe
boot/ipxe.efi
bott/coinboot-initramfs*
boot/coinboot-vmlinuz*
@@ -3,27 +3,35 @@ sudo: required
language: c
cache: ccache
env:
- KERNEL=4.4.0-139-generic
- KERNEL=4.15.0-39-generic
- KERNEL=4.15.0-43-generic
- KERNEL=4.15.0-45-generic
before_install:
# always quit the build something fails with a non-zero return value
#- set -e
- sudo apt update
- sudo apt install --yes sshpass build-essential pkg-config jq qemu-system-x86 ovmf
- sudo apt install --yes sshpass build-essential pkg-config jq qemu-system-x86 ovmf iputils-ping
- CURL='curl --max-time 5 --retry-max-time 20 --retry 999'
- RESPONSE=$($CURL --silent "https://api.github.com/repos/frzb/coinboot-debirf/tags")
- sleep 5
- >
while [[ -z $RELEASE ]]; do
export RELEASE=$(curl --connect-timeout 5 --max-time 5 --silent "https://api.github.com/repos/frzb/coinboot-debirf/releases/latest" | jq -r '.tag_name')
sleep 3
echo $RELEASE
while ! RELEASE=$(echo $RESPONSE | jq -r '.[0].name'); do
echo "Calling the Github API has failed, repeat ..."
RESPONSE=$(curl --silent "https://api.github.com/repos/frzb/coinboot-debirf/tags")
sleep 5
done
- echo "Latest release is $RELEASE"
- echo $PATH
- sudo mkdir -p /etc/qemu
- echo 'allow all' | sudo tee /etc/qemu/bridge.conf
script:
- sudo docker-compose -f docker-compose_travis-ci.yml up -d
- sudo docker-compose exec coinboot env
- sudo docker-compose ps
#- sudo docker-compose logs -f coinboot
- while ! [ $(curl --silent http://192.168.1.2 | jq length) -gt 4 ]; do curl --silent http://192.168.1.2 | jq '.[].name'; echo '------------'; sleep 10; done
#- sudo docker-compose logs -f coinboot
# This line can be used for debugging Qemu iPXE purposes.
#- sudo qemu-system-x86_64 -m 2048 -smp 2 -nographic -boot n -net nic,model=e1000 -net bridge,br=$(ip a|grep 192.168.1.1/24 | grep -oP br-.*)
#- sudo qemu-system-x86_64 -m 2048 -smp 2 -nographic -serial mon:stdio -boot n -net nic,model=e1000 -net bridge,br=$(ip a|grep 192.168.1.1/24 | grep -oP br-.*)
# Boot with legacy BIOS
- >
sudo qemu-system-x86_64
@@ -34,6 +42,8 @@ script:
-boot n
-net nic,model=e1000
-net bridge,br=$(ip a|grep 192.168.1.1/24 | grep -oP br-.*)
- while ! ping -c 1 192.168.1.10; do echo 'Waiting for Coinboot machine to respond to our ICMP echo requests ...'; sleep 10; done
- ping -c 10 192.168.1.10
- while ! nc -z 192.168.1.10 22; do echo 'Waiting for Coinboot machine to listen on port 22/SSH ...'; sleep 5; done
- sshpass -p ubuntu ssh -v -o StrictHostKeyChecking=no ubuntu@192.168.1.10 "grep -C 10 ${RELEASE} /etc/motd && uname -a"
- sudo killall qemu-system-x86_64
@@ -51,5 +61,7 @@ script:
-net nic,model=e1000
-net bridge,br=$(ip a|grep 192.168.1.1/24 | grep -oP br-.*)
-bios /usr/share/OVMF/OVMF_CODE.fd
- while ! ping -c 1 192.168.1.10; do echo 'Waiting for Coinboot machine to respond to our ICMP echo requests ...'; sleep 10; done
- ping -c 10 192.168.1.10
- while ! nc -z 192.168.1.10 22; do echo 'Waiting for Coinboot machine to listen on port 22/SSH ...'; sleep 10; done
- sshpass -p ubuntu ssh -v -o StrictHostKeyChecking=no ubuntu@192.168.1.10 "grep -C 10 ${RELEASE} /etc/motd && ls -la /sys/firmware/efi && uname -a"
@@ -1,4 +1,4 @@
![Logo of Coinboot](coinboot.png)
![Logo of Coinboot](img/coinboot.png)


## Coinboot [![Build Status](https://travis-ci.com/frzb/coinboot.svg?branch=master)](https://travis-ci.com/frzb/coinboot)
@@ -30,6 +30,11 @@ Its core features are:
Need to expand your machines with further configuration, software, libraries, proprietary drivers?
By packing them as Coinboot plugin you can use them right after your machines have booted.

* **Insights out of the box**

Coinboot comes with Graylog integrated for log managment.
Providing the log files of your worker nodes at a glance.

This repository contains the Coinboot Server Docker container.
This container includes all services to get Coinboot up and running and boot diskless Coinboot Worker nodes over network.

@@ -57,7 +62,16 @@ You can hand over environment variables to the worker nodes booting with Coinboo
This way you can keep the configuration of your Coinboot Worker nodes at one point.
Just put these variables in a file in the directory `./conf/environment/`.
These variables are added to `/etc/environment` on the worker nodes during boot and are exported and available for login shells on these nodes.
If these variables are not exported and available, e.g. in Systemd units, just source the file `/etc/environment` to make them available.
If these variables are not exported and available, e.g. in Systemd units, just source the file `/etc/environment` to make them available.

There are also mandatory environment variables which are required to be configured.

#### Mandatory environment variables

| Variable | Default | Description |
| -------------------- |:-------------:| -----------------------------------------------------------------------|
| `COINBOOT_SERVER_IP` | `192.168.1.2` | IP address at which the services of the Coinboot server should listen. |


#### RootFS and Kernel

@@ -93,17 +107,9 @@ For example the Docker host has assigned `192.168.1.2` then a matching DHCP-rang

Also verify that the network adapter you assigned this IP address on your Docker host is connected to the same L2/broadcast domain as the machines you want to boot with Coinboot.

#### Environment variables

You can hand over environment variables to the machines booting with Coinboot.
This is the way to keep the configuration for your machines at one point.
Just put these variables in a file in the directory `./conf/environment/`.
These varibales are added to `/etc/environment` on your machines and are exported and available for login shells.
If these variables are no exported and available, e.g. in Systemd units, just source the file `/etc/environment` to make them available.

### Start the Coinboot Server Docker container

Just bring the Coinboot Server Docker container up with `docker-compose`.
Just bring the Coinboot Server Docker and Graylog containers up with `docker-compose`.

```
$ docker-compose up -d
@@ -122,13 +128,26 @@ Please change the password via creating a Coinboot Plugin.

### Logfiles

To see what's currently going on you can look at the logfiles of the Coinboot Docer container.
To see what's currently going on you can look at the logfiles of the Coinboot Docker container.
For instance to see the DHCP lease hand-shakes happen or what plugins are delivered.

```
$ docker-compose logs -f
$ docker-compose logs -f coinboot
```

### Centralized log managment with Graylog

Coinboot comes with Graylog as centralized log management collecting iPXE bootloader and Kernel message of all your worker nodes.

![Screenshot of Graylog](img/graylog.png)

Login with your web browser at: http://<your-Docker-host-IP:9000>`

* login `admin`

* password: `admin`


## Test and development environment

There is Vagrant environment for developing purposes.
@@ -142,27 +161,47 @@ $ vagrant up

## Pack your own Coinboot plugins

A Coinboot plugin is the way to go to extend the functionality of machines that boot with Coinboot.
A Coinboot plugin is the way to go to extend the functionality of machines that boot with Coinboot.
Basically a Coinboot plugin is just set of file system changes that is applied at boot time.

Clone the https://github.com/frzb/coinboot-plugins repository to get `coinbootmaker`.

```
$ git clone git@github.com:frzb/coinboot-plugins.git
```
### `coinbootmaker`

```
Usage: coinbootmaker [-i] -p <file name> <path to initramfs>
Basically a Coinboot plugin is just set of file system changes that is applied at boot time.
-i Interactive mode - opens a shell in the build environment
-p <file name> Plugin to build
-h Display this help
```

### Example

All you need to create your own plugins is:
Run `coinbootmaker` interactivly (`-i`)

```
$ ./coinbootmaker -i /tmp/coinboot-initramfs-4.15.0-43-generic
```

* Boot the Coinboot base image
* You are entering the build environment

* Execute `$ create_plugin start`
* Execute `$ create_plugin.py start `

* Do your changes to the system - e.g. install packages and edit configuration files.

* When your are done: Execute `$ create_plugin finish <name-of-your-plugin>`
* When your are done: Execute `$ create_plugin.py finish <name-of-your-plugin>`

* Place the created plugin archive into `./plugins` on the host where you run the Coinboot Docker container

Up on the next boot the changes your made in your plugin are ready to be used on your Coinboot machines!

Creation of plugins can also be scripted. Just do whatever you want to do between the lines `$ create_plugin start` and `$ create_plugin finish <name-of-your-plugin>`.
Creation of plugins can also be scripted. Just do whatever you want to do between the lines `$ create_plugin.py start` and `$ create_plugin.py finish <name-of-your-plugin>`.

For more details about creating plugins and example plugins please refer to https://github.com/frzb/coinboot-plugins .
## License

GNU GPLv3
@@ -1,10 +1,15 @@
#!ipxe
# Remote logging server is with the DHCP-server
set syslog ${dhcp-server}

show syslog

set base-url http://${dhcp-server}

echo Welcome to Coinboot at ${hostname}.
echo Welcome to Coinboot at ${ip}.
echo Logs are send to ${dhcp-server} on port 514/UDP.
echo Proceeding with booting right now...
kernel ${base-url}/vmlinuz coinboot-server=${base-url} initrd=initramfs console=ttyS0 console=tty0 ignore_loglevel net.ifnames=0 biosdevname=0
kernel ${base-url}/vmlinuz coinboot-server=${base-url} initrd=initramfs ignore_loglevel netconsole=5555@${ip}/eth0,5555@${dhcp-server} console=tty0 net.ifnames=0 biosdevname=0
initrd ${base-url}/initramfs
boot ||
# If everything failed, give the user some options.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.
@@ -1,6 +1,8 @@
# These environment variables are just for demonstration.
# Mandatatory environment variables.
COINBOOT_SERVER_IP=192.168.1.2
# These environment variables below are just for demonstration.
# Replace them with your own one.
RELEASE=latest
PURPOSE=test
COMPANY=ACME
ETHEREUM_ADDRESS=<add your Ethtereum ID here>
ETHEREUM_ADDRESS=<add your Ethereum ID here>
@@ -15,3 +15,51 @@ services:
- NET_ADMIN
- NET_BROADCAST
network_mode: "host"
mongodb:
image: mongo:4.0.6
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.6.0
environment:
- http.host=0.0.0.0
- transport.host=localhost
- network.host=0.0.0.0
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
# Graylog: https://hub.docker.com/r/graylog/graylog/
graylog:
image: graylog/graylog:2.5
volumes:
- ./conf/graylog/contentpacks:/usr/share/graylog/data/contentpacks
env_file:
- ./conf/environment/default.env
environment:
- GRAYLOG_CONTENT_PACKS_AUTO_LOAD=coinboot-remote-logging.json
- GRAYLOG_CONTENT_PACKS_DIR=data/contentpacks
- GRAYLOG_CONTENT_PACKS_LOADER_ENABLED=true
# CHANGE ME!
- GRAYLOG_PASSWORD_SECRET=somepasswordpepper
# Password: admin
- GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
- GRAYLOG_WEB_ENDPOINT_URI=http://${COINBOOT_SERVER_IP}:9000/api
links:
- mongodb:mongo
- elasticsearch
depends_on:
- mongodb
- elasticsearch
ports:
# Graylog web interface and REST API
- 9000:9000
# Syslog TCP
- 514:514
# Syslog UDP
- 514:514/udp
# Netconsole raw UDP
- 5555:5555/udp
# GELF TCP
- 12201:12201
# GELF UDP
- 12201:12201/udp
@@ -8,7 +8,6 @@ networks:
config:
-
subnet: 192.168.1.0/24

services:
coinboot:
container_name: coinboot
@@ -28,3 +27,51 @@ services:
networks:
coinboot:
ipv4_address: 192.168.1.2
mongodb:
image: mongo:4.0.6
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.6.0
environment:
- http.host=0.0.0.0
- transport.host=localhost
- network.host=0.0.0.0
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
# Graylog: https://hub.docker.com/r/graylog/graylog/
graylog:
image: graylog/graylog:2.5
volumes:
- ./conf/graylog/contentpacks:/usr/share/graylog/data/contentpacks
env_file:
- ./conf/environment/default.env
environment:
- GRAYLOG_CONTENT_PACKS_AUTO_LOAD=coinboot-remote-logging.json
- GRAYLOG_CONTENT_PACKS_DIR=data/contentpacks
- GRAYLOG_CONTENT_PACKS_LOADER_ENABLED=true
# CHANGE ME!
- GRAYLOG_PASSWORD_SECRET=somepasswordpepper
# Password: admin
- GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
- GRAYLOG_WEB_ENDPOINT_URI=http://${COINBOOT_SERVER_IP}:9000/api
links:
- mongodb:mongo
- elasticsearch
depends_on:
- mongodb
- elasticsearch
ports:
# Graylog web interface and REST API
- 9000:9000
# Syslog TCP
- 514:514
# Syslog UDP
- 514:514/udp
# Netconsole raw UDP
- 5555:5555/udp
# GELF TCP
- 12201:12201
# GELF UDP
- 12201:12201/udp
@@ -1,19 +1,15 @@
FROM alpine:3.8

#TODO add syslinux/pxelinux files
RUN apk --no-cache add nginx supervisor dnsmasq jq curl openssl ca-certificates && update-ca-certificates
RUN apk --no-cache add nginx supervisor dnsmasq jq curl wget openssl ca-certificates && update-ca-certificates

RUN mkdir -p /run/nginx /var/lib/tftpboot \
etc/dnsmasq.d /etc/supervisor /srv/plugins /opt/ipxe \
&& wget http://boot.ipxe.org/ipxe.efi -P /opt/ipxe \
&& wget http://boot.ipxe.org/undionly.kpxe -P /opt/ipxe \
&& ln -vfs /opt/ipxe/ipxe.efi /var/lib/tftpboot/ipxe.efi \
&& ln -vfs /opt/ipxe/undionly.kpxe /var/lib/tftpboot/undionly.kpxe
RUN mkdir -p /run/nginx /var/lib/tftpboot /etc/dnsmasq.d /etc/supervisor /srv/plugins

COPY ./dnsmasq/dnsmasq.conf /etc/dnsmasq.conf
COPY ./nginx /etc/nginx/conf.d
COPY ./supervisor /etc/supervisor
COPY ./coinboot-download-helper /usr/local/bin/coinboot-download-helper
COPY ./graylog-contentpack-helper /usr/local/bin/graylog-contentpack-helper

CMD /usr/bin/supervisord -c /etc/supervisor/coinboot.ini

Oops, something went wrong.

0 comments on commit 374f54e

Please sign in to comment.
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.