adding tests with a Docker container; updating documentation

.gitignore

@@ -1 +1,3 @@
 playback.cfg
+*~
+

README.md

@@ -116,6 +116,64 @@ To do playbacks of all available data in a time span run:
 
 This will create all the necessary files for the playback in `data/yyyymmddHHMMSS-yyyymmddHHMMSS_N_events`.
 
+### Setting up and running the playback manually
+The `playback.sh` script is only a convenient wrapper around the two Python
+scripts `playback.py` and `make-mseed-playback.py`. Below are some notes in case
+you don't want to use the `playback.sh` script.
+
+### Setting up the sqlite3 database
+Assuming that you have a SeisComP3 compatible inventory in `inventory.xml` and
+the corresponding configuration in `configuration.xml` then you can setup an
+sqlite3 database as follows:
+
+```
+# Initialise a database file called test.db with the SeisComP3 schema
+sqlite3 -batch -init ${SEISCOMP_ROOT}/share/db/sqlite3.sql test.db .exit
+
+# Import the inventory
+scdb --plugins dbsqlite3 -d sqlite3://test.db -i inventory.xml
+
+# Import the configuration
+scdb --plugins dbsqlite3 -d sqlite3://test.db -i configuration.xml
+
+# Make a copy of the empty database; you can then easily erase the results of a
+# test run by copying the empty database file over the one containing the test
+# results   
+cp test.db test_empty.db
+```
+
+### Generating the waveform file
+Using the script `make-mseed-playback.py` you can use any record source
+supported by SeisComP3 to generate a multiplexed MiniSEED file suitable for
+playback. To see all supported record sources run:
+```
+./make-mseed-playback.py --record-driver-list
+```
+
+Assuming you have an SDS archive under `/data/sds` you could then, for example,
+request an hour long chunk of data including all stations that were active
+during that time span by running:
+
+```
+./make-mseed-playback.py  --plugins dbsqlite3 -d sqlite3://test.db --start "2017-04-01T14:59:59" --end "2017-04-01T15:59:59"
+```
+This will produce a file called `2017-04-01T14:59:59.sorted-mseed` in your
+current directory. Note that this also requires a database containing the
+inventory. Again, you can use any database type supported by SeisComP3. For a
+complete list of options run:
+```
+./make-mseed-playback.py -h
+```
+
+### Running the playback
+Once you've setup the playback files you can start the playback running:
+```
+./playback.py test.db 2017-04-01T14:59:59.sorted-mseed
+```
+For a complete list of options run:
+```
+./playback.py -h
+```
 
 ## Notes on the SC3 configuration
 
@@ -141,4 +199,5 @@ If you want to run playbacks with this particular version you have to put a
 ```
 ~/my_special_sc3/bin/seiscomp exec
 ```
-in front of every command.
+in front of every command. See also the [tests](tests/README.md) directory for instructions on
+how to use Docker for testing.

tests/Dockerfile

@@ -0,0 +1,138 @@
+FROM debian:jessie 
+
+MAINTAINER Yannik Behr <y.behr@gns.cri.nz>
+
+ENV WORK_DIR /usr/local/src/
+ENV INSTALL_DIR /opt/seiscomp3
+
+
+# Fix Debian  env
+ENV DEBIAN_FRONTEND noninteractive
+ENV INITRD No
+ENV FAKE_CHROOT 1
+
+# Setup sysop's user and group id
+ENV USER_ID 1000
+ENV GROUP_ID 1000
+
+WORKDIR $WORK_DIR
+
+RUN echo 'force-unsafe-io' | tee /etc/dpkg/dpkg.cfg.d/02apt-speedup \
+    && echo 'DPkg::Post-Invoke {"/bin/rm -f /var/cache/apt/archives/*.deb || true";};' | tee /etc/apt/apt.conf.d/no-cache \
+    && apt-get update \
+    && apt-get dist-upgrade -y --no-install-recommends \
+    && apt-get install -y \
+    build-essential \
+    festival \
+    cmake \
+    cmake-curses-gui \
+    flex \
+    g++ \
+    libgfortran3 \
+    libncurses5-dev \
+    gfortran \
+    git \
+    libboost-dev \
+    libboost-filesystem-dev \
+    libboost-iostreams-dev \
+    libboost-program-options-dev \
+    libboost-regex-dev \
+    libboost-thread-dev \
+    libboost-system-dev \
+    libboost-signals-dev \
+    python \
+    python2.7 \
+    libpython2.7 \
+    libpython2.7-dev \
+    libqt4-dev \
+    libsqlite3-dev \
+    sqlite3 \
+    python-twisted \
+    libxml2 \
+    libxml2-dev \
+    openssh-server \
+    openssl \
+    libssl-dev \
+    net-tools \
+    vim \
+    libfaketime \
+    && git clone --depth=1 https://github.com/yannikbehr/seiscomp3.git \
+    && mkdir -p $WORK_DIR/seiscomp3/build \
+    && cd $WORK_DIR/seiscomp3/build \
+    && cmake .. -DSC_GLOBAL_GUI=OFF -DSC_TRUNK_DB_MYSQL=OFF \
+       -DSC_TRUNK_DB_POSTGRESQL=OFF -DSC_TRUNK_DB_SQLITE3=ON \
+       -DCMAKE_INSTALL_PREFIX=$INSTALL_DIR \
+    && make -j $(grep -c processor /proc/cpuinfo) \
+    && make install
+
+WORKDIR /tmp
+ADD sc3-files.tar.bz .
+
+# Now install evscore plugin
+RUN cp -r evscore $WORK_DIR/seiscomp3/src/sed/plugins/ \
+    && cp CMakeLists.evscore $WORK_DIR/seiscomp3/src/sed/plugins/CMakeLists.txt \
+    && cd $WORK_DIR/seiscomp3/build \
+    && make -j $(grep -c processor /proc/cpuinfo) \
+    && make install \
+    && apt-get autoremove -y --purge \
+    && apt-get clean 
+#    && rm -rf /var/lib/apt/lists/* /var/tmp/* 
+
+# Setup ssh access
+RUN mkdir /var/run/sshd
+RUN echo 'root:password' | chpasswd
+RUN sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config
+
+# SSH login fix. Otherwise user is kicked off after login
+RUN sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' \
+    -i /etc/pam.d/sshd
+
+RUN groupadd --gid $GROUP_ID -r sysop && useradd -m -s /bin/bash --uid $USER_ID -r -g sysop sysop \
+    && echo 'sysop:sysop' | chpasswd \
+    && chown -R sysop:sysop $INSTALL_DIR
+
+RUN cp -r seiscomp3 /home/sysop/.seiscomp3 \
+    && chown -R sysop:sysop /home/sysop/.seiscomp3 
+
+USER sysop
+
+### SeisComp3 settings ###
+ENV SEISCOMP_ROOT=/opt/seiscomp3 PATH=/opt/seiscomp3/bin:$PATH \
+    LD_LIBRARY_PATH=/opt/seiscomp3/lib:$LD_LIBRARY_PATH \
+    PYTHONPATH=/opt/seiscomp3/lib/python:$PYTHONPATH \
+    MANPATH=/opt/seiscomp3/share/man:$MANPATH \
+    LC_ALL=C
+
+# Setup SeisComP3 + seedlink
+RUN seiscomp --stdin setup <sc3_config.cfg \
+    && mkdir -p /opt/seiscomp3/var/lib/seedlink \
+    && cp seedlink.ini /opt/seiscomp3/var/lib/seedlink/ \
+    && mkdir -p /opt/seiscomp3/var/run/seedlink \
+    && mkfifo -m=666 /opt/seiscomp3/var/run/seedlink/mseedfifo
+
+# Setup pipelines
+RUN seiscomp alias create NLoB_amp scamp \
+    && seiscomp alias create NLoB_apick scautopick \
+    && seiscomp alias create NLoB_auloc scautoloc \
+    && seiscomp alias create NLoB_mag scmag \
+    && seiscomp alias create NTeT_amp scamp \
+    && seiscomp alias create NTeT_apick scautopick \
+    && seiscomp alias create NTeT_auloc scautoloc \
+    && seiscomp alias create NTeT_mag scmag \
+    && seiscomp enable scevent scautopick scautoloc scmag scamp seedlink NLoB_amp \
+    NLoB_apick NLoB_auloc NLoB_mag NTeT_amp NTeT_apick NTeT_auloc NTeT_mag
+
+WORKDIR /home/sysop
+
+VOLUME ["/home/sysop/data"]
+VOLUME ["/home/sysop/sc3-playback"]
+
+USER root
+
+EXPOSE 22
+
+# Start sshd
+CMD ["/usr/sbin/sshd", "-D"]
+
+
+

tests/README.md

@@ -0,0 +1,60 @@
+# Testing
+## Setting up the docker container
+
+This directory contains a Dockerfile to generate a fully configured SeisComP3
+installation using a setup similar to the real-time system operated by the
+Swiss Seismological Service. To generate the docker image using the Dockerfile
+run:
+
+```
+docker build -t seiscomp3-ch .
+```
+
+The resulting image has two mount points, one for the directory containing the
+playback scripts and one for the waveform data and the database for the
+playback. The directory [data](data/) contains a test database file with a
+SeisComP3 configuration, inventory, and a multiplexed MiniSEED file for one
+event.
+
+Assuming that the playback script is in the directory
+`${HOME}/sc3-playback` and the playback data is under `${PWD}/data` you can
+start the container as follows:
+
+```
+docker run -d --name sc3-ch -p 9999:22 -v ${PWD}/data:/home/sysop/data -v ${HOME}/sc3-playback/:/home/sysop/sc3-playback seiscomp3-ch
+```
+
+This will also start an ssh server that you can connect to with (password the
+same as the user name):
+
+```
+ssh -p 9999 sysop@localhost
+```
+
+## Running the playback
+First get a shell in your container by running:
+```
+docker exec -it --user sysop seiscomp3-ch /bin/bash
+```
+
+Once you have a shell in your container and assuming your waveform file is
+called `test-event.sorted-mseed` and your sqlite3 database `test.db` you can
+start a playback as follows:
+```
+cd sc3-playback
+./playback.py ../data/test.db ../data/test-event.sorted-mseed
+```
+You can then use `scolv` to analyse the results from the playback. Note that you
+have to start `scolv` on your host machine as no guis are installed in the
+container.
+
+```
+scolv --offline --plugins dbsqlite3 -d sqlite3://${PWD}/data/test.db
+```
+By default the user `sysop` inside the container has `USER_ID=1000` and
+`GROUP_ID=1000`. If this doesn't match with the permissions of the playback
+files on your host machine add the following options to your `docker run`
+command:
+```
+-e USER_ID=`id -u` -e GROUP_ID=`id -g`
+```

tests/data/.gitattributes

@@ -0,0 +1,2 @@
+test.db filter=lfs diff=lfs merge=lfs -text
+test-event.sorted-mseed filter=lfs diff=lfs merge=lfs -text