Skip to content

Commit

Permalink
Initial Version of DockerFiles and restructire to support build
Browse files Browse the repository at this point in the history
  • Loading branch information
Peter Clarke authored and Peter Clarke committed Oct 18, 2020
1 parent ca27549 commit 67c3d65
Show file tree
Hide file tree
Showing 11 changed files with 115 additions and 11 deletions.
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
*.log
sym20*
./kdb-tick/sym
.DS_Store
*.env
69 changes: 58 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,65 @@
# kdb+tick
# kdb+tick docker

Files previously at code.kx.com/wsvn/kx/kdb+tick
The goal of this code is to demonstrate how to spin up an instance of a kdb+ data pipeline in a docker container.
It's currently a piece of learning work towards an easily deployable low footprint installation.

As well as taking the tick code directly, this repo also relies on the work done dockerising embedPy and uses this as a base image for the Q installation.

## Hot-linking
Forked from https://github.com/KxSystems/kdb-tick
Using a base build from https://github.com/KxSystems/embedPy

You are welcome to download and use this code according to the terms of the licence.

Kx Systems recommends you do not link your application to this repository,
which would expose your application to various risks:
# Tick changes

- This is not a high-availability hosting service
- Updates to the repo may break your application
- Code refactoring might return 404s to your application
The base of the program is an almost vanilla installation of kdb+tick.
The following changes have been made (so far): -
- time column is expected to be `timestamp` instead of `timespan`.
- Added sym.q containing trade and quote tables
- Added feed.q to publish some data

Instead, download code and subject it to the version control and regression testing
you use for your application.
In order to start up the system you can run the following commands
(TODO - TP port is currently hardcoded into the feed)

```
cd $QTICKHOME/kdb-tick
nohup q tick.q sym . -p 5010 < /dev/null > $QTICKHOME/kdb-tick/tick.log 2>&1 &
nohup q tick/r.q :5010 -p 5011 < /dev/null > $QTICKHOME/kdb-tick/rdb.log 2>&1 &
nohup q sym -p 5012 < /dev/null > $QTICKHOME/kdb-tick/hdb.log 2>&1 &
nohup q tick/feed.q < /dev/null > $QTICKHOME/kdb-tick/feed.log 2>&1 &
```

# Dockerized q

First stage in development is to get embedPy running
I created my own build locally for this using the instructions

I also created an environment file so that I could build and run the application easily.
This creates auto answers for the license questions and I've included an example here.

to run a stock docker container using the environment files, create `mytick.env` from the example file and run

```docker run --env-file mytick.env -ti kxsys/embedPy```

```
bash-3.2$ docker run --env-file mytick.env -ti kxsys/embedpy
KDB+ 3.6 2018.06.14 Copyright (C) 1993-2018 Kx Systems
l64/ 2()core 1991MB kx XXXXXXXX XXX.XXX.XXX.XXXX EXPIRE 2021.04.09 peteclarkez@XXX.com KOD #XXXXXXX
q)
```

> Note: this will create a new container each time.
# Dockerized Tick


Finally to merge the tick system and docker container, a new Dockerfile has been introduced.

This file copies the tick system to the appropriate folder & also has a shell script to run processes.

In order to run this up, first build the image and then run the container

```
docker build -t kdb-tick -f docker/Dockerfile .
docker run --env-file mytick.env -p 5010:5010 -p 5011:5011 -ti kdb-tick
```
16 changes: 16 additions & 0 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
FROM kxsys/embedpy as base

ENV Q_TICKHOME=/opt/kx/kdb-tick

RUN mkdir -p $Q_TICKHOME/tick

RUN chmod 774 $Q_TICKHOME && \
chmod 774 $Q_TICKHOME/tick && \
chown -R kx:kx $Q_TICKHOME

COPY --chown=kx docker/tick.sh $Q_TICKHOME
COPY --chown=kx kdb-tick/*.q $Q_TICKHOME
COPY --chown=kx kdb-tick/tick/*.q $Q_TICKHOME/tick

ENTRYPOINT ["/init"]
CMD ["/opt/kx/kdb-tick/tick.sh"]
13 changes: 13 additions & 0 deletions docker/tick.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
#!/bin/bash
cd /opt/kx/kdb-tick
touch /opt/kx/kdb-tick/tick.log
#Tick
nohup q tick.q sym . -p 5010 < /dev/null > /opt/kx/kdb-tick/tick.log 2>&1 &
#RDB
nohup q tick/r.q :5010 -p 5011 < /dev/null > /opt/kx/kdb-tick/rdb.log 2>&1 &
#HDB
#nohup q sym -p 5012 < /dev/null > /opt/kx/kdb-tick/hdb.log 2>&1 &
#Feedhanler
nohup q tick/feed.q < /dev/null > /opt/kx/kdb-tick/feed.log 2>&1 &

tail -f /opt/kx/kdb-tick/tick.log
18 changes: 18 additions & 0 deletions kdb-tick/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# kdb+tick

Files previously at code.kx.com/wsvn/kx/kdb+tick


## Hot-linking

You are welcome to download and use this code according to the terms of the licence.

Kx Systems recommends you do not link your application to this repository,
which would expose your application to various risks:

- This is not a high-availability hosting service
- Updates to the repo may break your application
- Code refactoring might return 404s to your application

Instead, download code and subject it to the version control and regression testing
you use for your application.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
5 changes: 5 additions & 0 deletions mytick.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# This is a comment
KDB_LICENSE_AGREE=yes
COMPANY=MyCo
NAME=Joe Smith
EMAIL=joesmith@joesmith.com

0 comments on commit 67c3d65

Please sign in to comment.