forked from KxSystems/kdb-tick
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Initial Version of DockerFiles and restructire to support build
- Loading branch information
Peter Clarke
authored and
Peter Clarke
committed
Oct 18, 2020
1 parent
ca27549
commit 67c3d65
Showing
11 changed files
with
115 additions
and
11 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
*.log | ||
sym20* | ||
./kdb-tick/sym | ||
.DS_Store | ||
*.env |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,18 +1,65 @@ | ||
# kdb+tick | ||
# kdb+tick docker | ||
|
||
Files previously at code.kx.com/wsvn/kx/kdb+tick | ||
The goal of this code is to demonstrate how to spin up an instance of a kdb+ data pipeline in a docker container. | ||
It's currently a piece of learning work towards an easily deployable low footprint installation. | ||
|
||
As well as taking the tick code directly, this repo also relies on the work done dockerising embedPy and uses this as a base image for the Q installation. | ||
|
||
## Hot-linking | ||
Forked from https://github.com/KxSystems/kdb-tick | ||
Using a base build from https://github.com/KxSystems/embedPy | ||
|
||
You are welcome to download and use this code according to the terms of the licence. | ||
|
||
Kx Systems recommends you do not link your application to this repository, | ||
which would expose your application to various risks: | ||
# Tick changes | ||
|
||
- This is not a high-availability hosting service | ||
- Updates to the repo may break your application | ||
- Code refactoring might return 404s to your application | ||
The base of the program is an almost vanilla installation of kdb+tick. | ||
The following changes have been made (so far): - | ||
- time column is expected to be `timestamp` instead of `timespan`. | ||
- Added sym.q containing trade and quote tables | ||
- Added feed.q to publish some data | ||
|
||
Instead, download code and subject it to the version control and regression testing | ||
you use for your application. | ||
In order to start up the system you can run the following commands | ||
(TODO - TP port is currently hardcoded into the feed) | ||
|
||
``` | ||
cd $QTICKHOME/kdb-tick | ||
nohup q tick.q sym . -p 5010 < /dev/null > $QTICKHOME/kdb-tick/tick.log 2>&1 & | ||
nohup q tick/r.q :5010 -p 5011 < /dev/null > $QTICKHOME/kdb-tick/rdb.log 2>&1 & | ||
nohup q sym -p 5012 < /dev/null > $QTICKHOME/kdb-tick/hdb.log 2>&1 & | ||
nohup q tick/feed.q < /dev/null > $QTICKHOME/kdb-tick/feed.log 2>&1 & | ||
``` | ||
|
||
# Dockerized q | ||
|
||
First stage in development is to get embedPy running | ||
I created my own build locally for this using the instructions | ||
|
||
I also created an environment file so that I could build and run the application easily. | ||
This creates auto answers for the license questions and I've included an example here. | ||
|
||
to run a stock docker container using the environment files, create `mytick.env` from the example file and run | ||
|
||
```docker run --env-file mytick.env -ti kxsys/embedPy``` | ||
|
||
``` | ||
bash-3.2$ docker run --env-file mytick.env -ti kxsys/embedpy | ||
KDB+ 3.6 2018.06.14 Copyright (C) 1993-2018 Kx Systems | ||
l64/ 2()core 1991MB kx XXXXXXXX XXX.XXX.XXX.XXXX EXPIRE 2021.04.09 peteclarkez@XXX.com KOD #XXXXXXX | ||
q) | ||
``` | ||
|
||
> Note: this will create a new container each time. | ||
# Dockerized Tick | ||
|
||
|
||
Finally to merge the tick system and docker container, a new Dockerfile has been introduced. | ||
|
||
This file copies the tick system to the appropriate folder & also has a shell script to run processes. | ||
|
||
In order to run this up, first build the image and then run the container | ||
|
||
``` | ||
docker build -t kdb-tick -f docker/Dockerfile . | ||
docker run --env-file mytick.env -p 5010:5010 -p 5011:5011 -ti kdb-tick | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
FROM kxsys/embedpy as base | ||
|
||
ENV Q_TICKHOME=/opt/kx/kdb-tick | ||
|
||
RUN mkdir -p $Q_TICKHOME/tick | ||
|
||
RUN chmod 774 $Q_TICKHOME && \ | ||
chmod 774 $Q_TICKHOME/tick && \ | ||
chown -R kx:kx $Q_TICKHOME | ||
|
||
COPY --chown=kx docker/tick.sh $Q_TICKHOME | ||
COPY --chown=kx kdb-tick/*.q $Q_TICKHOME | ||
COPY --chown=kx kdb-tick/tick/*.q $Q_TICKHOME/tick | ||
|
||
ENTRYPOINT ["/init"] | ||
CMD ["/opt/kx/kdb-tick/tick.sh"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
#!/bin/bash | ||
cd /opt/kx/kdb-tick | ||
touch /opt/kx/kdb-tick/tick.log | ||
#Tick | ||
nohup q tick.q sym . -p 5010 < /dev/null > /opt/kx/kdb-tick/tick.log 2>&1 & | ||
#RDB | ||
nohup q tick/r.q :5010 -p 5011 < /dev/null > /opt/kx/kdb-tick/rdb.log 2>&1 & | ||
#HDB | ||
#nohup q sym -p 5012 < /dev/null > /opt/kx/kdb-tick/hdb.log 2>&1 & | ||
#Feedhanler | ||
nohup q tick/feed.q < /dev/null > /opt/kx/kdb-tick/feed.log 2>&1 & | ||
|
||
tail -f /opt/kx/kdb-tick/tick.log |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
# kdb+tick | ||
|
||
Files previously at code.kx.com/wsvn/kx/kdb+tick | ||
|
||
|
||
## Hot-linking | ||
|
||
You are welcome to download and use this code according to the terms of the licence. | ||
|
||
Kx Systems recommends you do not link your application to this repository, | ||
which would expose your application to various risks: | ||
|
||
- This is not a high-availability hosting service | ||
- Updates to the repo may break your application | ||
- Code refactoring might return 404s to your application | ||
|
||
Instead, download code and subject it to the version control and regression testing | ||
you use for your application. |
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
# This is a comment | ||
KDB_LICENSE_AGREE=yes | ||
COMPANY=MyCo | ||
NAME=Joe Smith | ||
EMAIL=joesmith@joesmith.com |