Skip to content
Switch branches/tags
Go to file

Latest commit

* Add segmentedTP source file

* Rdb subcription to segmented tp logic

* Modify startup timestamp, add error catch mode

* Pub/sub logic, sub all and sub filtered modes

* Configurable update functions, closed connection logic

* Restructure segmented tp codebase

* Fix log table update

* STP unit tests

* Rename proctype, add rdb unit tests

* Rename proctype

* Add log directory env variable, fix typo

* Update test launcher

* Make stp libs stand-alone, additional commenting

* Add immediate publish mode and old API filtering

* Update unit tests

* Typo

* Add custom mode, add log handles view

* Add standard mode, log directory, additional tests

* Update autobatching mode, add total message counter

* Comments

* Format batch and multilog functions, remove log directory

* Adding custom mode, fix log handles view (#305)

* add period-only, tabular-only, and custom logging options in logname function

* add periodic and tabular as their own modes

* add stp config settings file

* Add custom mode, update meta info, update tests

* add periodic mode unit tests, update segmentedtickerplant.q, update stplog.q

* Expand custom mode, fix handles view

* Typos

Co-authored-by: robsketch <>

* Rdb log replaying, timezone configuration (#309)

* add period-only, tabular-only, and custom logging options in logname function

* add periodic and tabular as their own modes

* add stp config settings file

* Add custom mode, update meta info, update tests

* add periodic mode unit tests, update segmentedtickerplant.q, update stplog.q

* Expand custom mode, fix handles view

* Typos

* Documentation

* Add rdb log replay, configurable timezone, additional error trapping

* Replay fixes, logic to apply filters to replayed data

* Error handling for failed subscriptions

* Auto batch fixes

* Auto batch fixes

* Disk write bug fix in auto batching

* Update subscription meta

Co-authored-by: robsketch <>

* added timecheck on each upd; made upd/ts functions more extendible (#310)

* added timecheck on each upd; made upd/ts functions more extendible

* several fixes - see main PR body for detail

* fixed indentation, added explicit arg to immediate mode

* bug fixes

* remove attributes from in memory TP schemas

* removed logs added by accident

* get distinct list of handles to ensure message is only sent once

* fixed bug where openlog overwrites the log file on start up

* make time comparisons for end-of-day and end-of-period based on UTC, fixes case where TP rolls directory early

* removed modifications to eodtime.q, modified currtime and nexttime accordingly, added some debug messages

* added log messages around log open/close and period end; don't roll logs if also doing eod

* change name autobatch to memorybatch (#313)

* bug fix for currlog reference

* bug fix for using incorrect time for end of day check; improvement to not invoke checkend function un-necessarily

* add log directory back which mysteriously disappeared

* Update tickerlogreplay.q to work with STP (#315)

* add segmentedmode flag, tplogdir and tplogfile defs for stp, realupd def

* add segmentedmode config

* add expandlog function for stp log directories, more stp logic

* fix brackets

* add replaydate stp logic, append timestamp for none and tabular modes

* rundo copy of old code and update logname function as wanted

* fix date for segmentedmode

* define meta table globally, skip meta table replay

* change openlogerr to add timestamp instead of date for logfile uniformity

* remove stpmeta definition for now

* Added new parameter to .u.end (#320)

* added extra parameter to .u.end/.u.endp

* uncommented lines

* cleaning up

* added new parameter to chainedtp codebase

* added function definitions to prevent changes to standard tp behaviour

* cleaning up

* cleaning up

* endofday dict parameter (#322)

* added extra parameter to .u.end/.u.endp

* uncommented lines

* cleaning up

* added new parameter to chainedtp codebase

* added function definitions to prevent changes to standard tp behaviour

* cleaning up

* cleaning up

* moved dictionary argument logic to subscribers

* cleaning up

* cleaning up

* correcting function namespace

* tables in dictionary taken from stp

* cleaning up

* sending stp data

* Housekeeping stp (#319)

* Add functionality to create tar balls of daily stplogs

* Fix typo

* Add checkfordirectory parameter to housekeeping

* Update housekeeping documention

* Suppress unwanted output to logs

* Tidy up housekeeping

* Add function to execute system commands

* Return error instead of string representation

* Improve method of returning error

* Add error trapping, setmeta function, adjust namespaces (#318)

* add error trapping, setmeta function, adjust namespaces

* fix namespace scope issue

* add tptype to logic in subscriptions.q

* fix namespace

* add segmented logic

* combine end and endp into a single function

* add missing distinct

* change filtered publish to just use handle as it is always atomic

* add tptype to config

* bug fixes for load error message in main torq script; bug fix for log file creation and using correct variable for subscription

* refactor subscriptions library and bug fixes for STP subscription mechanism

Co-authored-by: Jonny Press <>

* Segmented tp tests (#340)

* stop timestamping while in CTP mode

* catching up to SegmentedTP

* added upd function to deal with incoming tables in CTP mode

* chained stp subscribes to stp

* sctp config and altered .u.init definition in stp

* sctp end only called from upstream processes, not timer

* removed need for sctp duplicate library, included logging boolean

* clearing up

* Fix for stp log generation
Change input for .stplg.init to dbname in all stp tests

* Add functions to handle torq processes in unit tests

* Update order.txt to include helperfunctions.q

* Add setup for stp recovery tests

* STP unit tests 1.1.1 - 1.1.5

* Update setup for stp recovery tests

* Add 2.1.1 to 2.1.3 stp unit tests

* Add unit test 1.1.6

* Tidy up subscription file test start script

* Move all STP subscription tests into a separate folder

* Tidy up script to run recovery tests

* More test script tidying

* Tidy up recovery tests setup

* Tidy up tests 2.1.1 to 2.1.3

* Add tests 2.2.1 to 2.3.3

* Add batch mode tests (4.1.x)

* Add 2.3.4 and tweaks to 1.1.3, 1.1.4

* Example folder using TorQ connection logic and repeatable structure

* Updated helper functions

* Add vanilla TP tests (2.4.x)

* Standardise folder structure and test run script names

* Delete old test runner scripts

* added tests 5.1.1 - 5.2.3

* Add tests for period end and EoD routines

* Ensure error log is rolled at EOD

* Put testing change back in

* Add files & docs for runalltests mechanic

* Test tweaks & standardising settings

* working chained stp

* Add ability to timestamp tests when they are all run

* TP log replay for old tp log format

Added tests forsingle table replay and all table replay.
A test tp log needs to be included.
Tests need to be adjusted to run sequentially in the one process.

* improved upd function tests

* clearing up upd test database file

* Add custom logging mode tests (6.3.x)

* Add immediate batch mode test (4.1.3)

* Add a test to period end suite (6.1.5)

* stp and sctp differentiated by type and small corrections

* clearing up

* Rework 1.1.x tests and add in top level test contents

* Delete old top-level tests now they are all incorporated

* clearing up default value of .stplg.i

* Add run-time timestamping to all tests script

* Make sure endofperiod function is only called once

* Settings update for the former change

* bug fix for immediate mode not throwing length errors

* test corrections for stp immediate mode

* tidying up chained stp code

* clearing up

* clearing up

* clearing up

* Improved old tp log format tests.

* Fix and re-work recovery tests

* stp and sctp log to same directory

* Grouping similar tests into one test

Also added test log for old log format

* Testing tweaks

* Add custom test logging structure and more robust k4unit logging

* Tweaks gethandles helper function

* small changes

* organising namespacing

* Add test tp log file using old format

* making comments consistent

* changes allowing sctp to grab table schemas from stp

* Add WDB tests (7.x.x)

* Tweak vanilla TP tests and add WDB validation

* Add a .z.pc call to gethandles function to ensure a new handle is opened

* Add CSV line numbers to error messages

* Incorporate run, true and fail error logging into main k4unit script

* moved .stplg.init execution, log dir only created when createlogs is true

* Add test tp logs for stp tests.

* Improve and add settings.q folder to tickerlog tests.

* Add initial versions of stp tickerlog tests.

All tests follow the same blank template. All tests currently replay different tp logs for different formats.
All tests replay full tp log, changes needed to test single table replays.

* Improve old log ticker log test

* Minor fixes and cleaning of ticker log tests code

Removed repeated use of long string and fix minor code issues.

* Add stop, quiet and runtime flags to test run scripts, attempt to make user friendly

* Add debug flag to run scripts and fix the interaction between stop mode and logging

* adding sctp tests 8.1.1 - 8.2.3

* Make everything backwards compatible and streamline run scripts

* New setup for tickerlog tests

Individual folders for tests with individual files due to issue from tickerlog replay script.

* Make runall script generic and move to root test folder

* sctp passes on endofday/period messages to subscribers

* adding tests 8.1.8 and 8.1.9

* Remove unneeded files for new format and add single tab replay folders.

* SCTP dies with STP

* restricting kill9proc search to current user procs

* adding test 8.1.7

* fixed log rolling logic and small changes

* small corrections

* added tests 8.2.4

* Add fail tests to stop mechanic, tweak error logging

* added test 8.2.2

* removing hardcoded dbname

* addition to logging logic for sctp rollover

* adding tests 8.4.1 and 8.4.3

* adding test 8.4.2

* adding pass through logging functionality

* fixing dbname definition bug

* Add TZ tests (6.5.x)

* Fix log clearup

* adding test 8.3.1

* Fix housekeeping issues for real

* small fixes

* Add housekeeping zipping test

* added comment for .z.pc

* removing vscode directory

* Fix for relative pathway for sort csv file for tickerlogreplay script

* Initial working version of housekeeping zipping test

* Change results flag to testresults

* Add a user flag to the kill proc function

* updating chainedstp and upds tests to be consistent with framework

* Performance testing initial commit

* various refactoring, added sctploggingmode config

* chained test updates

* test uses bad messages instead of bad upd function

* Rewrite feed and consumer processes, being on observer

* Basic working version of performance framework

* refactoring endof* functions, re-adding chainedtp config

* All STP performance tests working

* updated .z.pc function

* removed unused line

* updating default logging mode

* updating tests

* Fix bulk update behaviour in performance tests

* Add STP docs v1.

* Getstats function for performance tests for stp

* Fix for runcomplete performance test function

* Correct function for bulk messages and sort csv for tickerlogreplay

* Add performance stats to segmented TP doc

* Streamline tickerlog tests

* Revamp housekeeping tests

* refactoring endof* functions to remove 'overlapping arguments'

* updates to eod tests

* STP doc added details about startup

* added sctp logging mode info to docs

* extra SCTP details for docs

* Add log directory examples and formatting fixes

* Fix markdown formatting for logging examples

* Fix performance testing framework

* updates to stp endof* functions

* updates to eod tests

* Fix STP immediate mode publishing

* Add Kx tick to performance tests

* Get rid of unneeded line

* Add details for adding new logging modes

* Code for sequence number example

* small fixes

* Separate tables in stp doc

* Update performance documentation

* Add CPU/memory info to performance tests

* Add unit testing docs

* Sort out formatting issues

* Sort out formatting issues for realgs

* making error log name more specific

* Add section on sub logic to stp docs

Co-authored-by: JCarton94 <>
Co-authored-by: Andrew West <>
Co-authored-by: Michael <>
Co-authored-by: Jonny Press <>
Co-authored-by: Jamie Carton <>
Co-authored-by: Michael Potter <>

* setting tp for wdb to be stp

* Update performance & sub docs

* further sctp logging info to docs

* Update

* Update unit tests to reflect recent code changes

* moved SCTP documentation to end of file

* Add worked debug eg to unit testing docs

* Add stp spec csv

* Add endofperiod function to subscribers

* updating chainedeop tests

* Move pubsub init code to pubsub file

* updating parent logging diagram

* Update

* Add info on stpmeta table, stp restart and eod process

* Add info on stp restart logs

* alterations to chainedTP docs

* further editing to STP docs

* Update docs/

Co-authored-by: Jonny Press <>

* Update docs/

Co-authored-by: Jonny Press <>

* Update docs/

Co-authored-by: Jonny Press <>

* Update docs/

Co-authored-by: Jonny Press <>

* Update docs/

Co-authored-by: Jonny Press <>

* Update docs/

Co-authored-by: Jonny Press <>

* Update docs/

Co-authored-by: Jonny Press <>

* Update docs/

Co-authored-by: Jonny Press <>

* Update docs/

Co-authored-by: Jonny Press <>

* Update docs/

Co-authored-by: Jonny Press <>

* Modify tests to accommodate FSP changes

* Remove KDBSTPLOG env var

* Add top-level functions to metrics process

* further updates to STP docs

* Add null symbol handling for .u.sub

* Update

* Replace none logging name with Singular

* Revert testing framework changes

* further STP docs updates

* Add process name to log file names

* Fix typo

* updates to STP docs and filter function

* Update

* Make log names more descriptive, move unit testing docs to other branch

* Changes to tickerlogreplay code

Spacing for comments
Minor code changes

* add unit test for columns subscription filtering

* Remove unnecessary functional form

* Change spacing and sortcsv for settings tickerlogreplay file

* Comments for STP settings variables

* addressing comments on rdb files

* addressing comments on subscription files

* Add .z.exit, non-kdb+ process sub interface and associated unit tests

* Undo file change

* And another one

* Move some .z.pc logic into SCTP

* Quick docs formatting change

* errlog name change in docs

* correcting stp log file names in docs

* Fix getmeta bug, update unit tests with new singular mode name

* Address PR comments and add standalone pubsub unit test

* Replace common pubsub logic with new STP version (replace u.q)

* Only strip table attributes on STP

* Fix default eod metadata function

* Don't error trap testing code, set .KU.DEBUG based on command line

* Revert changes - wrong branch

* removing incorrect log from trees in STP docs

* Remove metrics process from TorQ (belong in FSP)

* Remove last testing changes

* Add pub/sub documentation

* Address initial comments on pubsub module and refactor STP main init sequence

* Address pubsub comments and update unit tests

* removing duplicate STP logs

* Add eval to pub function and tweak unit tests

* Address comments

* Remove commented out line

* Address comments on stpmeta table

Table initialised at init, also remove logs for skipping meta table when replaying

* nullifying log functionality when sctp not in create mode (#356)

* nullifying log functionality when sctp not in create mode

* .z.exit correction

* unit tests for nullifying logic

* adding examples to docs

* logging mode correction in STP docs

* Address tickerlogreplay comments and tweak unit tests

* Test HDB cleaning as well

* Address tickerlog replay comments

* Build tickerlogreplay segmented log paths, unit test error handling

* Remove weird whitespace changes

* turn .rdb.schema off for filtered subscriptions

* updating docs on filtered subscriptions to stp

* Apply suggestions from code review

Co-authored-by: Jonny Press <>

* stp password file

* Address STP and subscriptions comments

* Undo overwrites

* Add replaying of zipped files over named pipes

* Address comments

* Address OS comments

* Implement distinct log name check

* randomise fifo name with PID

* Only allow fifo replay strictly on Linux systems

Co-authored-by: JRobbins91 <>
Co-authored-by: Josh Robbins <>
Co-authored-by: robsketch <>
Co-authored-by: Jonny Press <>
Co-authored-by: Oliver Kaye <>
Co-authored-by: robsketch <>
Co-authored-by: etennoQ <>
Co-authored-by: andrew-west <>
Co-authored-by: JCarton94 <>
Co-authored-by: Andrew West <>
Co-authored-by: Jamie Carton <>
Co-authored-by: etennoQ <>

Git stats


Failed to load latest commit information.

TorQ Logo

Read our documentation

The framework forms the basis of a production kdb+ system by implementing some core functionality and utilities on top of kdb+, allowing developers to concentrate on the application business logic. It incorporates as many best practices as possible, with particular focus on performance, process management, diagnostic information, maintainability and extensibility. Wherever possible, we have tried to avoid re-inventing the wheel and instead have used contributed code from (either directly or modified). This framework will be suitable for those looking to create a new kdb+ system from scratch or those looking to add additional functionality to their existing kdb+ systems.

Have a skim through our brochure for a bit more information. The easiest way to get a production capture started is to download and install one of the Starter Packs, or read through our Github-Pages site. We also have a Google Group for questions/discussions.

Quick Start

To launch a process wrapped in the framework, you need to set the environment variables and give the process a type and name. The type and name can be explicitly passed on the command line. is an example of how to set the environment variables on a unix type system. For a windows system, see kdb+ expects all paths to be / (forward-slash) separated so all paths on all OSs should be forward-slash separated.

To avoid standard out/err being redirected, used the -debug flag

./         /- Assuming unix type OS
q torq.q -proctype test -procname mytest -debug

To load a file, use -load

q torq.q -load mytest.q -proctype test -procname mytest -debug

This will launch the a process running within the framework with all the default values. For the rest, read the document!

Updating the Documentation with Mkdocs

To make changes to the documentation website you must simply use this command while in the branch you have made the changes on:

mkdocs gh-deploy

You will be prompted to enter a username and password, after this the site should have been updated. You can test the site locally if you want using mkdocs. First use the command:

mkdocs build


mkdocs serve -a YourIp:Port

Head to the address it gives you to check if your changes have worked. More information about using mkdocs can be found here

Release Notes

  • 4.0.0, Dec 2020
    • Updated documemtation
    • Segmented Tickerplant
    • Test Framework
  • 3.8.0, Oct 2020
    • Updated documentation
    • Install script added
    • Inclusive terminology
    • DQS,DQC and DQE upgrades
    • PCAP functionality additions
  • 3.7.0, May 2020
  • 3.6.0, Apr 2020:
  • 3.5.0, Aug 2019:
    • .torq stop and .torq summary functions updated
    • .torq restart function added
    • Grafana updated
    • Monitor checks script added
    • Email disabled if trying to run on Win64
    • Opencon modified to append user:password to tcps connections
    • Bug fixes
  • 3.4.1, Jan 2019:
    • Gateway Query .gw.syncexec bug identified and fixed
  • 3.4.0, Dec 2018:
    • Update of the script to include debug and print start lines.
    • Integration of Monit for monitoring TorQ.
    • Integration of Grafana adaptor, allowing visualization of TorQ process data.
    • Added service layer for client specific customisation across all aplications.
    • Bug Fixes including:
    • Ability to specify parentproctype for processes when starting with torq.q
    • Function to check .proc.getattributes for non-TorQ processes.
    • Exit when setenv not sourced
    • qcon now allowed in debug mode
    • Hanging when null result returned from aysnc gateway request
  • 3.3.0, May 2018:
    • Upgraded to run with kdb+ v3.6. Note that it remains backwardly compatible with prior versions.
    • Implemented concurrent synchronous requests, as part of 3.6 upgrade
    • Added Response Formatter for gateway
    • Added Low Power Mode - aimed at use with Kx On Demand Licence
    • Added Initialisation Function invoked after a script is loaded and connections are established.
    • Changed the start script, to start or stop processes separately, in a batch or all at once. All the process configuration is now at one place in $KDBCONFIG/process.csv.
    • Bug fix for erroneous brackets that were causing permissions config to be loaded in incorrect order
  • 3.2.0, February 2018:
  • 3.1.0, May 2017:
    • added kafka which is a real time messaging system with persistent storage in message logs
    • added datareplay.q with functionality for generating tickerplant function calls from historical data which can be executed by subscriber functions
    • added subscribercutoff.q with functionality that can be used to cut off slow subscribers from processes
    • added new write down method for tickerlogreplay so match the writedown methods in the WDB
  • 3.0.0, January 2017:
    • Added a permissioning system, allowing granular control of access to users & groups
    • Added LDAP support, allowing a user to authenticate against an LDAP server
    • Improved documentation now available at
  • 2.7.0, November 2016:
    • Tickerplant incorporated. Tickerplant has faster recovery for real time subscribers, and better timezone handling
    • Filealerter uses a splayed table to store the table of already processed data. If it finds a table in current format (flat) it will change it to a splay
    • Gateway bug fixes
    • Small improvements
  • 2.6.2, September 2016:
    • .z.pd and peach logic added to wdb.q
    • Bug fixes
  • 2.6.0, August 2016:
    • Added broadcast publishing
    • Added domain sockets and tcps as ipc connection mechanisms
    • Added fallback from domain sockets to tcp functionality
    • New gateway (which allows attributes to be sent rather than specifying the processes to hit)
    • Heartbeat subscriptions are easier
    • Supports snappy compression
    • Tested with kdb+ 3.4
    • Bug fixes
  • 2.5.0, April 2016:
    • Added u.q - publish/subscribe code from KDB+ Tick
    • Improved memusage.q to do sampling
    • Updated dataloader.q
    • Make sure all config can be read from KDBAPPCONFIG
  • 2.4.0, February 2016:
    • Added k4unit to allow tests to be automatically run via the -test parameter
    • Extended application config handling so all default config can be found in $KDBAPPCONFIG
    • Bug Fixes
  • 2.3.0, December 2015:
    • Added optional write down method to wdb process, which allows data to be written to custom partition schemes during the day. At the end of day before being moved to the hdb, data is merged instead of sorted, which will allow the data to be accessed sooner. The optional method may present a significant saving in time for datasets with a low cardinality (small distinct number of elements), i.e. FX market data
    • Added write access control to message handlers (using reval), which restricts the ability of querying clients to modify data in place
    • Added functionality to return approximate memory size of kdb+ objects
    • Bug fixes
  • 2.2.1, November 2015:
    • Bug fix - fixed endofdaysort to not throw type error when par.txt is available
    • Bug fix - fixed gateway so results are not dropped when a client loses connection and another is querying multiple servers
  • 2.2.0, October 2015:
    • Added application configuration management using $KDBAPPCONFIG environmnet variable
    • Added pid, host and port to heartbeat table
    • Changed gateway to be non intrusive
    • Bug fixes
  • 2.1.0, July 2015:
    • Added a chained tickerplant process
    • Updated housekeeping.csv to take in an extra column agemin which represents whether to use minutes or days in find function
    • Updated email libraries
  • 2.0.1, May 2015:
    • Added RDB process which extends r.q from kdb+ tick.
    • Added WDB to write down data periodically throughout the day. Extends w.q.
    • RDB and WDB allow seamless end-of-day event (no data outage, no tickerplant back pressure)
    • Added Reporting Process to run reports periodically and process the results
    • Added environment variable resolution to process.csv to allow greater portability. If a process is started without a port specified it will look it up from process.csv based on the proctype and procname.
    • Added -localtime flag to allow process to run in localtime rather than GMT (log message, timer calls etc.). The change is backwardly compatible - without -localtime flag the process will print logs etc. in GMT but can also have a different .z.P
    • Added Subscription code to manage multiple subscriptions to different data sources
    • Added email library which uses libcurl. Used to send emails from TorQ processes
    • Added standard monitoring checks to the database code
    • Added data loader script. Utility functions to load a directory of data into a database in chunks, sort and part at the end
    • Added tickerplant log recovery utilities to recover as many messages as possible from a log file rather than just stopping at the first bad message
    • Added compression process to run and compress a given database
    • Modified compression code to handle par.txt databases
    • Modified compression code and housekeeping process to run with kdb+ 2.*
    • Modified std out/err logging and usage logging to include process name and process type (the logmsg table had changed along with some of the functions in the .lg namespace so you might need to check in case you have overridden any of them)
    • Removed launchtick scripts and some default configuration: to create a test system, install a starter pack
  • 1.2.0, Sep 2014:
    • Tested on kdb+ 3.2
    • Added connections to external (non TorQ) processes using nonprocess.csv
    • Modified file alerter with optional switch to move or not move a file if any function fails to process the file
    • Discovery service(s) host:port(s) can be passed on the command line (.servers.DISCOVERY) to a process (this should enable complete bypassing of process.csv if required)
    • Add custom hook (.servers.connectcustom) which is invoked whenever a new connection is made (allows, for example, subscription to a new process)
    • Add optional application detail file ($KDBCONFIG/application.txt) to allow customisation of the start up banner (application version etc.)
    • If required env. variables (KDBCODE, KDBCONFIG, KDBLOG) are not set they will default to $QHOME/code, $QHOME/config, $QHOME/logs respectively (previously the process failed and exited)
  • 1.1.0, Apr 2014:
    • Added compression utilities, HTML5 utilities, housekeeping process, file alerter process, kdb+tick quick start
  • 1.0.0, Feb 2014:
    • Initial public release of TorQ