notebooks/Building Structs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A spec debugging app should be built using the same publicly available Specs that will be seen in prod. To that end, I'm using [Kaitai](https://kaitai.io/) parsers built from modified copies of the specs.\n",
    " \n",
    " - The yaml-based `.ksy` is build from public specs\n",
    " - The parsing is for position/length of fields only, \n",
    "     values remain strings (vs casting a Price to decimal and back, etc)\n",
    "    \n",
    "\n",
    "Steps:\n",
    "\n",
    " 1. Get hands on the public specs. I used [tabula](https://tabula.technology/) on the pdf you gave me to make YAMLs representing the \"Official\" PITCH specs.\n",
    " 2. Convert the specs into Kaitai-compatible `.ksy` output\n",
    " 3. Use the `.ksy` to compile js/py/etc modules that will parse the specs in various environments \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "from pathlib import Path\n",
    "\n",
    "class CONFIG:\n",
    "    rootdir = Path.cwd().parent\n",
    "    specsdir = rootdir / 'structs' / 'specs'\n",
    "    outfile = rootdir / 'structs' / 'cboe.ksy'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Map the spec yaml -> `.ksy` yaml types\n",
    "\n",
    "This is just mapping the key names and whatnot into Kaitai `.ksy` equivilents.\n",
    "\n",
    "Before:\n",
    "```yaml\n",
    "---\n",
    "- Name: Symbol Clear\n",
    "  Section: \"4.2\"\n",
    "  Description: >-\n",
    "      The Symbol Clear message instructs feed recipients to clear all orders for\n",
    "      the Cboe book in the specified symbol. This message will be sent at\n",
    "      startup each day. It would also be distributed in certain recovery events\n",
    "      such as a data center fail-over.\n",
    "  Fields:\n",
    "      - Field Name: Timestamp\n",
    "        Offset: 0\n",
    "        Length: 8\n",
    "        Data Type: Timestamp\n",
    "        Description: TimeStamp\n",
    "      - Field Name: Message Type\n",
    "        Offset: 8\n",
    "        Length: 1\n",
    "        Data Type: s\n",
    "        Description: Symbol Clear Message\n",
    "      - Field Name: Stock Symbol\n",
    "        Offset: 9\n",
    "        Length: 8\n",
    "        Data Type: Printable ASCII\n",
    "        Description: Stock symbol right padded with spaces.\n",
    "```\n",
    "\n",
    "After:\n",
    "```yaml\n",
    "symbol_clear_message:\n",
    "    doc: >-\n",
    "        The Symbol Clear message instructs feed recipients to clear all orders for\n",
    "        the Cboe book in the specified symbol. This message will be sent at\n",
    "        startup each day. It would also be distributed in certain recovery events\n",
    "        such as a data center fail-over.\n",
    "    seq:\n",
    "      - id: timestamp\n",
    "        doc: TimeStamp\n",
    "        type: block('timestamp', 0, 8)\n",
    "      - id: message_type\n",
    "        doc: Symbol Clear Message\n",
    "        type: block('alpha', 8, 1)\n",
    "      - id: stock_symbol\n",
    "        doc: Stock symbol right padded with spaces.\n",
    "        type: block('printable_ascii', 9, 8)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from boltons.strutils import slugify\n",
    "\n",
    "\n",
    "def convertspec(spec):\n",
    "    res = {}\n",
    "    res['doc'] = spec['Description']\n",
    "    res['seq'] = []\n",
    "    name = None\n",
    "    for field in spec['Fields']:\n",
    "        f = convertfield(field)\n",
    "        \n",
    "        if f['id'] == 'message_type':\n",
    "            name = (slugify(field['Description']), field['Data Type'])\n",
    "        res['seq'].append(f)\n",
    "\n",
    "    return name, res\n",
    "    \n",
    "    \n",
    "\n",
    "def convertfield(field):\n",
    "    res = {}\n",
    "    res['id'] = slugify(field['Field Name'])\n",
    "    res['doc'] = field['Description']\n",
    "    if res['id'] == 'message_type':\n",
    "        res['type'] = f\"block('alpha', {field['Offset']}, {field['Length']})\"\n",
    "    else:\n",
    "        dtype = slugify(field['Data Type'])\n",
    "        res['type'] = f\"block('{dtype}', {field['Offset']}, {field['Length']})\"\n",
    "    return res\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Yaml template that will become our `.ksy` file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "YAML_TEMPLATE = \"\"\"\n",
    "meta:\n",
    "  id: cboe\n",
    "  # ks-debug: true\n",
    "  encoding: ASCII\n",
    "  endian: le\n",
    "seq:\n",
    "  - id: records\n",
    "    type: record(_index, _io.pos)\n",
    "    # size-eos: true\n",
    "    eos-error: false\n",
    "    repeat: eos\n",
    "instances:\n",
    "  num_record_entries:\n",
    "    value: records.size\n",
    "  record_type_mask:\n",
    "    value: 0x08\n",
    "    doc: Magic number to offset the position of the msgtype char in record.\n",
    "types:\n",
    "  record:\n",
    "    params:\n",
    "      - id: idx\n",
    "        type: u4\n",
    "      - id: ofs\n",
    "        type: u4\n",
    "    seq:\n",
    "      - id: start_of_line\n",
    "        contents: \"S\"\n",
    "        doc: Magic starting char \"S\"\n",
    "      - id: raw\n",
    "        type: strz\n",
    "        terminator: 0xd\n",
    "        consume: false\n",
    "        include: false\n",
    "      - id: end_of_line\n",
    "        contents: [0xd, 0xa] # CRLF\n",
    "    instances:\n",
    "      type_indicator:\n",
    "        io: _root._io\n",
    "        pos: data_offset + _root.record_type_mask\n",
    "        size: 1\n",
    "        type: str\n",
    "      data:\n",
    "        io: _root._io\n",
    "        pos: data_offset\n",
    "        size: data_size\n",
    "        type:\n",
    "          switch-on: type_indicator\n",
    "          cases: {}\n",
    "      data_offset:\n",
    "        value: ofs + 1\n",
    "      data_size:\n",
    "        value: raw.length\n",
    "  block:\n",
    "    params:\n",
    "      - id: type\n",
    "        type: str\n",
    "      - id: offset\n",
    "        type: u2\n",
    "      - id: length\n",
    "        type: u2\n",
    "    seq:\n",
    "      - id: value\n",
    "        size: length\n",
    "        type: strz\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Main Loop\n",
    "\n",
    "The main thing here is is building the list of message type codes -> their generated type\n",
    "\n",
    "```yaml\n",
    "types:\n",
    "    record:\n",
    "        ...\n",
    "        instances: # this stuff was built in previous step\n",
    "            type_indicator:\n",
    "                io: _root._io\n",
    "                pos: data_offset + _root.record_type_mask\n",
    "                size: 1\n",
    "                type: str\n",
    "            data:\n",
    "                io: _root._io\n",
    "                pos: data_offset\n",
    "                size: data_size\n",
    "                type:\n",
    "                    switch-on: type_indicator\n",
    "                    cases:\n",
    "                        '\"B\"': trade_break_message    # <-- here\n",
    "                        '\"X\"': order_cancel_message\n",
    "                        \n",
    "    trade_break_message:\n",
    "        ...\n",
    "    order_cancel_message:\n",
    "        ...\n",
    "```             "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import lib.specs\n",
    "import lib.yaml\n",
    "\n",
    "\n",
    "def build_specs(specs, stream):\n",
    "    ksy = lib.yaml.load(YAML_TEMPLATE)\n",
    "    cases = ksy['types']['record']['instances']['data']['type']['cases']\n",
    "    ktypes = ksy['types']\n",
    "    \n",
    "    for spec in specs:\n",
    "        namecode, newspec = convertspec(spec)\n",
    "        name, code = namecode\n",
    "        cases[f'\"{code}\"'] = name # must have double quotes in string\n",
    "        ktypes[name]  = newspec\n",
    "        \n",
    "    lib.yaml.dump(ksy, stream)\n",
    "\n",
    "    \n",
    "def build_ksy_specs(base, outpath):\n",
    "    yaml_specs = list(lib.specs.read_specs_in(base))\n",
    "    with Path(outpath).open('w') as stream:\n",
    "        # should pass an io instance as stream instead\n",
    "        build_specs(yaml_specs, stream)\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "build_ksy_specs(CONFIG.specsdir.resolve(), CONFIG.outfile.resolve())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cool, now run `node compile-structs.js` from root dir to generate parsers.\n",
    "It will compile the `/structs/cboe.ksy` into `/structs/compiled/Cboe.js|cboe.py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}