omx_validate.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "omx_validate",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wW4AtO3MNYH3",
        "colab_type": "text"
      },
      "source": [
        "# OMX Validator\n",
        "\n",
        "The OMX validator is a [Jupyter notebook](https://jupyter.org/) hosted in Google Colaboratory. It is an interactive Python environment that validates OMX matrices using the [openmatrix](https://github.com/osPlanning/omx-python) library.  The validator has been tested with the [example](https://github.com/osPlanning/omx/blob/master/example.omx?raw=true) omx file.  OMX files can also be inspected with the [OMX Viewer](https://github.com/osPlanning/omx/wiki/OMX-Viewer).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ojb_3G0iJdxO",
        "colab_type": "text"
      },
      "source": [
        "# Upload Files\n",
        "\n",
        "Select local files for upload via the Table of Contents + Files + Upload.  Run the code cell below to list the OMX files uploaded by clicking on the [ ] play button.  While testing, you may need to reset the UI control, which you can do via Runtime + Restart runtime.  The Files control can be accessed by clicking the File button.\n",
        "\n",
        "![upload](https://github.com/osPlanning/omx/raw/master/upload.png)\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Jx_VaZZdGYon",
        "colab_type": "code",
        "outputId": "d6f3696f-7b5f-4631-b1b7-b50b2e4ab4a8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "!ls -d *omx"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "example.omx\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IFGg33IoELVR",
        "colab_type": "text"
      },
      "source": [
        "# Validator Functions\n",
        "\n",
        "This step installs the openmatrix package from [pypi.org](https://pypi.org/project/OpenMatrix/), which includes the OMX validation functions and command line tool.  Run the code cell to install the package."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7rvtFHB8EVPz",
        "colab_type": "code",
        "outputId": "606854d5-4750-4de7-aa33-da7a7ece1528",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 173
        }
      },
      "source": [
        "!pip install openmatrix"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting openmatrix\n",
            "  Downloading https://files.pythonhosted.org/packages/05/e8/6a8227e97127a2a1db98f5a22ca565bfe942e89de0644837627920101213/OpenMatrix-0.3.5.0-py3-none-any.whl\n",
            "Requirement already satisfied: tables>=3.1.0 in /usr/local/lib/python3.6/dist-packages (from openmatrix) (3.4.4)\n",
            "Requirement already satisfied: numpy>=1.5.0 in /usr/local/lib/python3.6/dist-packages (from openmatrix) (1.18.4)\n",
            "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from tables>=3.1.0->openmatrix) (1.12.0)\n",
            "Requirement already satisfied: numexpr>=2.5.2 in /usr/local/lib/python3.6/dist-packages (from tables>=3.1.0->openmatrix) (2.7.1)\n",
            "Installing collected packages: openmatrix\n",
            "Successfully installed openmatrix-0.3.5.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2xBorP6FJi_j",
        "colab_type": "text"
      },
      "source": [
        "# Validate\n",
        "\n",
        "This section validates the OMX file against the [Specification](https://github.com/osPlanning/omx/wiki/Specification) by using the omx-validate command line tool.  Specify the file to validate in the command line call below and then run the code cell.\n",
        "\n",
        "The following checks are run and an overall Pass or Fail is returned at the end.\n",
        "1.   Has OMX_VERSION attribute set to 0.2\n",
        "2.   Has SHAPE array attribute set to two item integer array\n",
        "1.   Has data group for matrices\n",
        "1.   Matrix shape matches file shape\n",
        "1.   Uses common data types (float or int) for matrices\n",
        "1.   Matrices chunked for faster I/O\n",
        "2.   Uses zlib compression and level 1 if compression used\n",
        "2.   Has NA attribute if desired (but not required)\n",
        "2.   Has lookup group for labels/indexes if desired (but not required)\n",
        "2.   Lookup length matches shape\n",
        "1.   Uses common data types (int or str) for lookups\n",
        "2.   Has Lookup DIM attribute of 0 (row) or 1 (column) if desired (but not required)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a8VncE6qKKLC",
        "colab_type": "code",
        "outputId": "305b785c-679c-4b0a-addc-f4e54c6b107c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "!omx-validate example.omx"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "File contents: example.omx\n",
            "example.omx (File) ''\n",
            "Last modif.: 'Sun May 17 14:24:38 2020'\n",
            "Object Tree: \n",
            "/ (RootGroup) ''\n",
            "/data (Group) ''\n",
            "/data/FARE (CArray(485, 485), shuffle, zlib(1)) 'Fare Transit'\n",
            "/data/IVT (CArray(485, 485), shuffle, zlib(1)) 'In-vehicle time Transit'\n",
            "/data/IVTBUS (CArray(485, 485), shuffle, zlib(1)) 'In-vehicle time-TSys(Bus) Transit'\n",
            "/data/IVTRAIL (CArray(485, 485), shuffle, zlib(1)) 'In-vehicle time-TSys(Rail) Transit'\n",
            "/data/IVTTRAM (CArray(485, 485), shuffle, zlib(1)) 'In-vehicle time-TSys(Tram) Transit'\n",
            "/data/OWT (CArray(485, 485), shuffle, zlib(1)) 'Origin wait time Transit'\n",
            "/data/TRANSFERS (CArray(485, 485), shuffle, zlib(1)) 'Number of transfers Transit'\n",
            "/data/TWT (CArray(485, 485), shuffle, zlib(1)) 'Transfer wait time Transit'\n",
            "/lookup (Group) ''\n",
            "/lookup/NO (Array(485,)) ''\n",
            "\n",
            "\n",
            "Check 1: Has OMX_VERSION attribute set to 0.2\n",
            "  File version is 0.2: Pass\n",
            "\n",
            "Check 2: Has SHAPE array attribute set to two item integer array\n",
            "  Length is 2: Pass\n",
            "  First item is integer: Pass\n",
            "  Second item is integer: Pass\n",
            "  Shape: (485, 485)\n",
            "\n",
            "Check 3: Has data group for matrices\n",
            "  Group: Pass\n",
            "  Number of Matrices: 8\n",
            "  Matrix names: ['FARE', 'IVT', 'IVTBUS', 'IVTRAIL', 'IVTTRAM', 'OWT', 'TRANSFERS', 'TWT']\n",
            "\n",
            "Check 4: Matrix shape matches file shape\n",
            "  Matrix shape:  FARE : (485, 485) : Pass\n",
            "  Matrix shape:  IVT : (485, 485) : Pass\n",
            "  Matrix shape:  IVTBUS : (485, 485) : Pass\n",
            "  Matrix shape:  IVTRAIL : (485, 485) : Pass\n",
            "  Matrix shape:  IVTTRAM : (485, 485) : Pass\n",
            "  Matrix shape:  OWT : (485, 485) : Pass\n",
            "  Matrix shape:  TRANSFERS : (485, 485) : Pass\n",
            "  Matrix shape:  TWT : (485, 485) : Pass\n",
            "\n",
            "Check 5: Uses common data types (float or int) for matrices\n",
            "  Matrix:  FARE : float64 : Pass\n",
            "  Matrix:  IVT : float64 : Pass\n",
            "  Matrix:  IVTBUS : float64 : Pass\n",
            "  Matrix:  IVTRAIL : float64 : Pass\n",
            "  Matrix:  IVTTRAM : float64 : Pass\n",
            "  Matrix:  OWT : float64 : Pass\n",
            "  Matrix:  TRANSFERS : float64 : Pass\n",
            "  Matrix:  TWT : float64 : Pass\n",
            "\n",
            "Check 6: Matrices chunked for faster I/O\n",
            "  Matrix chunkshape:  FARE : (16, 485) : Pass\n",
            "  Matrix chunkshape:  IVT : (16, 485) : Pass\n",
            "  Matrix chunkshape:  IVTBUS : (16, 485) : Pass\n",
            "  Matrix chunkshape:  IVTRAIL : (16, 485) : Pass\n",
            "  Matrix chunkshape:  IVTTRAM : (16, 485) : Pass\n",
            "  Matrix chunkshape:  OWT : (16, 485) : Pass\n",
            "  Matrix chunkshape:  TRANSFERS : (16, 485) : Pass\n",
            "  Matrix chunkshape:  TWT : (16, 485) : Pass\n",
            "\n",
            "Check 7: Uses zlib compression if compression used\n",
            "  Matrix compression library and level:  FARE : zlib : 1 : Pass\n",
            "  Matrix compression library and level:  IVT : zlib : 1 : Pass\n",
            "  Matrix compression library and level:  IVTBUS : zlib : 1 : Pass\n",
            "  Matrix compression library and level:  IVTRAIL : zlib : 1 : Pass\n",
            "  Matrix compression library and level:  IVTTRAM : zlib : 1 : Pass\n",
            "  Matrix compression library and level:  OWT : zlib : 1 : Pass\n",
            "  Matrix compression library and level:  TRANSFERS : zlib : 1 : Pass\n",
            "  Matrix compression library and level:  TWT : zlib : 1 : Pass\n",
            "\n",
            "Check 8: Has NA attribute if desired (but not required)\n",
            "  Matrix NA attribute:  FARE : Fail\n",
            "  Matrix NA attribute:  IVT : Fail\n",
            "  Matrix NA attribute:  IVTBUS : Fail\n",
            "  Matrix NA attribute:  IVTRAIL : Fail\n",
            "  Matrix NA attribute:  IVTTRAM : Fail\n",
            "  Matrix NA attribute:  OWT : Fail\n",
            "  Matrix NA attribute:  TRANSFERS : Fail\n",
            "  Matrix NA attribute:  TWT : Fail\n",
            "\n",
            "Check 9: Has lookup group for labels/indexes if desired (but not required)\n",
            "  Group: Pass\n",
            "  Number of Lookups: 1\n",
            "  Lookups names: ['NO']\n",
            "\n",
            "Check 10: Lookup shapes are 1-d and match file shape\n",
            "  Lookup:  NO : (485,) : Pass\n",
            "\n",
            "Check 11: Uses common data types (int or str) for lookups\n",
            "  Lookup:  NO : uint32 : Pass\n",
            "\n",
            "Check 12: Has Lookup DIM attribute of 0 (row) or 1 (column) if desired (but not required)\n",
            "  Not supported at this time by the Python openmatrix package\n",
            "\n",
            "Overall result \n",
            "  Check 1 : Required : Pass\n",
            "  Check 2 : Required : Pass\n",
            "  Check 3 : Required : Pass\n",
            "  Check 4 : Required : Pass\n",
            "  Check 5 : Required : Pass\n",
            "  Check 6 : Required : Pass\n",
            "  Check 7 : Not required : Pass\n",
            "  Check 8 : Not required : Fail\n",
            "  Check 9 : Not required : Pass\n",
            "  Check 10 : Not required : Pass\n",
            "  Check 11 : Not required : Pass\n",
            "  Check 12 : Not required : Fail\n",
            "  Overall :  Pass\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}