KNEX

Latest Release (v0.5.0)

Python library for creating chainable data transformers.

Installation

pip install knex

Supported Transformers

❌ - Planned

✔️ - Implemented

General	String	Number	Date	Other
✔️ Append	✔️ Base64Decode	❌ Absolute	❌ BetweenDates	❌ AsnToInt
✔️ Count	✔️ Base64Encode	❌ Add	❌ BetweenHours	❌ DecodeCiscoType7
✔️ FirstElement	✔️ Concat	❌ Ceil	❌ DateStringToISOFormat	❌ EncodeCiscoType7
✔️ GetField	❌ Cut	❌ Divide	❌ DateToEpoch	❌ EncryptCiscoType5
✔️ GetIndex	❌ DumpJSON	❌ Floor	❌ DateToString	❌ FuzzyWuzzyFind
❌ If-Then-Else	❌ FromString	❌ Modulus	❌ FormattedDateToEpoch	❌ InterfaceRangeExpand
✔️ IndexOf	❌ GenieParse	❌ Multiply	❌ ModifyDateTime	❌ IpProtocolNameToNumber
✔️ Join	❌ JSONUnescape	❌ Round	❌ TimeStampToDate	❌ IpProtocolNumberToName
❌ Jq	❌ Jinja2Render	❌ Subtract		❌ JsonSchemaValidate
❌ JsonToTable	✔️ Length	❌ SumList		✔️ MacAddress
✔️ LastElement	❌ LoadJSON	❌ ToPercent		❌ NormalInterfaceName
❌ ReverseList	✔️ RegexExtractAll			❌ PythonFunction
❌ SetIfEmpty	❌ RegexReplace			❌ Random
❌ Slice	✔️ Split			❌ SortInterfaceList
❌ Sort	❌ Substring			✔️ TextFSMParse
❌ Stringify	❌ TemplateRender			❌ ThisOrThat
❌ Unique	✔️ ToLower			❌ ToCamelCase
❌ WhereFieldEquals	❌ ToString			❌ ToSnakeCase
	✔️ ToUpper			❌ Ttp
	❌ Trim			❌ URLDecode
	❌ YamlDumps			❌ URLEncode
	❌ YamlLoads			❌ ValidateCiscoType5
				❌ ValidateCiscoType7
				❌ VlanConfigToList
				❌ YamaleValidate

Usage

>>> from knex.parsers import *
>>>
>>> input_data = """
... Interface             IP-Address      OK?    Method Status          Protocol
... GigabitEthernet0/1    unassigned      YES    unset  up              up
... GigabitEthernet0/2    192.168.190.235 YES    unset  up              up
... GigabitEthernet0/3    unassigned      YES    unset  up              up
... GigabitEthernet0/4    192.168.191.2   YES    unset  up              up
... TenGigabitEthernet2/1 unassigned      YES    unset  up              up
... Virtual36             unassigned      YES    unset  up              up
... """
>>>
>>> pattern = r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b"
>>>
>>> end = (
                Start(input_data)
                > RegexExtractAll(pattern)
                > GetIndex(0)
                > Concat("", "/24")
                > IpNetwork()
             )
>>>
>>> print(end.result)
192.168.190.0/24
>>> print(json.dumps(end.history, indent=4))
[
    {
        "parser": "RegexExtractAll",
        "input": "...omitted for brevity...",
        "args": {
            "pattern": "\\b\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\b"
        },
        "error": false,
        "output": [
            "192.168.190.235",
            "192.168.191.2"
        ]
    },
    {
        "parser": "GetIndex",
        "input": [
            "192.168.190.235",
            "192.168.191.2"
        ],
        "args": {
            "idx": 0
        },
        "error": false,
        "output": "192.168.190.235"
    },
    {
        "parser": "Concat",
        "input": "192.168.190.235",
        "args": {
            "prefix": "",
            "suffix": "/24"
        },
        "error": false,
        "output": "192.168.190.235/24"
    },
    {
        "parser": "IpNetwork",
        "input": "192.168.190.235/24",
        "args": {},
        "error": false,
        "output": "192.168.190.0/24"
    }
]
>>>

Development

Environment Setup

1. Install Poetry
2. Clone the repo: git clone https://github.com/clay584/knex && cd knex
3. Install pre-requisits for developement: poetry install
4. Activate the environment: poetry shell
5. Install git pre-commit hook: pre-commit install && pre-commit autoupdate

Making Changes

1. Run tests and validate coverage: pytest -v --cov=knex --cov-report html tests
   • The HTML coverage report will be located at ./htmlcov/index.html.
2. Commit all changes, and have clean git repo on main branch.
3. Bump version: bump2version <major|minor|patch>
4. Push to git: git push && git push --tags
5. Build for PyPI: Automatically done by Github Actions when a tag is pushed.
6. Publish to PyPI: Automatically done by Github Actions when a tag is pushed.

Documentation

Docs are created automatically using mkdocs and mkdocstring. Testing docs live can be done using mkdocs serve -a localhost:<port>. Docs are built and published automatically using Github Actions.

Adding a Parser

1. Add the tests for the parser in ./tests/test_knex.py.
2. Add the parser in ./knex/parsers.py with docstrings.
3. Write amazing code until all tests are passing.
4. Add the parser in ./knex/__init__.py.
5. Add the parser in ./gen_table.py.
6. Run ./gen_table.py. It will generate that stupid markdown table
7. Run tests and validate coverage: pytest -v --cov=knex --cov-report html tests
   • The HTML coverage report will be located at ./htmlcov/index.html.

Generating Plain Old requirements.txt

There are a few of the CI tools that require standard requirements.txt format (not Poetry). Therefore when adding new dependencies with Poetry, we need to sync those to regular old requirements.txt files. This can be done with the following commands.

1. poetry export --without-hashes -f requirements.txt > requirements.txt
2. poetry export --without-hashes -f requirements.txt --dev > requirements-dev.txt

Pre-commit will make sure these are in sync if we forget.