Skip to content

phdesign/tap-nikabot

Repository files navigation

tap-nikabot

This is a Singer tap that produces JSON-formatted data following the Singer spec to retrieve Nikabot timesheeting data from the Nikabot API.

Quickstart

Install the package

$ pip install .

Discover the streams

$ tap-nikabot -c config.json --discover > catalog.json

Run the tap (you will need to edit the catalog to add "selected": true to a stream's metadata)

$ tap-nikabot -c config.json --catalog catalog.json

See the Singer documentation for full usage.

Streams

tap-nikabot supports the following streams:

  • users
  • roles
  • groups
  • teams
  • projects
  • records

Where records are the timesheet entries for a user.

Config

A sample config is provided in config_sample.json.

Property Default Description
access_token None Required. Nikabot API access token
page_size 1000 Allows configuring the number of records returned in each server request.
start_date None The timesheet date to start pulling records from. If not provided will sync from the beginning of time.
end_date None The timesheet date to start pulling records up to. If not provided will sync up to todays date, but see note about cutoff days.
cutoff_days 10 When using incremental replication, will only sync records up to this many days before todays date.

A note on bookmarks

Start date and end date take preferrence over bookmarks. If you have a bookmark that is earlier than the configured start date, syncing will start from start date. If you have a bookmark that is greater than end date, no records will be returned.

Cutoff days

The Nikabot API only allows for filtering timesheet records by timesheet day, and only returns the created date not a modified date, his makes incremental replication a challenge. To support incremental replication, we've introduced a concept of cutoff days.

When replication-method: "INCREMENTAL" is specified in the catalog for the records stream, the timesheet date is used as a bookmark / replication key and the tap will only sync records up to "cutoff_days" (defaults to 10 days) before today. Users have the cutoff days period to enter their timesheet information after which their entries will not be synced.

Note that because the timesheet date is a date only (not time), you will only be update to sync at most once per day.

Cutoff days cannot be combined with an end date in config. If an end date is provided, cutoff days will be ignored and records will be synced right up to end date.

Example

Cutoff Days Example Diagram

Reformatted dates

The Nikabot API returns dates in ISO 8601format without any timezone information. This is not compatible with JSON Schema which requires RFC 3339 formatted dates, which are a subset of the ISO specification but mandates timezones. To allow the Nikabot data to be successfully vaidated against the schema, we post-process the data to add timezone information by assuming all dates are in UTC.

So 2019-09-02T05:13:43.151 becomes 2019-09-02T05:13:43.151000Z.

This process uses the schema to determine which dates to reformat, if there is no schema information for the datetime field (e.g. blank schema {} ), then the dates will not be adjusted.

Development

A Makefile is provided to manage a virtual environment.

$ make init

Will setup a virtual environment in .venv and install the package and all (development and production) dependencies.

To run the linter / autoformatter (provided by black)

$ make lint

To run the test suite (which also runs the linter)

$ make test

To run the tap in discovery mode (loads config from config.json)

$ make discover

Or to run it in sync mode (loads catalog from catalog.json and config from config.json)

$ make sync

Proxying requests

You can proxy all requests through a proxy tool like OWASP ZAP, BurpSuite, Fiddler or Charles Proxy using environment variables to control the python requests pacakge. And easy way to manage this is with a shell script that sets environment variables. Create the script env.sh:

#!/bin/sh
##
# Script to execute a command with specific environment variables set.
# Note that environment variables passed as arguments will need to be escaped to avoid shell expansion.
#
# USAGE:
#   ./env.sh my-command
#   ./env.sh 'echo $MY_VAR'
##

set -e

export HTTP_PROXY="http://localhost:8080"
export HTTPS_PROXY="http://localhost:8080"
export REQUESTS_CA_BUNDLE="C:\path\to\owasp_zap_root_ca.cer"

eval "$@"

Update REQUESTS_CA_BUNDLE to point to the CA certificate for the proxy tool. Now pass your command to the script to run it.

$ ./env.sh make discover > catalog.json
$ ./env.sh tap-nikabot -c config.json --catalog catalog.json

Copyright © 2020 Stitch