Skip to content

singer-io/tap-ujet

Repository files navigation

tap-ujet

This is a Singer tap that produces JSON-formatted data following the Singer spec.

This tap:

  • Pulls raw data from the UJET API
  • Extracts the following resources:
    • Agents
    • Agent Activity Logs
    • Calls
    • Chats
    • Menus
    • Menu Tree
    • Teams
    • Team Tree
    • User Statuses
  • Outputs the schema for each resource
  • Incrementally pulls data based on the input state

Streams

agents

  • Endpoint: https://{subdomain}.ujet.co/manager/api/v1/agents
  • Primary key fields: id
  • Foreign key fields: teams > id
  • Replication strategy: INCREMENTAL (query filtered)
    • Bookmark: status_updated_at
    • Bookmark query field: status_updated_at[from]
  • Transformations: none

agent_activity_logs

  • Endpoint: https://{subdomain}.ujet.co/manager/api/v1/agent_activity_logs
  • Primary key fields: id
  • Foreign key fields: teams > id
  • Replication strategy: INCREMENTAL (query filtered)
    • Bookmark: started_at
    • Bookmark query field: started_at[from]
  • Transformations: none

calls

  • Endpoint: https://{subdomain}.ujet.co/manager/api/v1/calls
  • Primary key fields: id
  • Foreign key fields: teams > id
  • Replication strategy: INCREMENTAL (query filtered)
    • Bookmark: updated_at
    • Bookmark query field: updated_at[from]
  • Transformations: none

chats

  • Endpoint: https://{subdomain}.ujet.co/manager/api/v1/chats
  • Primary key fields: id
  • Foreign key fields: teams > id
  • Replication strategy: INCREMENTAL (query filtered)
    • Bookmark: updated_at
    • Bookmark query field: updated_at[from]
  • Transformations: none

menu_tree

  • Endpoint: https://{subdomain}.ujet.co/manager/api/v1/menu/tree
  • Primary key fields: id
  • Foreign key fields: none
  • Replication strategy: FULL_TABLE
  • Transformations: de-nest all recursive 'children'

menus

  • Endpoint: https://{subdomain}.ujet.co/manager/api/v1/menus
  • Primary key fields: id
  • Foreign key fields: none
  • Replication strategy: FULL_TABLE
  • Transformations: none

team_tree

  • Endpoint: https://{subdomain}.ujet.co/manager/api/v1/team/tree
  • Primary key fields: id
  • Foreign key fields: none
  • Replication strategy: FULL_TABLE
  • Transformations: de-nest all recursive 'children'

teams

  • Endpoint: https://{subdomain}.ujet.co/manager/api/v1/teams
  • Primary key fields: id
  • Foreign key fields: none
  • Replication strategy: FULL_TABLE
  • Transformations: none

user_statuses

  • Endpoint: https://{subdomain}.ujet.co/manager/api/v1/user_statuses
  • Primary key fields: id
  • Foreign key fields: none
  • Replication strategy: FULL_TABLE
  • Transformations: none

Quick Start

  1. Install

    Clone this repository, and then install using setup.py. We recommend using a virtualenv:

    > virtualenv -p python3 venv
    > source venv/bin/activate
    > python setup.py install
    OR
    > cd .../tap-ujet
    > pip install .
  2. Dependent libraries The following dependent libraries were installed.

    > pip install singer-python
    > pip install singer-tools
    > pip install target-stitch
    > pip install target-json
    
  3. Create your tap's config.json file. The server_subdomain is everything before .ujet.com. in the ujet URL. The account_name is everything between .ujet.com. and api in the ujet URL..

    {
        "company_key": "YOUR_COMPANY_KEY",
        "company_secret": "YOUR_COMPANY_SECRET_CODE",
        "subdomain": "YOUR_COMPANY",
        "domain": "ujet",
        "start_date": "2019-01-01T00:00:00Z",
        "user_agent": "tap-ujet <api_user_email@your_company.com>"
    }

    Optionally, also create a state.json file. currently_syncing is an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off.

    {
        "currently_syncing": "calls",
        "bookmarks": {
            "agnets": "2019-09-27T22:34:39.000000Z",
            "agent_activity_logs": "2019-09-28T15:30:26.000000Z",
            "calls": "2019-09-28T18:23:53Z",
            "chats": "2019-09-27T22:40:30.000000Z"
        }
    }
  4. Run the Tap in Discovery Mode This creates a catalog.json for selecting objects/fields to integrate:

    tap-ujet --config config.json --discover > catalog.json

    See the Singer docs on discovery mode here.

  5. Run the Tap in Sync Mode (with catalog) and write out to state file

    For Sync mode:

    > tap-ujet --config tap_config.json --catalog catalog.json > state.json
    > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

    To load to json files to verify outputs:

    > tap-ujet --config tap_config.json --catalog catalog.json | target-json > state.json
    > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

    To pseudo-load to Stitch Import API with dry run:

    > tap-ujet --config tap_config.json --catalog catalog.json | target-stitch --config target_config.json --dry-run > state.json
    > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json
  6. Test the Tap

    While code/bytecode/Stitchoping the ujet tap, the following utilities were run in accordance with Singer.io best practices: Pylint to improve code quality:

    > pylint tap_ujet -d missing-docstring -d logging-format-interpolation -d too-many-locals -d too-many-arguments

    Pylint test resulted in the following score:

    TBD LATER
    Your code has been rated at TBD/10

    To check the tap and verify working:

    > tap-ujet --config tap_config.json --catalog catalog.json | singer-check-tap > state.json
    > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

    Check tap resulted in the following:

      Checking stdin for valid Singer-formatted data
      The output is valid.
      It contained 3704 messages for 9 streams.
    
            9 schema messages
         3640 record messages
           55 state messages
    
      Details by stream:
      +---------------------+---------+---------+
      | stream              | records | schemas |
      +---------------------+---------+---------+
      | agent_activity_logs | 3148    | 1       |
      | agents              | 28      | 1       |
      | menus               | 50      | 1       |
      | teams               | 5       | 1       |
      | team_tree           | 5       | 1       |
      | menu_tree           | 34      | 1       |
      | chats               | 0       | 1       |
      | calls               | 356     | 1       |
      | user_statuses       | 14      | 1       |
      +---------------------+---------+---------+

Copyright © 2019 Stitch