TDL-19422 add new streams (#34)

* adding bookmarks test

* added the all fields test which includes suggested changes

* made changes to tests based on child stream

* modified tests based on child stream

* made changes on PR comments

* - fixed existing integration test

* - fixed integration tests

* - added interrupted sync test

* - fixed interrutped sync test

* refactor init and client.py

* - added assertion exception for interrupted sync ticket

* - added exception for full_table assertions

* refactor client.py with proper HTTP error contexts

* fix type hinting

* - fixed review comments

* Refactor discovery process

* delete schema.py

* modify replication key for workflows

* add pylint step to circleci job

* remove unwanted comments, modify type hinting

* add extra_requires in setup.py

* key_properties to list

* uncomment discovery test for automatic fields

* remove bugfix comment and code for automatic field selection

* revert to remove conflicts for PR-28

* update workflow endpoint in readme

* add key_properties to catalog

* check integrations are successful

* check integrations are successful

* sync refactoring..

* Adding properties field to customers schema (#32)

* Adding properties field to customers schema

* Adding properties field to transformation list

* bookmark methods

* sync refactoring

* add ratings stream

* integration tests for ratings stream

* add teams stream

* set default bookmark value

* point to sync to new file

* rename ratings to happiness_ratings, update readme

* rename ratings to happiness_ratings

* add unittests

* add unittests

* sync for child streams

* fix parent name in child records

* fix bug for child streams

* remove foreign keys as automatically selected

* point to child schema

* point to child schema

* fix foreign key assertions

* parse dates for comparing

* type hinting

* remove teams from expected streams

* remove tests running for teams stream

* include tests for teams

* add stream team_users

* add stream team_users

* all_fields revert for teams

* chnage primary keys for ratings stream

* change stream names, remove nulls for PKs, update README.md

* update integration tests

* remove rating field from ratings stream

* add comment for start end params affect

* merge with master

* change log and version bump

---------

Co-authored-by: Manoj Kumar Anand <manand@talend.com>
Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com>
Co-authored-by: Rushikesh Todkar <98420315+RushiT0122@users.noreply.github.com>
Co-authored-by: Arthur Borcato <66625056+arthur-borcato@users.noreply.github.com>

CHANGELOG.md

@@ -1,5 +1,8 @@
 # Changelog
 
+## 1.2.0
+  * Adds new streams `happiness_ratings_report`, `teams`, `team_members` [#34](https://github.com/singer-io/tap-helpscout/pull/34)
+
 ## 1.1.1
   * Makes replication keys as automatic fields [#37](https://github.com/singer-io/tap-helpscout/pull/37)
   * Adds `properties` field to `customers` stream [#32](https://github.com/singer-io/tap-helpscout/pull/32)

README.md

@@ -93,10 +93,34 @@ This tap:
   - Bookmark: modified_at (date-time)
 - Transformations: Fields camelCase to snake_case.
 
+[**happiness_ratings_report**](https://developer.helpscout.com/mailbox-api/endpoints/reports/happiness/reports-happiness-ratings/)
+- Endpoint: https://api.helpscout.net/v2/reports/happiness/ratings
+- Primary keys: thread_id, rating_created_at, conversation_id
+- Foreign keys: conversation_id(conversations), thread_id(conversation_threads), rating_customer_id(customers), rating_user_id(users)
+- Replication strategy: Full table (query all)
+  - Bookmark: None
+- Transformations: Fields camelCase to snake_case.
+
+[**teams**](https://developer.helpscout.com/mailbox-api/endpoints/teams/list-teams/)
+- Endpoint: https://api.helpscout.net/v2/teams
+- Primary keys: id
+- Foreign keys: None
+- Replication strategy: Incremental (query all, filter results)
+  - Bookmark: updated_at (date-time)
+- Transformations: Fields camelCase to snake_case.
+
+[**team_members**](https://developer.helpscout.com/mailbox-api/endpoints/teams/list-team-members/)
+- Endpoint: https://api.helpscout.net/v2/teams/{team_id}/members
+- Primary keys: team_id, user_id
+- Foreign keys: team_id(teams), user_id(users)
+- Replication strategy: Full table (ALL for each parent Conversation)
+  - Bookmark: None
+- Transformations: Fields camelCase to snake_case.
+
 
 ## Authentication
 [Refresh Access Token](https://developer.helpscout.com/mailbox-api/overview/authentication/#4-refresh-access-token)
-The tap should provides a `refresh_token`, `client_id` and `client_secret` to get an `access_token` when the tap starts. If/when the access_token expires in the middle of a run, the tap gets a new `access_token` and `refresh_token`. The `refresh_token` expires every use and new one is generated and persisted in the tap `config.json` until the next authentication.
+The tap should provide a `refresh_token`, `client_id` and `client_secret` to get an `access_token` when the tap starts. If/when the access_token expires in the middle of a run, the tap gets a new `access_token` and `refresh_token`. The `refresh_token` expires every use and new one is generated and persisted in the tap `config.json` until the next authentication.
 To generate the necessary API keys: `client_id` and `client_secret`, follow these instructions to [Create My App](https://developer.helpscout.com/mailbox-api/overview/authentication/#oauth2-application) in your User Profile of the HelpScout web console application.
 - App Name: tap-helpscout
 - Redirect URL: https://app.stitchdata.test:8080/v2/integrations/platform.helpscout/callback

setup.py

@@ -4,7 +4,7 @@
 
 setup(
     name="tap-helpscout",
-    version="1.1.1",
+    version="1.2.0",
     description="Singer.io tap for extracting data from the HelpScout Mailbox API 2.0",
     author="jeff.huth@bytecode.io",
     classifiers=["Programming Language :: Python :: 3 :: Only"],

tap_helpscout/__init__.py

@@ -26,6 +26,7 @@ def main():
 
     with HelpScoutClient(parsed_args.config_path, parsed_args.config, parsed_args.dev) as helpscout_client:
 
+        state = parsed_args.state or {}
         if parsed_args.discover:
             do_discover()
         else:

tap_helpscout/client.py

@@ -4,8 +4,8 @@
 
 import backoff
 import requests
-from singer import metrics
 
+from singer import metrics
 from . import exceptions as errors
 
 

tap_helpscout/schemas/customers.json

@@ -258,6 +258,31 @@
         }
       ]
     },
+    "properties": {
+      "anyOf": [
+        {
+          "type": "array",
+          "items": {
+            "type": "object",
+            "additionalProperties": false,
+            "properties": {
+              "type": {
+                "type": ["null", "string"]
+              },
+              "name": {
+                "type": ["null", "string"]
+              },
+              "value": {
+                "type": ["null", "string"]
+              }
+            }
+          }
+        },
+        {
+          "type": "null"
+        }
+      ]
+    },
     "phones": {
       "anyOf": [
         {

tap_helpscout/schemas/happiness_ratings_report.json

@@ -0,0 +1,41 @@
+{
+  "type": "object",
+  "additionalProperties": false,
+  "properties": {
+    "thread_id": {
+      "type": ["null", "integer"]
+    },
+    "conversation_id": {
+      "type": ["integer"]
+    },
+    "thread_created_at": {
+      "type": ["null", "string"],
+      "format": "date-time"
+    },
+    "rating_id": {
+      "type": ["null", "integer"]
+    },
+    "rating_customer_id": {
+      "type": ["integer"]
+    },
+    "rating_customer_name": {
+      "type": ["null", "string"]
+    },
+    "rating_comments": {
+      "type": ["null", "string"]
+    },
+    "rating_created_at": {
+      "type": ["string"],
+      "format": "date-time"
+    },
+    "rating_user_id": {
+      "type": ["null", "integer"]
+    },
+    "rating_user_name": {
+      "type": ["null", "string"]
+    },
+    "type": {
+      "type": ["null", "string"]
+    }
+  }
+}

tap_helpscout/schemas/team_members.json

@@ -0,0 +1,12 @@
+{
+  "type": "object",
+  "additionalProperties": false,
+  "properties": {
+    "team_id": {
+      "type": ["integer"]
+    },
+    "user_id": {
+      "type": ["integer"]
+    }
+  }
+}

tap_helpscout/schemas/teams.json

@@ -0,0 +1,32 @@
+{
+  "type": "object",
+  "additionalProperties": false,
+  "properties": {
+    "id": {
+      "type": ["integer"]
+    },
+    "name": {
+      "type": ["null", "string"]
+    },
+    "timezone": {
+      "type": ["null", "string"]
+    },
+    "mention": {
+      "type": ["null", "string"]
+    },
+    "photo_url": {
+      "type": ["null", "string"]
+    },
+    "created_at": {
+      "type": ["null", "string"],
+      "format": "date-time"
+    },
+    "updated_at": {
+      "type": ["null", "string"],
+      "format": "date-time"
+    },
+    "initials": {
+      "type": ["null", "string"]
+    }
+  }
+}

tap_helpscout/streams/__init__.py

@@ -1,19 +1,26 @@
-from .conversation_threads import ConversationThreads
 from .conversations import Conversations
+from .conversation_threads import ConversationThreads
 from .customers import Customers
+from .happiness_ratings_report import HappinessRatingsReport
+from .mailboxes import MailBoxes
 from .mailbox_fields import MailBoxFields
 from .mailbox_folders import MailBoxFolders
-from .mailboxes import MailBoxes
+from .teams import Teams
+from .team_members import TeamMembers
 from .users import Users
 from .workflows import Workflows
 
+
 STREAMS = {
     Conversations.tap_stream_id: Conversations,
     ConversationThreads.tap_stream_id: ConversationThreads,
     Customers.tap_stream_id: Customers,
+    HappinessRatingsReport.tap_stream_id: HappinessRatingsReport,
     MailBoxes.tap_stream_id: MailBoxes,
     MailBoxFields.tap_stream_id: MailBoxFields,
     MailBoxFolders.tap_stream_id: MailBoxFolders,
+    Teams.tap_stream_id: Teams,
+    TeamMembers.tap_stream_id: TeamMembers,
     Users.tap_stream_id: Users,
-    Workflows.tap_stream_id: Workflows,
+    Workflows.tap_stream_id: Workflows
 }

tap_helpscout/streams/abstract.py

@@ -1,5 +1,6 @@
 from abc import ABC, abstractmethod
 from typing import Dict, Iterator, List, Set, Tuple
+from datetime import datetime, timezone
 
 import singer
 from singer import Transformer, metrics, write_state
@@ -100,30 +101,44 @@ def make_request_params(self, state) -> str:
         """Generates request params required to send an API request."""
         if self.replication_query_field:
             self.params[self.replication_query_field] = self.get_bookmark(state)
-        return "&".join([f"{key}={value}" for (key, value) in self.params.items()])
+        if self.tap_stream_id == "happiness_ratings_report":
+            # start and end params filters out records based on ratingCreatedAt field
+            self.params["start"] = self.get_bookmark(state)
+            self.params["end"] = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
+        return '&'.join([f'{key}={value}' for (key, value) in self.params.items()])
 
     def get_records(self, state: Dict, parent_id=None) -> Iterator[Dict]:
-        """Retrieves records from API as paginated streams."""
+        """Retrieves records from API as paginated streams"""
         page = total_pages = 1
         path = self.path.format(parent_id) if parent_id else self.path
         query_string = self.make_request_params(state)
         while page <= total_pages:
             query_string_tmp = f"{query_string}&page={page}"
-            logger.info(f"URL for {self.tap_stream_id}: https://api.helpscout.net/v2{path}?" f"{query_string_tmp}")
+            logger.info(f'URL for {self.tap_stream_id}: https://api.helpscout.net/v2{path}?'
+                        f'{query_string_tmp}')
             data = self.client.get(path, params=query_string_tmp, endpoint=self.tap_stream_id)
             yield from self.transform_records(data)
-            page = data["page"]["number"]
-            total_pages = data["page"]["totalPages"]
+            page = data["page"] if self.tap_stream_id == "happiness_ratings_report" else \
+                data["page"]["number"]
+            total_pages = data["pages"] if self.tap_stream_id == "happiness_ratings_report" else \
+                data["page"]["totalPages"]
             if page == 0:
                 break
             page += 1
 
     def transform_records(self, data: Dict) -> List:
-        """Transforms keys in extracted data."""
-        return transform_json(data["_embedded"], self.data_key)[self.data_key] if "_embedded" in data else []
+        """Transforms keys in extracted data"""
+        if self.tap_stream_id == "happiness_ratings_report":
+            return transform_json(data, self.data_key, self.tap_stream_id)[self.data_key]
+        elif "_embedded" in data:
+            return transform_json(data["_embedded"], self.data_key, self.tap_stream_id)[self.data_key]
+        else:
+            return []
+
+    def process_records(self, state: Dict, schema: Dict, stream_metadata: Dict, is_parent=False,
+                        parent_id=None) -> Set:
+        """Processes and writes transformed data"""
 
-    def process_records(self, state: Dict, schema: Dict, stream_metadata: Dict, is_parent=False, parent_id=None) -> Set:
-        """Processes and writes transformed data."""
         parent_ids = set()
         current_bookmark = max_bookmark_value = self.get_bookmark(state)
         with Transformer() as transformer:

tap_helpscout/streams/happiness_ratings_report.py

@@ -0,0 +1,10 @@
+from .abstract import FullStream
+
+
+class HappinessRatingsReport(FullStream):
+    """Class for `happiness_ratings_report` stream"""
+    stream = tap_stream_id = "happiness_ratings_report"
+    path = "/reports/happiness/ratings"
+    key_properties = ["rating_customer_id", "conversation_id", "rating_created_at"]
+    data_key = "results"
+    is_child = False

tap_helpscout/streams/mailbox_folders.py

@@ -3,6 +3,7 @@
 
 class MailBoxFolders(IncrementalStream):
     """Class for `mailbox_folders` stream."""
+
     stream = tap_stream_id = "mailbox_folders"
     path = "/mailboxes/{}/folders"
     key_properties = ["id"]

tap_helpscout/streams/team_members.py

@@ -0,0 +1,11 @@
+from .abstract import FullStream
+
+
+class TeamMembers(FullStream):
+    """Class for `team_members` stream"""
+    stream = tap_stream_id = "team_members"
+    path = "/teams/{}/members"
+    key_properties = ["team_id", "user_id"]
+    data_key = "users"
+    is_child = True
+    parent = "team"

tap_helpscout/streams/teams.py

@@ -0,0 +1,14 @@
+from .abstract import IncrementalStream
+
+
+class Teams(IncrementalStream):
+    """Class for `conversations` stream"""
+    stream = tap_stream_id = "teams"
+    path = "/teams"
+    key_properties = ["id"]
+    replication_key = "updated_at"
+    replication_key_type = "datetime"
+    valid_replication_keys = ("updated_at",)
+    data_key = "teams"
+    child_streams = ["team_members"]
+    is_child = False

tap_helpscout/transform.py

@@ -75,10 +75,38 @@ def transform_conversations(this_json, path=None):
     return this_json
 
 
+def transform_ratings(this_json, path):
+    if path is None:
+        return this_json
+
+    for record in this_json[path]:
+        if 'id' in record:
+            record["conversation_id"] = record["id"]
+            record.pop("id")
+
+        record["thread_id"] = record["threadid"]
+        record.pop("threadid")
+
+    return this_json
+
+
+def transform_team_users(this_json, path):
+    for record in this_json[path]:
+        record["user_id"] = record["id"]
+
+    return this_json
+
+
 # Run all transforms: de-nests _embedded, removes _embedded/_links, and
 # Converts camelCase to snake_case for field_name keys.
-def transform_json(this_json, path):
+def transform_json(this_json, path, stream_name):
     de_nested_json = denest_embedded_nodes(this_json, path)
     no_links_json = remove_embedded_links(de_nested_json)
     converted_json = convert_json(no_links_json)
-    return transform_conversations(converted_json, path) if path == "conversations" else converted_json
+    if stream_name == "conversations":
+        return transform_conversations(converted_json, path)
+    elif stream_name == "happiness_ratings_report":
+        return transform_ratings(converted_json, path)
+    elif stream_name == "team_members":
+        return transform_team_users(converted_json, path)
+    return converted_json