Skip to content

Commit

Permalink
Source hubspot: add Feedback Submissions stream and update Ticket Pip…
Browse files Browse the repository at this point in the history
…elines stream (#9081)

* update: on api.py update url for ticket_pipelines

* update: on client.py add new entries for feedback_submissions streams and ticket_pipelines

* update: add to shchemas feedback_submissions and ticket_pipelines

* update: to integration_tests_abnormal_state add new value for feedback_submissions

* update: in sample_files add new entries for feedback_submissions and ticket_pipelines

* update: bump connector version

* fix: in feedback submission replace timestamp to updatedAt

* update version in source_definitons and Dockerfile

* run format

* config seed

* bump connector version in config file

Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
  • Loading branch information
vladimir-remar and marcosmarxm committed Jan 25, 2022
1 parent f5beea1 commit d616279
Show file tree
Hide file tree
Showing 13 changed files with 227 additions and 10 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"sourceDefinitionId": "36c891d9-4bd9-43ac-bad2-10e12756272c",
"name": "HubSpot",
"dockerRepository": "airbyte/source-hubspot",
"dockerImageTag": "0.1.33",
"dockerImageTag": "0.1.35",
"documentationUrl": "https://docs.airbyte.io/integrations/sources/hubspot",
"icon": "hubspot.svg"
}
Original file line number Diff line number Diff line change
Expand Up @@ -307,7 +307,7 @@
- name: HubSpot
sourceDefinitionId: 36c891d9-4bd9-43ac-bad2-10e12756272c
dockerRepository: airbyte/source-hubspot
dockerImageTag: 0.1.34
dockerImageTag: 0.1.35
documentationUrl: https://docs.airbyte.io/integrations/sources/hubspot
icon: hubspot.svg
sourceType: api
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3057,7 +3057,7 @@
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-hubspot:0.1.34"
- dockerImage: "airbyte/source-hubspot:0.1.35"
spec:
documentationUrl: "https://docs.airbyte.io/integrations/sources/hubspot"
connectionSpecification:
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-hubspot/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -34,5 +34,5 @@ COPY source_hubspot ./source_hubspot
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.1.34
LABEL io.airbyte.version=0.1.35
LABEL io.airbyte.name=airbyte/source-hubspot
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,14 @@ tests:
basic_read:
- config_path: "secrets/config.json"
configured_catalog_path: "sample_files/full_refresh_catalog.json"
empty_streams: ["workflows", "form_submissions"]
empty_streams: ["workflows", "form_submissions", "ticket_pipelines"]
- config_path: "secrets/config_oauth.json"
configured_catalog_path: "sample_files/configured_catalog_for_oauth_config.json"
# The `campaigns` stream is empty in this case, because we use a catalog with
# incremental streams: subscription_changes and email_events (it takes a long time to read)
# and therefore the start date is set at 2021-10-10 for `config_oauth.json`,
# but the campaign was created on 2021-01-11
empty_streams: ["campaigns", "workflows", "contacts_list_memberships", "form_submissions"]
empty_streams: ["campaigns", "workflows", "contacts_list_memberships", "form_submissions", "ticket_pipelines"]
incremental:
- config_path: "secrets/config.json"
configured_catalog_path: "sample_files/configured_catalog.json"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@
"email_events": {
"timestamp": "2221-10-12T13:37:56.412000+00:00"
},
"feedback_submissions": {
"updatedAt": "2221-10-12T13:37:56.412000+00:00"
},
"engagements": {
"lastUpdated": 7945393076412
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,18 @@
"cursor_field": ["lastUpdated"],
"destination_sync_mode": "append"
},
{
"stream": {
"name": "feedback_submissions",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"default_cursor_field": ["updatedAt"]
},
"sync_mode": "incremental",
"cursor_field": ["updatedAt"],
"destination_sync_mode": "append"
},
{
"stream": {
"name": "forms",
Expand Down Expand Up @@ -189,6 +201,15 @@
"cursor_field": ["updatedAt"],
"destination_sync_mode": "append"
},
{
"stream": {
"name": "ticket_pipelines",
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "workflows",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,15 @@
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "ticket_pipelines",
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "workflows",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -781,10 +781,10 @@ class DealPipelineStream(Stream):
class TicketPipelineStream(Stream):
"""Ticket pipelines, API v1
This endpoint requires the tickets scope.
Docs: https://legacydocs.hubspot.com/docs/methods/pipelines/get_pipelines_for_object_type
Docs: https://developers.hubspot.com/docs/api/crm/pipelines
"""

url = "/crm-pipelines/v1/pipelines/tickets"
url = "/crm/v3/pipelines/tickets"
updated_at_field = "updatedAt"
created_at_field = "createdAt"

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
MarketingEmailStream,
OwnerStream,
SubscriptionChangeStream,
TicketPipelineStream,
WorkflowStream,
)

Expand All @@ -50,6 +51,7 @@ def __init__(self, start_date, credentials, **kwargs):
"deals": DealStream(associations=["contacts"], **common_params),
"email_events": EmailEventStream(**common_params),
"engagements": EngagementStream(**common_params),
"feedback_submissions": CRMObjectIncrementalStream(entity="feedback_submissions", associations=["contacts"], **common_params),
"forms": FormStream(**common_params),
"form_submissions": FormSubmissionStream(**common_params),
"line_items": CRMObjectIncrementalStream(entity="line_item", **common_params),
Expand All @@ -58,6 +60,7 @@ def __init__(self, start_date, credentials, **kwargs):
"products": CRMObjectIncrementalStream(entity="product", **common_params),
"subscription_changes": SubscriptionChangeStream(**common_params),
"tickets": CRMObjectIncrementalStream(entity="ticket", **common_params),
"ticket_pipelines": TicketPipelineStream(**common_params),
"workflows": WorkflowStream(**common_params),
}

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": ["null", "object"],
"properties": {
"id": {
"type": ["null", "string"]
},
"properties": {
"type": ["null", "object"],
"properties": {
"hs_all_accessible_team_ids": {
"type": ["null", "string"]
},
"hs_all_assigned_business_unit_ids": {
"type": ["null", "string"]
},
"hs_contact_email_rollup": {
"type": ["null", "string"]
},
"hs_contact_id": {
"type": ["null", "string"]
},
"hs_content": {
"type": ["null", "string"]
},
"hs_created_by_user_id": {
"type": ["null", "string"]
},
"hs_createdate": {
"type": ["null", "string"]
},
"hs_engagement_id": {
"type": ["null", "string"]
},
"hs_form_guid": {
"type": ["null", "string"]
},
"hs_ingestion_id": {
"type": ["null", "string"]
},
"hs_knowledge_article_id": {
"type": ["null", "string"]
},
"hs_lastmodifieddate": {
"type": ["null", "string"]
},
"hs_merged_object_ids": {
"type": ["null", "string"]
},
"hs_object_id": {
"type": ["null", "string"]
},
"hs_response_group": {
"type": ["null", "string"]
},
"hs_submission_name": {
"type": ["null", "string"]
},
"hs_submission_timestamp": {
"type": ["null", "string"]
},
"hs_submission_url": {
"type": ["null", "string"]
},
"hs_survey_channel": {
"type": ["null", "string"]
},
"hs_survey_id": {
"type": ["null", "string"]
},
"hs_survey_name": {
"type": ["null", "string"]
},
"hs_survey_type": {
"type": ["null", "string"]
},
"hs_unique_creation_key": {
"type": ["null", "string"]
},
"hs_updated_by_user_id": {
"type": ["null", "string"]
},
"hs_user_ids_of_all_notification_followers": {
"type": ["null", "string"]
},
"hs_user_ids_of_all_notification_unfollowers": {
"type": ["null", "string"]
},
"hs_user_ids_of_all_owners": {
"type": ["null", "string"]
},
"hs_value": {
"type": ["null", "string"]
},
"hs_visitor_id": {
"type": ["null", "string"]
}
}
},
"createdAt": {
"type": ["null", "string"],
"format": "date-time"
},
"updatedAt": {
"type": ["null", "string"],
"format": "date-time"
},
"archived": {
"type": ["null", "boolean"]
},
"archivedAt": {
"type": ["null", "string"],
"format": "date-time"
},
"contacts": {
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": ["null", "object"],
"properties": {
"label": {
"type": ["null", "string"]
},
"id": {
"type": ["null", "string"]
},
"archived": {
"type": ["null", "boolean"]
},
"stages": {
"type": ["null", "array"],
"items": {
"type": ["null", "object"],
"properties": {
"label": {
"type": ["null", "string"]
},
"displayOrder": {
"type": ["null", "integer"]
},
"metadata": {
"type": ["null", "object"],
"properties": {
"ticketState": {
"type": ["null", "string"]
},
"isClosed": {
"type": ["null", "string"]
}
}
},
"id": {
"type": ["null", "string"]
},
"createdAt": {
"type": ["null", "integer"]
},
"updatedAt": {
"type": ["null", "integer"]
},
"active": {
"type": ["null", "boolean"]
}
}
}
},
"createdAt": {
"type": ["null", "integer"]
},
"updatedAt": {
"type": ["null", "integer"]
}
}
}
5 changes: 3 additions & 2 deletions docs/integrations/sources/hubspot.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,8 +110,9 @@ If you are using Oauth, most of the streams require the appropriate [scopes](htt

| Version | Date | Pull Request | Subject |
|:--------|:-----------| :--- |:-----------------------------------------------------------------------------------------------------------------------------------------------|
| 0.1.34 | 2022-01-20 | [9641](https://github.com/airbytehq/airbyte/pull/9641) | Add more fields for `email_events` stream |
| 0.1.33 | 2022-01-14 | [8887](https://github.com/airbytehq/airbyte/pull/8887) | More efficient support for incremental updates on Companies, Contact, Deals and Engagement streams |
| 0.1.35 | 2021-12-24 | [9081](https://github.com/airbytehq/airbyte/pull/9081) | Add Feedback Submissions stream and update Ticket Pipelines stream |
| 0.1.34 | 2022-01-20 | [9641](https://github.com/airbytehq/airbyte/pull/9641) | Add more fields for `email_events` stream |
| 0.1.33 | 2022-01-14 | [8887](https://github.com/airbytehq/airbyte/pull/8887) | More efficient support for incremental updates on Companies, Contact, Deals and Engagement streams |
| 0.1.32 | 2022-01-13 | [8011](https://github.com/airbytehq/airbyte/pull/8011) | Add new stream form_submissions |
| 0.1.31 | 2022-01-11 | [9385](https://github.com/airbytehq/airbyte/pull/9385) | Remove auto-generated `properties` from `Engagements` stream |
| 0.1.30 | 2021-01-10 | [9129](https://github.com/airbytehq/airbyte/pull/9129) | Created Contacts list memberships streams |
Expand Down

0 comments on commit d616279

Please sign in to comment.