Source Hubspot: add new fields to schema #28349

darynaishchenko · 2023-07-17T07:49:20Z

What

Add changes in scope of this epic:
#28202

How

Add fields to schema of campaigns stream (lastProcessingFinishedAt, lastProcessingStateChangeAt, lastProcessingStartedAt, processingState)
Add fields to schema of forms stream (formType)
Add fields to schema of contact_lists stream (ilsFilterBranch, internal, authorId, limitExempt, teamIds)
Add fields to schema of email_events stream (subscriptions, bcc, cc)
Add fields to schema of engagements stream (queueMembershipIds, scheduledTasks)
Add fields to schema of email_subscriptions stream (order, businessUnitId, internal, internalName, category, channel)
- Add fields to schema of subscription_changes stream (`normalizedEmailId``)
Add fields to schema of marketing_emails stream (publishedByEmail , sections, author, isCreatedFomSandboxSync, rssEmailUrl, teamPerms, securityState, isInstanceLayoutPage, etc.)
Add fields to schema of property_history stream (source-vids)
Add fields to schema of workflows stream (lastUpdatedByUserId, contactCounts, description, originalAuthorUserId, migrationStatus, updateSource, creationSource)
Add fields to schema of goals stream (hs_kpi_value_last_calculated_at)
Update expected records of streams deals, companies, pets, cars

github-actions · 2023-07-17T07:49:41Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan and you've followed all steps in the Breaking Changes Checklist
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
The connector tests are passing in CI
You've updated the connector's metadata.yaml file (new!)
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

octavia-squidington-iii · 2023-07-17T08:01:06Z

source-hubspot test report (commit `efb3508fa1`) - ❌

⏲️ Total pipeline duration: 10mn32s

Step	Result
Validate airbyte-integrations/connectors/source-hubspot/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-hubspot docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-hubspot test

octavia-squidington-iii · 2023-07-18T11:39:20Z

source-hubspot test report (commit `00d48a2dbe`) - ❌

⏲️ Total pipeline duration: 51.80s

Step	Result
Validate airbyte-integrations/connectors/source-hubspot/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-hubspot docker image for platform linux/x86_64	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-hubspot test

…paigns streams

octavia-squidington-iii · 2023-07-18T12:52:11Z

source-hubspot test report (commit `c3c7b2bb9e`) - ❌

⏲️ Total pipeline duration: 53.80s

Step	Result
Validate airbyte-integrations/connectors/source-hubspot/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-hubspot docker image for platform linux/x86_64	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-hubspot test

octavia-squidington-iii · 2023-07-18T13:07:22Z

source-hubspot test report (commit `1381e355df`) - ❌

⏲️ Total pipeline duration: 13mn22s

Step	Result
Validate airbyte-integrations/connectors/source-hubspot/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-hubspot docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-hubspot test

…cords

octavia-squidington-iii · 2023-07-18T17:15:49Z

source-hubspot test report (commit `c7af564854`) - ✅

⏲️ Total pipeline duration: 20mn14s

Step	Result
Validate airbyte-integrations/connectors/source-hubspot/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-hubspot docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-hubspot test

octavia-squidington-iii · 2023-07-19T08:14:50Z

source-hubspot test report (commit `94359c15b9`) - ❌

⏲️ Total pipeline duration: 13mn08s

Step	Result
Validate airbyte-integrations/connectors/source-hubspot/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-hubspot docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-hubspot test

octavia-squidington-iii · 2023-07-19T08:27:31Z

source-hubspot test report (commit `128a60ad3a`) - ❌

⏲️ Total pipeline duration: 11mn18s

Step	Result
Validate airbyte-integrations/connectors/source-hubspot/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-hubspot docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-hubspot test

octavia-squidington-iii · 2023-07-19T08:51:07Z

source-hubspot test report (commit `6812ffa120`) - ✅

⏲️ Total pipeline duration: 10mn15s

Step	Result
Validate airbyte-integrations/connectors/source-hubspot/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-hubspot docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-hubspot test

airbyte-integrations/connectors/source-hubspot/source_hubspot/schemas/workflows.json

octavia-squidington-iii · 2023-07-19T11:07:11Z

source-hubspot test report (commit `c331bf7722`) - ✅

⏲️ Total pipeline duration: 12mn47s

Step	Result
Validate airbyte-integrations/connectors/source-hubspot/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-hubspot docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-hubspot test

airbyte-integrations/connectors/source-hubspot/source_hubspot/schemas/email_events.json

octavia-squidington-iii · 2023-07-20T08:12:23Z

source-hubspot test report (commit `c33c7376fc`) - ✅

⏲️ Total pipeline duration: 10mn40s

Step	Result
Validate airbyte-integrations/connectors/source-hubspot/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-hubspot docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-hubspot test

* added unexpected fields * Update schema in forms stream * Update schema for email_events, email_subscriptions, engagements, campaigns streams * Update changelog * Update schema for contact_lists and forms streams, update expected records * Update version, remove fix step from dockerfile * Add fields to goals and forms schema, add some fields to ignored_fields * Add missing type declarations --------- Co-authored-by: Arsen Losenko <20901439+arsenlosenko@users.noreply.github.com>

added unexpected fields

efb3508

darynaishchenko self-assigned this Jul 17, 2023

octavia-squidington-iii added area/connectors Connector related issues connectors/source/hubspot labels Jul 17, 2023

arsenlosenko self-assigned this Jul 18, 2023

arsenlosenko changed the title ~~Source Hubspot: certificate to Silver~~ Source Hubspot: add new fields to schema Jul 18, 2023

Update schema in forms stream

00d48a2

arsenlosenko added 2 commits July 18, 2023 15:49

Update schema for email_events, email_subscriptions, engagements, cam…

c3c7b2b

…paigns streams

Update changelog

1381e35

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Jul 18, 2023

Update schema for contact_lists and forms streams, update expected re…

c7af564

…cords

arsenlosenko added 2 commits July 19, 2023 10:59

Merge branch 'master' into daryna/source-hubspot/to-silver

94359c1

Update version, remove fix step from dockerfile

128a60a

Add fields to goals and forms schema, add some fields to ignored_fields

6812ffa

arsenlosenko requested review from lazebnyi and davydov-d July 19, 2023 08:51

arsenlosenko assigned darynaishchenko and unassigned darynaishchenko Jul 19, 2023

davydov-d requested changes Jul 19, 2023

View reviewed changes

airbyte-integrations/connectors/source-hubspot/source_hubspot/schemas/workflows.json Show resolved Hide resolved

Add missing type declarations

c331bf7

arsenlosenko requested a review from davydov-d July 19, 2023 10:58

lazebnyi approved these changes Jul 19, 2023

View reviewed changes

arsenlosenko linked an issue Jul 19, 2023 that may be closed by this pull request

Source Hubspot: add new fields to schemas #28467

Closed

davydov-d approved these changes Jul 20, 2023

View reviewed changes

airbyte-integrations/connectors/source-hubspot/source_hubspot/schemas/email_events.json Show resolved Hide resolved

Merge branch 'master' into daryna/source-hubspot/to-silver

c33c737

arsenlosenko merged commit 28bc5dc into master Jul 20, 2023
22 of 23 checks passed

arsenlosenko deleted the daryna/source-hubspot/to-silver branch July 20, 2023 09:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Hubspot: add new fields to schema #28349

Source Hubspot: add new fields to schema #28349

darynaishchenko commented Jul 17, 2023 •

edited by arsenlosenko

Loading

github-actions bot commented Jul 17, 2023 •

edited by arsenlosenko

Loading

octavia-squidington-iii commented Jul 17, 2023

octavia-squidington-iii commented Jul 18, 2023

octavia-squidington-iii commented Jul 18, 2023

octavia-squidington-iii commented Jul 18, 2023

octavia-squidington-iii commented Jul 18, 2023

octavia-squidington-iii commented Jul 19, 2023

octavia-squidington-iii commented Jul 19, 2023

octavia-squidington-iii commented Jul 19, 2023

octavia-squidington-iii commented Jul 19, 2023

octavia-squidington-iii commented Jul 20, 2023

Source Hubspot: add new fields to schema #28349

Source Hubspot: add new fields to schema #28349

Conversation

darynaishchenko commented Jul 17, 2023 • edited by arsenlosenko Loading

What

How

github-actions bot commented Jul 17, 2023 • edited by arsenlosenko Loading

Before Merging a Connector Pull Request

octavia-squidington-iii commented Jul 17, 2023

source-hubspot test report (commit efb3508fa1) - ❌

octavia-squidington-iii commented Jul 18, 2023

source-hubspot test report (commit 00d48a2dbe) - ❌

octavia-squidington-iii commented Jul 18, 2023

source-hubspot test report (commit c3c7b2bb9e) - ❌

octavia-squidington-iii commented Jul 18, 2023

source-hubspot test report (commit 1381e355df) - ❌

octavia-squidington-iii commented Jul 18, 2023

source-hubspot test report (commit c7af564854) - ✅

octavia-squidington-iii commented Jul 19, 2023

source-hubspot test report (commit 94359c15b9) - ❌

octavia-squidington-iii commented Jul 19, 2023

source-hubspot test report (commit 128a60ad3a) - ❌

octavia-squidington-iii commented Jul 19, 2023

source-hubspot test report (commit 6812ffa120) - ✅

octavia-squidington-iii commented Jul 19, 2023

source-hubspot test report (commit c331bf7722) - ✅

octavia-squidington-iii commented Jul 20, 2023

source-hubspot test report (commit c33c7376fc) - ✅

darynaishchenko commented Jul 17, 2023 •

edited by arsenlosenko

Loading

github-actions bot commented Jul 17, 2023 •

edited by arsenlosenko

Loading

source-hubspot test report (commit `efb3508fa1`) - ❌

source-hubspot test report (commit `00d48a2dbe`) - ❌

source-hubspot test report (commit `c3c7b2bb9e`) - ❌

source-hubspot test report (commit `1381e355df`) - ❌

source-hubspot test report (commit `c7af564854`) - ✅

source-hubspot test report (commit `94359c15b9`) - ❌

source-hubspot test report (commit `128a60ad3a`) - ❌

source-hubspot test report (commit `6812ffa120`) - ✅

source-hubspot test report (commit `c331bf7722`) - ✅

source-hubspot test report (commit `c33c7376fc`) - ✅