🚨🚨 Source Pipedrive: Remove `followers_count` field from `Products` stream #27832

artem1205 · 2023-06-29T13:10:33Z

What

Fix Source Pipedrive Connector Health

How

Fix expected records;
update schema Products (Reason: PipeDrive Changelog)

🚨 User Impact 🚨

Products schema changed; schema update required

Playbook: https://docs.google.com/document/d/16rwD7XEstml1TiW8PH6mjk00q0vynlHDL0TvWPEG0io/edit

Pre-merge Actions

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Unit & integration tests added

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

github-actions · 2023-06-29T13:10:55Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan and you've followed all steps in the Breaking Changes Checklist
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
The connector tests are passing in CI
You've updated the connector's metadata.yaml file (new!)
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

octavia-squidington-iii · 2023-06-29T14:13:40Z

source-pipedrive test report (commit `9be7546d40`) - ❌

⏲️ Total pipeline duration: 05mn03s

Step	Result
Validate airbyte-integrations/connectors/source-pipedrive/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-pipedrive docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-pipedrive test

octavia-squidington-iii · 2023-06-29T14:19:58Z

source-pipedrive test report (commit `9be7546d40`) - ✅

⏲️ Total pipeline duration: 04mn28s

Step	Result
Validate airbyte-integrations/connectors/source-pipedrive/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-pipedrive docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-pipedrive test

alafanechere

@artem1205 does keeping the field in the schema is causing a sync issue?

artem1205 · 2023-06-29T14:34:36Z

@artem1205 does keeping the field in the schema is causing a sync issue?

No, as we don't receive it in response.

alafanechere · 2023-06-29T15:19:29Z

Would you mind publishing a pre-release and try it on a connection with the previous version to check if the column is deleted on the destination side? I believe normalization to be non destructive so hopefully the destination column is not deleted. Moreover if schema change is detected the user will be notified about this change. If it's still available on the destination we can move forward without a breaking change process

artem1205 · 2023-06-29T15:35:27Z

Would you mind publishing a pre-release and try it on a connection with the previous version to check if the column is deleted on the destination side? I believe normalization to be non destructive so hopefully the destination column is not deleted. Moreover if schema change is detected the user will be notified about this change. If it's still available on the destination we can move forward without a breaking change process

May I check this locally using Postgres?

alafanechere · 2023-06-29T16:44:07Z

@artem1205

May I check this locally using Postgres?

Yes sure, just make sure to first create and run a sync with 0.1.18 and then upgrade to 1.0.0 and share the results 🙏 Thanks!

docs/integrations/sources/pipedrive.md

artem1205 · 2023-06-29T19:10:50Z

@alafanechere

Actions Made

Sync with version 0.1.18
bumping to dev
schema refresh (both with resetting and without)
sync using dev version

Test Results

column is deleted

octavia-squidington-iii · 2023-06-29T19:16:39Z

source-pipedrive test report (commit `46319fbb74`) - ❌

⏲️ Total pipeline duration: 05mn11s

Step	Result
Validate airbyte-integrations/connectors/source-pipedrive/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-pipedrive docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-pipedrive test

octavia-squidington-iii · 2023-06-29T20:33:08Z

source-pipedrive test report (commit `1d1e7e41c0`) - ❌

⏲️ Total pipeline duration: 04mn07s

Step	Result
Validate airbyte-integrations/connectors/source-pipedrive/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-pipedrive docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-pipedrive test

octavia-squidington-iii · 2023-06-29T21:02:26Z

source-pipedrive test report (commit `18b8c12969`) - ✅

⏲️ Total pipeline duration: 04mn18s

Step	Result
Validate airbyte-integrations/connectors/source-pipedrive/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-pipedrive docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-pipedrive test

alafanechere · 2023-06-30T11:47:02Z

@artem1205 if the destination column was deleted by normalization please fill out start a breaking change procedure by creating a new document following this template: https://docs.google.com/document/d/1VYQggHbL_PN0dDDu7rCyzBLGRtX-R3cpwXaY8QxEgzw/edit#heading=h.r2xzw0kclodc

This is a breaking change because on connector upgrade the customer will lose data...

# Conflicts: # airbyte-integrations/connectors/source-pipedrive/Dockerfile # airbyte-integrations/connectors/source-pipedrive/integration_tests/expected_records.jsonl # airbyte-integrations/connectors/source-pipedrive/metadata.yaml # docs/integrations/sources/pipedrive.md

octavia-squidington-iii · 2023-07-06T14:03:11Z

source-pipedrive test report (commit `69a504bbf0`) - ❌

⏲️ Total pipeline duration: 05mn10s

Step	Result
Validate airbyte-integrations/connectors/source-pipedrive/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-pipedrive docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-pipedrive test

octavia-squidington-iii · 2023-07-06T14:54:55Z

source-pipedrive test report (commit `4e055ad8a0`) - ✅

⏲️ Total pipeline duration: 05mn09s

Step	Result
Validate airbyte-integrations/connectors/source-pipedrive/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-pipedrive docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-pipedrive test

artem1205 added 2 commits June 29, 2023 15:07

Source PipeDrive: update expected records

83c04f0

Source PipeDrive: update Products schema

af2e2f6

artem1205 added the breaking-change Don't merge me unless you are ready. label Jun 29, 2023

octavia-squidington-iii added area/connectors Connector related issues connectors/source/pipedrive labels Jun 29, 2023

artem1205 requested review from davydov-d and lazebnyi June 29, 2023 14:04

Source PipeDrive: update docs

9be7546

artem1205 marked this pull request as ready for review June 29, 2023 14:06

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Jun 29, 2023

artem1205 requested a review from a team June 29, 2023 14:26

alafanechere reviewed Jun 29, 2023

View reviewed changes

evantahler reviewed Jun 29, 2023

View reviewed changes

docs/integrations/sources/pipedrive.md Outdated Show resolved Hide resolved

Source Pipedrive: update docs

46319fb

artem1205 changed the title ~~🚨🚨 Source Pipedrive: Products schema update~~ 🚨🚨 Source Pipedrive: Remove followers_count field from Products stream Jun 29, 2023

Merge branch 'master' into artem1205/con-health-pipedrive

1d1e7e4

Source Pipedrive: update test config

18b8c12

artem1205 self-assigned this Jun 30, 2023

Update acceptance-test-config.yml

4e055ad

lazebnyi approved these changes Jul 6, 2023

View reviewed changes

artem1205 merged commit 7e232c5 into master Jul 6, 2023
22 of 24 checks passed

artem1205 deleted the artem1205/con-health-pipedrive branch July 6, 2023 15:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🚨🚨 Source Pipedrive: Remove `followers_count` field from `Products` stream #27832

🚨🚨 Source Pipedrive: Remove `followers_count` field from `Products` stream #27832

artem1205 commented Jun 29, 2023 •

edited

github-actions bot commented Jun 29, 2023 •

edited by artem1205

octavia-squidington-iii commented Jun 29, 2023

octavia-squidington-iii commented Jun 29, 2023

alafanechere left a comment

artem1205 commented Jun 29, 2023

alafanechere commented Jun 29, 2023

artem1205 commented Jun 29, 2023

alafanechere commented Jun 29, 2023

artem1205 commented Jun 29, 2023

octavia-squidington-iii commented Jun 29, 2023

octavia-squidington-iii commented Jun 29, 2023

octavia-squidington-iii commented Jun 29, 2023

alafanechere commented Jun 30, 2023

octavia-squidington-iii commented Jul 6, 2023

octavia-squidington-iii commented Jul 6, 2023

🚨🚨 Source Pipedrive: Remove followers_count field from Products stream #27832

🚨🚨 Source Pipedrive: Remove followers_count field from Products stream #27832

Conversation

artem1205 commented Jun 29, 2023 • edited

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Actions

Community member or Airbyter

Airbyter

github-actions bot commented Jun 29, 2023 • edited by artem1205

Before Merging a Connector Pull Request

octavia-squidington-iii commented Jun 29, 2023

source-pipedrive test report (commit 9be7546d40) - ❌

octavia-squidington-iii commented Jun 29, 2023

source-pipedrive test report (commit 9be7546d40) - ✅

alafanechere left a comment

Choose a reason for hiding this comment

artem1205 commented Jun 29, 2023

alafanechere commented Jun 29, 2023

artem1205 commented Jun 29, 2023

alafanechere commented Jun 29, 2023

artem1205 commented Jun 29, 2023

Actions Made

Test Results

octavia-squidington-iii commented Jun 29, 2023

source-pipedrive test report (commit 46319fbb74) - ❌

octavia-squidington-iii commented Jun 29, 2023

source-pipedrive test report (commit 1d1e7e41c0) - ❌

octavia-squidington-iii commented Jun 29, 2023

source-pipedrive test report (commit 18b8c12969) - ✅

alafanechere commented Jun 30, 2023

octavia-squidington-iii commented Jul 6, 2023

source-pipedrive test report (commit 69a504bbf0) - ❌

octavia-squidington-iii commented Jul 6, 2023

source-pipedrive test report (commit 4e055ad8a0) - ✅

🚨🚨 Source Pipedrive: Remove `followers_count` field from `Products` stream #27832

🚨🚨 Source Pipedrive: Remove `followers_count` field from `Products` stream #27832

artem1205 commented Jun 29, 2023 •

edited

github-actions bot commented Jun 29, 2023 •

edited by artem1205

source-pipedrive test report (commit `9be7546d40`) - ❌

source-pipedrive test report (commit `9be7546d40`) - ✅

source-pipedrive test report (commit `46319fbb74`) - ❌

source-pipedrive test report (commit `1d1e7e41c0`) - ❌

source-pipedrive test report (commit `18b8c12969`) - ✅

source-pipedrive test report (commit `69a504bbf0`) - ❌

source-pipedrive test report (commit `4e055ad8a0`) - ✅