Source Facebook Marketing: Dynamically exclude field `rule` from stream `CustomAudiences` #29674

artem1205 · 2023-08-21T17:10:05Z

What

Resolve https://github.com/airbytehq/oncall/issues/2765

How

Dynamically exclude field rule from stream CustomAudiences if record cannot be obtained due to Too much data was requested in batch error.

🚨 User Impact 🚨

no breaking changes

Pre-merge Actions

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Unit & integration tests added

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

…cebook-marketing-OC-2765 # Conflicts: # airbyte-integrations/connectors/source-facebook-marketing/source_facebook_marketing/streams/base_streams.py

…ing-OC-2765' into artem1205/source-facebook-marketing-OC-2765

…cebook-marketing-OC-2765

github-actions · 2023-08-21T18:58:13Z

source-facebook-marketing test report (commit `2569ee13e9`) - ❌

⏲️ Total pipeline duration: 86mn14s

Step	Result
Validate airbyte-integrations/connectors/source-facebook-marketing/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-facebook-marketing docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-facebook-marketing test

github-actions · 2023-08-21T22:07:54Z

source-facebook-marketing test report (commit `4bc1d5d17d`) - ✅

⏲️ Total pipeline duration: 88mn34s

Step	Result
Connector package install	✅
Build source-facebook-marketing docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-facebook-marketing/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-facebook-marketing test

lazebnyi

As discussed, we need to implement error handling and remove any unnecessary fields.

github-actions · 2023-08-22T17:44:24Z

source-facebook-marketing test report (commit `eddcf23742`) - ❌

⏲️ Total pipeline duration: 96mn15s

Step	Result
Connector package install	✅
Build source-facebook-marketing docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	❌
Validate airbyte-integrations/connectors/source-facebook-marketing/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-facebook-marketing test

github-actions · 2023-08-22T19:09:14Z

source-facebook-marketing test report (commit `eddcf23742`) - ❌

⏲️ Total pipeline duration: 82mn37s

Step	Result
Connector package install	✅
Build source-facebook-marketing docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌
Code format checks	❌
Validate airbyte-integrations/connectors/source-facebook-marketing/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-facebook-marketing test

github-actions · 2023-08-22T20:18:53Z

source-facebook-marketing test report (commit `7c2bec0778`) - ❌

⏲️ Total pipeline duration: 67mn52s

Step	Result
Connector package install	✅
Build source-facebook-marketing docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌
Code format checks	✅
Validate airbyte-integrations/connectors/source-facebook-marketing/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-facebook-marketing test

github-actions · 2023-08-22T21:26:49Z

source-facebook-marketing test report (commit `78c5d914f3`) - ✅

⏲️ Total pipeline duration: 61mn04s

Step	Result
Connector package install	✅
Build source-facebook-marketing docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-facebook-marketing/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-facebook-marketing test

lazebnyi · 2023-08-23T11:01:40Z

...tions/connectors/source-facebook-marketing/source_facebook_marketing/streams/base_streams.py

@@ -76,8 +78,13 @@ def success(response: FacebookResponse):
            self.max_batch_size = self._initial_max_batch_size
            records.append(response.json())

-        def reduce_batch_size():
-            if self.max_batch_size == 1:
+        def reduce_batch_size(request: FacebookRequest):


It would be great to add unit tests for that functionality.

will add unit test in another PR to speed up the process

…am `CustomAudiences` (airbytehq#29674)

artem1205 and others added 17 commits August 18, 2023 20:29

Source Facebook Marketing: automatically reduce batch size

1aadacc

Source Facebook Marketing: update spec limits

a9510f7

Source Facebook Marketing: update docs

6cef102

Source Facebook Marketing: fix formatting

20171ff

Source Facebook Marketing: add unit test

21895c2

Source Facebook Marketing: refactor

268bb4e

Merge remote-tracking branch 'origin/master' into artem1205/source-fa…

f116a8c

…cebook-marketing-OC-2765 # Conflicts: # airbyte-integrations/connectors/source-facebook-marketing/source_facebook_marketing/streams/base_streams.py

Source Facebook Marketing: update docs

95d0e93

retrigger ci

d7ee6dd

retrigger ci

778b25e

retrigger ci

44b58ce

Merge branch 'master' into artem1205/source-facebook-marketing-OC-2765

5200126

retrigger ci

1ea9d37

Merge branch 'master' into artem1205/source-facebook-marketing-OC-2765

cd97f40

Merge remote-tracking branch 'origin/artem1205/source-facebook-market…

8be6acd

…ing-OC-2765' into artem1205/source-facebook-marketing-OC-2765

Merge remote-tracking branch 'origin/master' into artem1205/source-fa…

e91e70c

…cebook-marketing-OC-2765

Source Facebook Marketing: exclude rule from stream CustomAudiences

ac93bac

artem1205 assigned davydov-d and lazebnyi Aug 21, 2023

octavia-squidington-iii added connectors/source/facebook-marketing area/connectors Connector related issues labels Aug 21, 2023

Source Facebook Marketing: update docs

2569ee1

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Aug 21, 2023

artem1205 assigned artem1205 and unassigned davydov-d and lazebnyi Aug 21, 2023

artem1205 requested review from davydov-d and lazebnyi August 21, 2023 17:30

artem1205 marked this pull request as ready for review August 21, 2023 17:30

Merge branch 'master' into artem1205/source-facebook-marketing-OC-2765

4bc1d5d

artem1205 changed the title ~~Source Facebook Marketing: Exclude rule from stream CustomAudiences~~ Source Facebook Marketing: Exclude field rule from stream CustomAudiences Aug 22, 2023

lazebnyi reviewed Aug 22, 2023

View reviewed changes

artem1205 marked this pull request as draft August 22, 2023 15:40

artem1205 added 2 commits August 22, 2023 17:41

Source Facebook Marketing: dynamically remove fields from request

3bcfd57

Source Facebook Marketing: update docs

eddcf23

artem1205 marked this pull request as ready for review August 22, 2023 16:01

artem1205 requested a review from lazebnyi August 22, 2023 16:01

Source Facebook Marketing: fix formatting

7c2bec0

Merge branch 'master' into artem1205/source-facebook-marketing-OC-2765

78c5d91

artem1205 changed the title ~~Source Facebook Marketing: Exclude field rule from stream CustomAudiences~~ Source Facebook Marketing: Dynamically exclude field rule from stream CustomAudiences Aug 22, 2023

lazebnyi approved these changes Aug 23, 2023

View reviewed changes

artem1205 merged commit 30d2f54 into master Aug 23, 2023
29 checks passed

artem1205 deleted the artem1205/source-facebook-marketing-OC-2765 branch August 23, 2023 12:18

artem1205 mentioned this pull request Aug 23, 2023

Source Facebook Marketing: add unit test for batch reducing with fiel… #29752

Closed

harrytou pushed a commit to KYVENetwork/airbyte that referenced this pull request Sep 1, 2023

Source Facebook Marketing: Dynamically exclude field rule from stre…

a33babf

…am `CustomAudiences` (airbytehq#29674)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Facebook Marketing: Dynamically exclude field `rule` from stream `CustomAudiences` #29674

Source Facebook Marketing: Dynamically exclude field `rule` from stream `CustomAudiences` #29674

artem1205 commented Aug 21, 2023 •

edited

github-actions bot commented Aug 21, 2023

github-actions bot commented Aug 21, 2023

lazebnyi left a comment

github-actions bot commented Aug 22, 2023

github-actions bot commented Aug 22, 2023

github-actions bot commented Aug 22, 2023

github-actions bot commented Aug 22, 2023

lazebnyi Aug 23, 2023

artem1205 Aug 23, 2023

Source Facebook Marketing: Dynamically exclude field rule from stream CustomAudiences #29674

Source Facebook Marketing: Dynamically exclude field rule from stream CustomAudiences #29674

Conversation

artem1205 commented Aug 21, 2023 • edited

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Actions

Community member or Airbyter

Airbyter

github-actions bot commented Aug 21, 2023

source-facebook-marketing test report (commit 2569ee13e9) - ❌

github-actions bot commented Aug 21, 2023

source-facebook-marketing test report (commit 4bc1d5d17d) - ✅

lazebnyi left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 22, 2023

source-facebook-marketing test report (commit eddcf23742) - ❌

github-actions bot commented Aug 22, 2023

source-facebook-marketing test report (commit eddcf23742) - ❌

github-actions bot commented Aug 22, 2023

source-facebook-marketing test report (commit 7c2bec0778) - ❌

github-actions bot commented Aug 22, 2023

source-facebook-marketing test report (commit 78c5d914f3) - ✅

lazebnyi Aug 23, 2023

Choose a reason for hiding this comment

artem1205 Aug 23, 2023

Choose a reason for hiding this comment

Source Facebook Marketing: Dynamically exclude field `rule` from stream `CustomAudiences` #29674

Source Facebook Marketing: Dynamically exclude field `rule` from stream `CustomAudiences` #29674

artem1205 commented Aug 21, 2023 •

edited

source-facebook-marketing test report (commit `2569ee13e9`) - ❌

source-facebook-marketing test report (commit `4bc1d5d17d`) - ✅

source-facebook-marketing test report (commit `eddcf23742`) - ❌

source-facebook-marketing test report (commit `eddcf23742`) - ❌

source-facebook-marketing test report (commit `7c2bec0778`) - ❌

source-facebook-marketing test report (commit `78c5d914f3`) - ✅