Source Facebook Marketing: remove unsupported fields by artem1205 · Pull Request #20402 · airbytehq/airbyte

Artem Inzhyyants (artem1205) · 2022-12-12T23:25:59Z

What

Resolving https://github.com/airbytehq/oncall/issues/1137

How

Exclude fields from request

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

Artem Inzhyyants (artem1205) · 2022-12-12T23:26:30Z

/test connector=connectors/source-facebook-marketing

🕑 connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/3680719172
✅ connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/3680719172
Python tests coverage:

Name                                                        Stmts   Miss  Cover
-------------------------------------------------------------------------------
source_facebook_marketing/streams/__init__.py                   2      0   100%
source_facebook_marketing/__init__.py                           2      0   100%
source_facebook_marketing/spec.py                              41      1    98%
source_facebook_marketing/api.py                               96     12    88%
source_facebook_marketing/streams/base_streams.py             138     27    80%
source_facebook_marketing/streams/common.py                    44     16    64%
source_facebook_marketing/utils.py                             24      9    62%
source_facebook_marketing/source.py                            58     22    62%
source_facebook_marketing/streams/streams.py                  119     46    61%
source_facebook_marketing/streams/base_insight_streams.py     145     64    56%
source_facebook_marketing/streams/async_job.py                217    136    37%
source_facebook_marketing/streams/async_job_manager.py         78     60    23%
-------------------------------------------------------------------------------
TOTAL                                                         964    393    59%
Name                                                        Stmts   Miss  Cover
-------------------------------------------------------------------------------
source_facebook_marketing/utils.py                             24      0   100%
source_facebook_marketing/streams/common.py                    44      0   100%
source_facebook_marketing/streams/async_job.py                217      0   100%
source_facebook_marketing/streams/__init__.py                   2      0   100%
source_facebook_marketing/spec.py                              41      0   100%
source_facebook_marketing/__init__.py                           2      0   100%
source_facebook_marketing/streams/async_job_manager.py         78      3    96%
source_facebook_marketing/api.py                               96      9    91%
source_facebook_marketing/streams/base_insight_streams.py     145     16    89%
source_facebook_marketing/source.py                            58      7    88%
source_facebook_marketing/streams/base_streams.py             138     30    78%
source_facebook_marketing/streams/streams.py                  119     27    77%
-------------------------------------------------------------------------------
TOTAL                                                         964     92    90%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     208     92    56%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-280, 288-301, 306-312, 319-330, 337-353
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1599    332    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:200: Skipping new incremental test based on acceptance-test-config.yml
============ 30 passed, 1 skipped, 31 warnings in 767.45s (0:12:47) ============

Serhii Chvaliuk (grubberr) · 2022-12-13T14:49:22Z

+                # https://github.com/airbytehq/oncall/issues/1137
+                logger.info(f"Following fields were removed from the Custom{insight.name} fields list due to error"
+                            f" on Facebook side: {insight_fields.intersection(UNSUPPORTED_FIELDS)}")
+                insight_fields -= UNSUPPORTED_FIELDS


I am not sure how it works, I see you just removed this fields from some temp variable
insight_fields and that's all ?

is it ok?

Denys Davydov (davydov-d) · 2022-12-13T15:03:58Z

+                # https://github.com/airbytehq/oncall/issues/1137
+                logger.info(f"Following fields were removed from the Custom{insight.name} fields list due to error"
+                            f" on Facebook side: {insight_fields.intersection(UNSUPPORTED_FIELDS)}")
+                insight_fields -= UNSUPPORTED_FIELDS


can we validate presence of these fields in the check_connection to prevent user from providing an invalid config and sooner highlighting the issue?

check_connection already uses this function, do you want to raise an error here instead of quiet deleting?

Yes, I think it's a better UX

Denys Davydov (davydov-d) · 2022-12-14T07:24:56Z

 ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]


-LABEL io.airbyte.version=0.2.78


You also need to update changelog in the doc file please

Artem Inzhyyants (artem1205) · 2022-12-15T13:20:20Z

/publish connector=connectors/source-facebook-marketing

🕑 Publishing the following connectors:
connectors/source-facebook-marketing
https://github.com/airbytehq/airbyte/actions/runs/3704484918

Connector	Did it publish?	Were definitions generated?
connectors/source-facebook-marketing	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

Source Facebook Marketing: remove unsupported fields

366c6b4

Octavia Squidington IV (octavia-squidington-iv) added area/connectors Connector related issues connectors/source/facebook-marketing labels Dec 12, 2022

Artem Inzhyyants (artem1205) requested review from Denys Davydov (davydov-d) and Serhii Chvaliuk (grubberr) December 13, 2022 14:33

Serhii Chvaliuk (grubberr) reviewed Dec 13, 2022

View reviewed changes

Denys Davydov (davydov-d) reviewed Dec 13, 2022

View reviewed changes

Artem Inzhyyants (artem1205) added 2 commits December 13, 2022 17:40

Source Facebook Marketing: fix

2abfbd9

Source Facebook Marketing: refactor; raise error

f42853f

Artem Inzhyyants (artem1205) requested a review from Denys Davydov (davydov-d) December 13, 2022 23:23

Denys Davydov (davydov-d) suggested changes Dec 14, 2022

View reviewed changes

Source Facebook marketing: docs update

21b4bc2

Octavia Squidington IV (octavia-squidington-iv) added the area/documentation Improvements or additions to documentation label Dec 15, 2022

Artem Inzhyyants (artem1205) requested a review from Denys Davydov (davydov-d) December 15, 2022 13:16

Denys Davydov (davydov-d) approved these changes Dec 15, 2022

View reviewed changes

auto-bump connector version

1066035

Octavia Squidington III (octavia-squidington-iii) temporarily deployed to more-secrets December 15, 2022 13:59 — with GitHub Actions Inactive

Artem Inzhyyants (artem1205) merged commit 6bf2448 into master Dec 15, 2022

Artem Inzhyyants (artem1205) deleted the artem1205/source-facebook-marketing-JobException-OC-1137 branch December 15, 2022 13:59

Octavia Squidington III (octavia-squidington-iii) temporarily deployed to more-secrets December 15, 2022 14:00 — with GitHub Actions Inactive

github-actions Bot mentioned this pull request Dec 15, 2022

Bump helm chart version reference to 0.43.5 #20526

Closed

Octavia Squidington III (octavia-squidington-iii) mentioned this pull request Dec 18, 2022

Bump Airbyte version from 0.40.25 to 0.40.26 #20630

Merged

Navigation Menu

Uh oh!

Source Facebook Marketing: remove unsupported fields - #20402

Source Facebook Marketing: remove unsupported fields#20402
Artem Inzhyyants (artem1205) merged 5 commits into
masterfrom
artem1205/source-facebook-marketing-JobException-OC-1137

Artem Inzhyyants (artem1205) commented Dec 12, 2022

Uh oh!

Artem Inzhyyants (artem1205) commented Dec 12, 2022 •

edited by github-actions Bot

Loading

Uh oh!

Serhii Chvaliuk (grubberr) Dec 13, 2022

Uh oh!

Artem Inzhyyants (artem1205) Dec 13, 2022

Uh oh!

Denys Davydov (davydov-d) Dec 13, 2022

Uh oh!

Artem Inzhyyants (artem1205) Dec 13, 2022 •

edited

Loading

Uh oh!

Denys Davydov (davydov-d) Dec 13, 2022

Uh oh!

Artem Inzhyyants (artem1205) Dec 13, 2022

Uh oh!

Denys Davydov (davydov-d) Dec 14, 2022

Uh oh!

Artem Inzhyyants (artem1205) Dec 15, 2022

Uh oh!

Artem Inzhyyants (artem1205) commented Dec 15, 2022 •

edited by github-actions Bot

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

		ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]


		LABEL io.airbyte.version=0.2.78

Uh oh!

Conversation

Artem Inzhyyants (artem1205) commented Dec 12, 2022

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

Uh oh!

Artem Inzhyyants (artem1205) commented Dec 12, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

Serhii Chvaliuk (grubberr) Dec 13, 2022

Choose a reason for hiding this comment

Uh oh!

Artem Inzhyyants (artem1205) Dec 13, 2022

Choose a reason for hiding this comment

Uh oh!

Denys Davydov (davydov-d) Dec 13, 2022

Choose a reason for hiding this comment

Uh oh!

Artem Inzhyyants (artem1205) Dec 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Denys Davydov (davydov-d) Dec 13, 2022

Choose a reason for hiding this comment

Uh oh!

Artem Inzhyyants (artem1205) Dec 13, 2022

Choose a reason for hiding this comment

Uh oh!

Denys Davydov (davydov-d) Dec 14, 2022

Choose a reason for hiding this comment

Uh oh!

Artem Inzhyyants (artem1205) Dec 15, 2022

Choose a reason for hiding this comment

Uh oh!

Artem Inzhyyants (artem1205) commented Dec 15, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Artem Inzhyyants (artem1205) commented Dec 12, 2022 •

edited by github-actions Bot

Loading

Artem Inzhyyants (artem1205) Dec 13, 2022 •

edited

Loading

Artem Inzhyyants (artem1205) commented Dec 15, 2022 •

edited by github-actions Bot

Loading