Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source Bing Ads: Add undeclared columns to spec #24481

Closed
erohmensing opened this issue Mar 24, 2023 · 1 comment · Fixed by #25668
Closed

Source Bing Ads: Add undeclared columns to spec #24481

erohmensing opened this issue Mar 24, 2023 · 1 comment · Fixed by #25668
Assignees

Comments

@erohmensing
Copy link
Contributor

What

In this PR, BasicReadTest.test_read test in the Connector Acceptance Tests was updated to fail if the connector produces
stream records which contain columns that haven't been declared in the spec.

Bing Ads currently fails the updated test. Its acceptance-test-config.yml was edited with the
fail_on_extra_columns: false parameter in order to avoid this change from making the connector fail CAT.

We want to add the undeclared columns to the spec.

How this will help:

  • Column selection is currently blocked for connectors that don't declare all columns that they pass. This
    is because turning on column selection will stop these undeclared columns from being sent, when they were
    previously being sent. Doing this allows us to enable column selection for Bing Ads!
  • Users will better understand the shape of the data they'll receive, since the columns will match the spec.

How

The following descriptions of streams that pass undeclared columns come from results of the failed connector acceptance test:

columns.txt:./source-bing-ads.txt-92-The accounts stream has the following schema errors:
columns.txt:./source-bing-ads.txt:93:Additional properties are not allowed ('PaymentMethodType', 'CurrencyCode', 'TaxInformation', 'TimeZone', 'ForwardCompatibilityMap', 'Language', 'LinkedAgencies' were unexpected)
columns.txt:./source-bing-ads.txt-183-The campaigns stream has the following schema errors:
columns.txt:./source-bing-ads.txt:184:Additional properties are not allowed ('MultimediaAdsBidAdjustment' was unexpected)
columns.txt:./source-bing-ads.txt-276-The account_performance_report_daily stream has the following schema errors:
columns.txt:./source-bing-ads.txt:277:Additional properties are not allowed ('ConversionsQualified' was unexpected)
columns.txt:./source-bing-ads.txt-358-The keyword_performance_report_daily stream has the following schema errors:
columns.txt:./source-bing-ads.txt:359:Additional properties are not allowed ('HistoricalLandingPageExperience', 'HistoricalQualityScore', 'HistoricalAdRelevance', 'HistoricalExpectedCtr' were unexpected)
columns.txt:./source-bing-ads.txt-481-The campaign_performance_report_daily stream has the following schema errors:
columns.txt:./source-bing-ads.txt:482:Additional properties are not allowed ('ConversionsQualified' was unexpected)
columns.txt:./source-bing-ads.txt-606-The account_performance_report_weekly stream has the following schema errors:
columns.txt:./source-bing-ads.txt:607:Additional properties are not allowed ('ConversionsQualified' was unexpected)
columns.txt:./source-bing-ads.txt-688-The campaign_performance_report_weekly stream has the following schema errors:
columns.txt:./source-bing-ads.txt:689:Additional properties are not allowed ('ConversionsQualified' was unexpected)
columns.txt:./source-bing-ads.txt-813-The account_performance_report_monthly stream has the following schema errors:
columns.txt:./source-bing-ads.txt:814:Additional properties are not allowed ('ConversionsQualified' was unexpected)
columns.txt:./source-bing-ads.txt-895-The campaign_performance_report_monthly stream has the following schema errors:
columns.txt:./source-bing-ads.txt:896:Additional properties are not allowed ('ConversionsQualified' was unexpected)

  1. Add the missing properties indicated by the Additional properties are not allowed ('<column>', 'column' were unexpected) logs to the connector's spec.
  2. Remove fail_on_extra_columns: false from the acceptance-test-config.yml file.
  3. Commit changes to spec and acceptance-test-config.yml and open a PR.
  4. Run tests on the connector, either automatically via CI or manually via the /test command
  5. Profit!

Definition of done: Bing Ads passes CAT without declaring fail_on_extra_columns: false. If the
API starts sending over extra columns in the future, we will catch and fix them as part of the #connector-health
movement.

@evantahler
Copy link
Contributor

When this issue is picked up, please also consult our DataDog Dashboard for Record Unexpected Fields for this connector. The API might be returning even more record properties that need to be added to the catalog.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants