🚨🚨 Source linkedin ads: Migrate to May 2023 #26372

artem1205 · 2023-05-22T21:14:49Z

What

Resolve #23609
Migrate to LinkedIn API version: May 2023

How

Migrate to LinkedIn API version: May 2023

Change Release Playbook: https://docs.google.com/document/d/1I8-4emBqKF8JiS3f3yET91MLNZSMZg_yLapaEGU927c/edit#

🚨 User Impact 🚨

Removed Streams:

Ad Direct Sponsored Contents (no longer supported)

Schema changed in streams:

Accounts
Ad Campaign Analytics
Ad Creative Analytics
Campaign Groups
Campaigns
Creatives

State changed in stream:

creatives (cursor field changed lastModified -> lastModifiedAt

Pre-merge Actions

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Unit & integration tests added

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

github-actions · 2023-05-22T21:15:11Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan and you've followed all steps in the Breaking Changes Checklist
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
You, or an Airbyter, have run /test successfully on this PR - or on a non-forked branch
You've updated the connector's metadata.yaml file (new!)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

# Conflicts: # airbyte-config-oss/init-oss/src/main/resources/seed/oss_registry.json

artem1205 · 2023-05-23T23:42:08Z

/test connector=connectors/source-linkedin-ads

🕑 connectors/source-linkedin-ads https://github.com/airbytehq/airbyte/actions/runs/5063072876
❌ connectors/source-linkedin-ads https://github.com/airbytehq/airbyte/actions/runs/5063072876
🐛 https://gradle.com/s/qtt6we46mvyfa

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Stream ad_ca...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:587: Backward compatibility tests are disabled for version 0.1.15.
============= 1 failed, 37 passed, 2 skipped in 2389.21s (0:39:49) =============

artem1205 · 2023-05-24T10:23:53Z

/test connector=connectors/source-linkedin-ads

🕑 connectors/source-linkedin-ads https://github.com/airbytehq/airbyte/actions/runs/5067534057
✅ connectors/source-linkedin-ads https://github.com/airbytehq/airbyte/actions/runs/5067534057
Python tests coverage:

Name                               Stmts   Miss  Cover
------------------------------------------------------
source_linkedin_ads/analytics.py      46      0   100%
source_linkedin_ads/__init__.py        2      0   100%
source_linkedin_ads/source.py         47      3    94%
source_linkedin_ads/utils.py         108      8    93%
source_linkedin_ads/streams.py       180     29    84%
------------------------------------------------------
TOTAL                                383     40    90%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:587: Backward compatibility tests are disabled for version 0.1.15.
================== 38 passed, 2 skipped in 2464.95s (0:41:04) ==================

bazarnov

Huge change! Looks good!

sherifnada · 2023-05-24T16:37:32Z

@artem1205 could you share the link to the breaking changes checklist for this change?

artem1205 · 2023-05-24T16:44:58Z

@artem1205 could you share the link to the breaking changes checklist for this change?

description updated

sherifnada · 2023-05-24T16:38:05Z

airbyte-integrations/connectors/source-linkedin-ads/setup.py

@@ -6,8 +6,7 @@
 from setuptools import find_packages, setup

 MAIN_REQUIREMENTS = [
-    "airbyte-cdk~=0.1",
-    "pendulum~=2.1",
+    "airbyte-cdk",


could you pin to a minor version?

GA connector should always use latest CDK version, isn't it?

latest minor version. Since major version changes are breaking we shouldn't "automatically" update this connector if we release a major CDK version

sherifnada · 2023-05-24T17:12:20Z

airbyte-integrations/connectors/source-linkedin-ads/source_linkedin_ads/utils.py

@@ -336,3 +338,34 @@ def transform_data(records: List) -> Iterable[Mapping]:
        record = transform_col_names(record, DESTINATION_RESERVED_KEYWORDS)

        yield record
+
+
+def _encode_params(data):


can you clarify in the docstring a few things:

why this is here instead of using the implementation in requests

this is being used to override a private method in RequestEncodingMixin -- this is a smell and could break if we upgrade the requests library. Is there any way around doing it that way?

refactored; will pass already encoded request params from now

pass already encoded request params

artem1205 · 2023-05-24T17:43:43Z

/test connector=connectors/source-linkedin-ads

🕑 connectors/source-linkedin-ads https://github.com/airbytehq/airbyte/actions/runs/5071799295
✅ connectors/source-linkedin-ads https://github.com/airbytehq/airbyte/actions/runs/5071799295
Python tests coverage:

Name                               Stmts   Miss  Cover
------------------------------------------------------
source_linkedin_ads/analytics.py      46      0   100%
source_linkedin_ads/__init__.py        2      0   100%
source_linkedin_ads/utils.py          91      5    95%
source_linkedin_ads/source.py         47      3    94%
source_linkedin_ads/streams.py       178     28    84%
------------------------------------------------------
TOTAL                                364     36    90%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:587: Backward compatibility tests are disabled for version 0.1.15.
================== 38 passed, 2 skipped in 2136.99s (0:35:36) ==================

sherifnada

In the rollout announcement is there a link we can point users to so they can know exactly which streams were changed/fields were added or removed? ideally it's a link to Linkedin's docs about the change

artem1205 · 2023-05-25T12:29:23Z

the only link we have is a migration guide and short video about versioning.
I had to check the streams and schemes empirically to accurately determine all changes (LinkedIn documentation is not complete)

# Conflicts: # airbyte-integrations/connectors/source-linkedin-ads/Dockerfile # airbyte-integrations/connectors/source-linkedin-ads/metadata.yaml # docs/integrations/sources/linkedin-ads.md

* Source LinkedIn Ads: update CDK version * Source LinkedIn Ads: update schemas * Source LinkedIn Ads: refactor source * Source LinkedIn Ads: Migrate to May 2023 * Source LinkedIn Ads: monkeypatch url encoding to bypass encode :,%() * Source LinkedIn Ads: fix test * Source LinkedIn Ads: fix unittest * Source Pinterest: refactor * Source LinkedIn Ads: update docstrings; Refactor * Source LinkedIn Ads: bump version * Source LinkedIn Ads: update docs * Source LinkedIn Ads: update schemas * Automated Change * Source LinkedIn Ads: update schemas; refactor * Source LinkedIn Ads: update expected records * Source LinkedIn Ads: refactor * Source LinkedIn Ads: fix unit test * Source LinkedIn Ads: fix acceptance test config * Source LinkedIn Ads: update source * Source LinkedIn Ads: update abnormal state * Source LinkedIn Ads: add TypeTransformer + fix schema * Source LinkedIn Ads: Format code * Source LinkedIn Ads: update expected records * Source LinkedIn Ads: update expected records * Source LinkedIn Ads: update doc links * Source LinkedIn Ads: pin CDK version * Source LinkedIn Ads: refactor; pass already encoded request params * Source LinkedIn Ads: fix acceptance test versioning --------- Co-authored-by: artem1205 <artem1205@users.noreply.github.com>

artem1205 added 10 commits May 20, 2023 14:29

Source LinkedIn Ads: update CDK version

2fcd70b

Source LinkedIn Ads: update schemas

4fdf4f1

Source LinkedIn Ads: refactor source

e925d4f

Source LinkedIn Ads: Migrate to May 2023

08fd7e7

Source LinkedIn Ads: monkeypatch url encoding to bypass encode :,%()

a97f799

Source LinkedIn Ads: fix test

2653cc8

Source LinkedIn Ads: fix unittest

062d3e2

Source Pinterest: refactor

1119e00

Source LinkedIn Ads: update docstrings; Refactor

63263fb

Source LinkedIn Ads: bump version

5ee4930

octavia-squidington-iii added area/connectors Connector related issues connectors/source/linkedin-ads labels May 22, 2023

Source LinkedIn Ads: update docs

3cbcf70

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label May 22, 2023

artem1205 and others added 9 commits May 22, 2023 23:23

Source LinkedIn Ads: update schemas

7cbd6e7

Automated Change

10190a8

Source LinkedIn Ads: update schemas; refactor

b2e0b96

Source LinkedIn Ads: update expected records

966e1e7

Source LinkedIn Ads: refactor

8ea1421

Source LinkedIn Ads: fix unit test

08614f1

Source LinkedIn Ads: fix acceptance test config

435d722

Source LinkedIn Ads: update source

f866fef

Merge branch 'master' into artem1205/source-linkedin-ads-migration-23609

6b0ebeb

# Conflicts: # airbyte-config-oss/init-oss/src/main/resources/seed/oss_registry.json

airbytehq deleted a comment from sonarcloud bot May 23, 2023

artem1205 added 3 commits May 24, 2023 01:15

Source LinkedIn Ads: update abnormal state

9462457

Source LinkedIn Ads: add TypeTransformer + fix schema

2281f97

Source LinkedIn Ads: Format code

f1064bd

Source LinkedIn Ads: update expected records

52d6281

Source LinkedIn Ads: update expected records

8623967

artem1205 requested review from bazarnov and lazebnyi May 24, 2023 12:55

artem1205 added the breaking-change Don't merge me unless you are ready. label May 24, 2023

artem1205 requested review from a team May 24, 2023 14:34

Source LinkedIn Ads: update doc links

b2e7a2f

bazarnov approved these changes May 24, 2023

View reviewed changes

sherifnada reviewed May 24, 2023

View reviewed changes

artem1205 added 2 commits May 24, 2023 19:18

Source LinkedIn Ads: pin CDK version

b382f04

Source LinkedIn Ads: refactor;

71fb145

pass already encoded request params

artem1205 requested a review from sherifnada May 24, 2023 18:30

sherifnada approved these changes May 25, 2023

View reviewed changes

lazebnyi approved these changes May 25, 2023

View reviewed changes

artem1205 added 2 commits May 30, 2023 12:32

Merge branch 'master' into artem1205/source-linkedin-ads-migration-23609

735573b

# Conflicts: # airbyte-integrations/connectors/source-linkedin-ads/Dockerfile # airbyte-integrations/connectors/source-linkedin-ads/metadata.yaml # docs/integrations/sources/linkedin-ads.md

Source LinkedIn Ads: fix acceptance test versioning

a72ed5a

artem1205 merged commit e3076e6 into master May 30, 2023
18 checks passed

artem1205 deleted the artem1205/source-linkedin-ads-migration-23609 branch May 30, 2023 10:58

This was referenced Aug 1, 2023

Source LinkedIn Ads: Breaking Changes to the LinkedIn Ads Connector starting 30.06.2023 #26107

Closed

Source LinkedIn Ads: Very slow sync of analytics streams #19128

Closed

FVidalCarneiro mentioned this pull request Feb 26, 2024

[source-linkedin-ads] Removed sponsored content stream #35635

Open

1 task

marcosmarxm mentioned this pull request Mar 20, 2024

🐛 [source-linkedin-ads] fix re-adding sponsored content stream #35655

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🚨🚨 Source linkedin ads: Migrate to May 2023 #26372

🚨🚨 Source linkedin ads: Migrate to May 2023 #26372

artem1205 commented May 22, 2023 •

edited

Loading

github-actions bot commented May 22, 2023 •

edited by artem1205

Loading

artem1205 commented May 23, 2023 •

edited by github-actions bot

Loading

artem1205 commented May 24, 2023 •

edited by github-actions bot

Loading

bazarnov left a comment

sherifnada commented May 24, 2023

artem1205 commented May 24, 2023

sherifnada May 24, 2023

artem1205 May 24, 2023

sherifnada May 24, 2023

sherifnada May 24, 2023

artem1205 May 24, 2023

artem1205 commented May 24, 2023 •

edited by github-actions bot

Loading

sherifnada left a comment

artem1205 commented May 25, 2023

🚨🚨 Source linkedin ads: Migrate to May 2023 #26372

🚨🚨 Source linkedin ads: Migrate to May 2023 #26372

Conversation

artem1205 commented May 22, 2023 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Actions

Community member or Airbyter

Airbyter

github-actions bot commented May 22, 2023 • edited by artem1205 Loading

Before Merging a Connector Pull Request

artem1205 commented May 23, 2023 • edited by github-actions bot Loading

Build Failed

artem1205 commented May 24, 2023 • edited by github-actions bot Loading

Build Passed

bazarnov left a comment

Choose a reason for hiding this comment

sherifnada commented May 24, 2023

artem1205 commented May 24, 2023

sherifnada May 24, 2023

Choose a reason for hiding this comment

artem1205 May 24, 2023

Choose a reason for hiding this comment

sherifnada May 24, 2023

Choose a reason for hiding this comment

sherifnada May 24, 2023

Choose a reason for hiding this comment

artem1205 May 24, 2023

Choose a reason for hiding this comment

artem1205 commented May 24, 2023 • edited by github-actions bot Loading

Build Passed

sherifnada left a comment

Choose a reason for hiding this comment

artem1205 commented May 25, 2023

artem1205 commented May 22, 2023 •

edited

Loading

github-actions bot commented May 22, 2023 •

edited by artem1205

Loading

artem1205 commented May 23, 2023 •

edited by github-actions bot

Loading

artem1205 commented May 24, 2023 •

edited by github-actions bot

Loading

artem1205 commented May 24, 2023 •

edited by github-actions bot

Loading