Source Marketo: New Stream Segmentation #23956

aazam-gh · 2023-03-11T11:23:56Z

What

Fixes #17547

How

Added new Stream Segmentation which returns list of segments from target segmentation

🚨 User Impact 🚨

No

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

Updating a connector

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

aazam-gh · 2023-03-18T12:06:34Z

marcosmarxm

Some comments.

airbyte-integrations/connectors/source-marketo/source_marketo/schemas/segmentations.json

airbyte-integrations/connectors/source-marketo/source_marketo/source.py

marcosmarxm · 2023-03-20T14:02:35Z

airbyte-integrations/connectors/source-marketo/source_marketo/source.py

+    def path(self, **kwargs) -> str:
+        return f"rest/asset/v1/segmentation/{self.segment_id}/segments.json"


Why are you using this endpoint and not the one to retrieve all segmentations?
https://developers.marketo.com/rest-api/endpoint-reference/asset-endpoint-reference/#!/Segments/getSegmentsForSegmentationUsingGET

The issue stated to add the stream which returns list of segments inside target segmentation

@YowanR this change adds a new spec field where users need to specify the segmentation they want to retrieve. If they need two or all it won't be possible. Marketo has the option to retrieve all segmentations and we could use that to be parent stream and do automatically retrieving all data. Any comments?

I prefer that route as well because it makes it convenient as users would not need to search for the segmentIDs

I'll make the changes to have it return the entire segmentation. Is that alright?

No, I think this stream you created must retrieve the endpoint https://developers.marketo.com/rest-api/endpoint-reference/asset-endpoint-reference/#!/Segments/getSegmentationUsingGET and you must create another called Segments which uses the parent data (segmentation id) to retrieve the data for https://developers.marketo.com/rest-api/endpoint-reference/asset-endpoint-reference/#!/Segments/getSegmentsForSegmentationUsingGET

So there won't be a segmentationId in the spec right? Also creating a stream which reads the segmentationId would then mean that it is returning the data of all the segments. Is that correct?

You're correct. you must create the parent stream to retrieve all segmentations and other to read the segments. Take a look in https://docs.airbyte.com/connector-development/cdk-python/http-streams#nested-streams--caching

@marcosmarxm are there any already implemented nested streams that you know of which I can refer to get a better understanding of it?

sajarin · 2023-04-05T19:47:07Z

Hey thanks for the contribution and apologies for the delay!

I'll be the DRI responsible for reviewing this and getting it across the finish line. We currently have a backlog of about a dozen PRs, but just wanted to let you know that this PR has been added to the queue and I'll be trying my best to get to this as soon as possible.

Thanks for the contribution and for being patient :)

aazam-gh · 2023-04-05T23:34:48Z

Hello and thank you for adding it to the list :D
Of course I understand things at Airbyte can get busy busy, I look forward to getting this PR finished and merged!

marcosmarxm · 2023-04-19T13:02:06Z

/test connector=connectors/source-marketo

🕑 connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/4743625532

marcosmarxm · 2023-04-20T11:17:22Z

/test connector=connectors/source-marketo

🕑 connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/4753653269

marcosmarxm

Thanks for the contribution @Alcadeus0 rigth now Marketo is timing out integration tests. I request the connector team to take a look. There isn't anything pending from your side.

marcosmarxm · 2023-05-04T18:24:12Z

/test connector=connectors/source-marketo

🕑 connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/4886038812
✅ connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/4886038812
Python tests coverage:

Name                         Stmts   Miss  Cover
------------------------------------------------
source_marketo/__init__.py       2      0   100%
source_marketo/utils.py         33      1    97%
source_marketo/source.py       292     34    88%
------------------------------------------------
TOTAL                          327     35    89%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
================== 39 passed, 1 skipped in 4802.78s (1:20:02) ==================

lazebnyi

LGTM! Minor comment for documentation.

lazebnyi · 2023-05-10T11:22:35Z

docs/integrations/sources/marketo.md

@@ -106,6 +106,7 @@ If the 50,000 limit is too stringent, contact Marketo support for a quota increa

 | Version  | Date       | Pull Request                                             | Subject                                                                                       |
 |:---------|:-----------|:---------------------------------------------------------|:----------------------------------------------------------------------------------------------|
+| `1.1.0`  | 2023-04-18 | [23956](https://github.com/airbytehq/airbyte/pull/23956) | Add `Segmentations` Stream                                                                     |


We need add stream to the Supported Streams section.

marcosmarxm · 2023-05-12T18:14:13Z

/publish connector=connectors/source-marketo

🕑 Publishing the following connectors:
connectors/source-marketo
https://github.com/airbytehq/airbyte/actions/runs/4961819640

Connector	Version	Did it publish?	Were definitions generated?
connectors/source-marketo	1.1.0	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

* Added Segmentation Stream * change endpoint to read all segments * remove unused params * add json schema header * bump connector version and update doc * remove f-string not used * fix schema and correct segmentations base stream class * remove incremental from int test segmenations config cat * fix segmentations params * add expected records * Update marketo.md * bump the metadata file * auto-bump connector version --------- Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

Added Segmentation Stream

4cf72f2

octavia-squidington-iii added area/connectors Connector related issues bounty community connectors/source/marketo labels Mar 11, 2023

aazam-gh changed the title ~~Added Segmentation Stream~~ Source Marketo: New Stream Segmentation Mar 13, 2023

aazam-gh added 2 commits March 18, 2023 12:12

Merge branch 'airbytehq:master' into segments

6c49e47

Merge branch 'master' into segments

185d650

marcosmarxm requested changes Mar 20, 2023

View reviewed changes

marcosmarxm added the contributor-program PRs submitted through the contributor program. label Apr 3, 2023

marcosmarxm added the ga-connector-change label Apr 7, 2023

marcosmarxm assigned sh4sh Apr 7, 2023

marcosmarxm self-assigned this Apr 14, 2023

marcosmarxm and others added 5 commits April 18, 2023 14:31

Merge branch 'master' into segments

2b4bfff

change endpoint to read all segments

13ecf2a

remove unused params

d2234ec

add json schema header

c257d83

bump connector version and update doc

1db12f0

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Apr 18, 2023

marcosmarxm added 4 commits April 18, 2023 15:14

remove f-string not used

adeec2f

fix schema and correct segmentations base stream class

1c45835

remove incremental from int test segmenations config cat

fb47705

fix segmentations params

5ccec01

marcosmarxm added the gl-review label Apr 20, 2023

marcosmarxm approved these changes Apr 25, 2023

View reviewed changes

marcosmarxm and others added 2 commits May 4, 2023 15:14

Merge branch 'master' into segments

9da72c7

add expected records

0fafccb

lazebnyi approved these changes May 10, 2023

View reviewed changes

aazam-gh and others added 3 commits May 12, 2023 22:39

Update marketo.md

ea0703a

Merge branch 'master' into segments

d93a8ba

bump the metadata file

cd5ba65

auto-bump connector version

d226de3

marcosmarxm merged commit 53658ac into airbytehq:master May 12, 2023
38 of 51 checks passed

aazam-gh deleted the segments branch June 7, 2023 16:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Marketo: New Stream Segmentation #23956

Source Marketo: New Stream Segmentation #23956

aazam-gh commented Mar 11, 2023

aazam-gh commented Mar 18, 2023

marcosmarxm left a comment

marcosmarxm Mar 20, 2023

aazam-gh Mar 20, 2023

marcosmarxm Mar 20, 2023

aazam-gh Mar 20, 2023

aazam-gh Mar 20, 2023

marcosmarxm Mar 20, 2023

aazam-gh Mar 20, 2023

marcosmarxm Mar 21, 2023

aazam-gh Mar 23, 2023

sajarin commented Apr 5, 2023

aazam-gh commented Apr 5, 2023

marcosmarxm commented Apr 19, 2023 •

edited by github-actions bot

marcosmarxm commented Apr 20, 2023 •

edited by github-actions bot

marcosmarxm left a comment

marcosmarxm commented May 4, 2023 •

edited by github-actions bot

lazebnyi left a comment

lazebnyi May 10, 2023

marcosmarxm commented May 12, 2023 •

edited by github-actions bot

		def path(self, **kwargs) -> str:
		return f"rest/asset/v1/segmentation/{self.segment_id}/segments.json"

Source Marketo: New Stream Segmentation #23956

Source Marketo: New Stream Segmentation #23956

Conversation

aazam-gh commented Mar 11, 2023

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

aazam-gh commented Mar 18, 2023

marcosmarxm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sajarin commented Apr 5, 2023

aazam-gh commented Apr 5, 2023

marcosmarxm commented Apr 19, 2023 • edited by github-actions bot

marcosmarxm commented Apr 20, 2023 • edited by github-actions bot

marcosmarxm left a comment

Choose a reason for hiding this comment

marcosmarxm commented May 4, 2023 • edited by github-actions bot

Build Passed

lazebnyi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcosmarxm commented May 12, 2023 • edited by github-actions bot

marcosmarxm commented Apr 19, 2023 •

edited by github-actions bot

marcosmarxm commented Apr 20, 2023 •

edited by github-actions bot

marcosmarxm commented May 4, 2023 •

edited by github-actions bot

marcosmarxm commented May 12, 2023 •

edited by github-actions bot