Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

馃悰 Source Marketo: Raise config error if quota exceeded #33999

Merged
merged 4 commits into from
Jan 8, 2024
Merged

馃悰 Source Marketo: Raise config error if quota exceeded #33999

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
aeb5787
Source Marketo: Raise config error if quota exceeded
artem1205 Jan 8, 2024
dd478b5
Source Marketo: add unit tests + docs update
artem1205 Jan 8, 2024
cd05187
Merge branch 'master' into artem1205/source-marketo-3644
artem1205 Jan 8, 2024
4a6377d
Source Marketo: update tag
artem1205 Jan 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 2 additions & 2 deletions airbyte-integrations/connectors/source-marketo/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,11 @@ data:
hosts:
- "*.mktorest.com"
connectorBuildOptions:
baseImage: docker.io/airbyte/python-connector-base:1.1.0@sha256:bd98f6505c6764b1b5f99d3aedc23dfc9e9af631a62533f60eb32b1d3dbab20c
baseImage: docker.io/airbyte/python-connector-base:1.2.0@sha256:c22a9d97464b69d6ef01898edf3f8612dc11614f05a84984451dde195f337db9
connectorSubtype: api
connectorType: source
definitionId: 9e0556f4-69df-4522-a3fb-03264d36b348
dockerImageTag: 1.2.3
dockerImageTag: 1.2.4
dockerRepository: airbyte/source-marketo
documentationUrl: https://docs.airbyte.com/integrations/sources/marketo
githubIssueLabel: source-marketo
Expand Down
11 changes: 9 additions & 2 deletions airbyte-integrations/connectors/source-marketo/source_marketo/source.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
import csv
import datetime
import json
import re
from abc import ABC
from time import sleep
from typing import Any, Iterable, List, Mapping, MutableMapping, Optional, Tuple
Expand All @@ -17,6 +18,8 @@
from airbyte_cdk.sources.streams.availability_strategy import AvailabilityStrategy
from airbyte_cdk.sources.streams.http import HttpStream
from airbyte_cdk.sources.streams.http.auth import Oauth2Authenticator
from airbyte_cdk.utils import AirbyteTracedException
from airbyte_protocol.models import FailureType

from .utils import STRING_TYPES, clean_string, format_value, to_datetime_str

Expand Down Expand Up @@ -300,8 +303,12 @@ def path(self, **kwargs) -> str:
def should_retry(self, response: requests.Response) -> bool:
if response.status_code == 429 or 500 <= response.status_code < 600:
return True
record = next(self.parse_response(response, {}), {})
status, export_id = record.get("status", "").lower(), record.get("exportId")
if errors := response.json().get("errors"):
if errors[0].get("code") == "1029" and re.match("Export daily quota \d+MB exceeded", errors[0].get("message")):
message = "Daily limit for job extractions has been reached (resets daily at 12:00AM CST)."
raise AirbyteTracedException(internal_message=response.text, message=message, failure_type=FailureType.config_error)
result = response.json().get("result")[0]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this construction looks a little bit strange for me. You are getting result using safe .get method which means you expect respone may not have result in response body. But at the same time you are getting first element [0] of it like you expect result is allways present. I think it will be more clear if we replace it with this:

response.json()["result"][0]

if you are sure result will allways be present in response, or this:

response.json().get("result", [None])[0]

status, export_id = result.get("status", "").lower(), result.get("exportId")
if status != "created" or not export_id:
self.logger.warning(f"Failed to create export job! Status is {status}!")
return True
Expand Down
35 changes: 34 additions & 1 deletion airbyte-integrations/connectors/source-marketo/unit_tests/test_source.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,19 @@

import pendulum
import pytest
import requests
from airbyte_cdk.models.airbyte_protocol import SyncMode
from source_marketo.source import Activities, Campaigns, IncrementalMarketoStream, Leads, MarketoStream, Programs, SourceMarketo
from airbyte_cdk.utils import AirbyteTracedException
from source_marketo.source import (
Activities,
Campaigns,
IncrementalMarketoStream,
Leads,
MarketoExportCreate,
MarketoStream,
Programs,
SourceMarketo,
)


def test_create_export_job(mocker, send_email_stream, caplog):
Expand All @@ -26,6 +37,28 @@ def test_create_export_job(mocker, send_email_stream, caplog):
assert "Failed to create export job! Status is failed!" in caplog.records[-1].message


def test_should_retry_quota_exceeded(config, requests_mock):
create_job_url = "https://602-euo-598.mktorest.com/rest/v1/leads/export/create.json?batchSize=300"
response_json = {
"requestId": "d2ca#18c0b9833bf",
"success": False,
"errors": [
{
"code": "1029",
"message": "Export daily quota 500MB exceeded."
}
]
}
requests_mock.register_uri("GET", create_job_url, status_code=200, json=response_json)

response = requests.get(create_job_url)
with pytest.raises(AirbyteTracedException) as e:
MarketoExportCreate(config).should_retry(response)

assert e.value.message == "Daily limit for job extractions has been reached (resets daily at 12:00AM CST)."



@pytest.mark.parametrize(
"activity, expected_schema",
(
Expand Down
49 changes: 25 additions & 24 deletions docs/integrations/sources/marketo.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,27 +115,28 @@ If the 50,000 limit is too stringent, contact Marketo support for a quota increa

## Changelog

| Version | Date | Pull Request | Subject |
|:---------|:-----------|:---------------------------------------------------------|:-------------------------------------------------------------------------------------------------|
| `1.2.3` | 2023-08-02 | [28999](https://github.com/airbytehq/airbyte/pull/28999) | Fix for ` _csv.Error: line contains NUL` |
| `1.2.2` | 2023-10-19 | [31599](https://github.com/airbytehq/airbyte/pull/31599) | Base image migration: remove Dockerfile and use the python-connector-base image |
| `1.2.1` | 2023-09-18 | [30533](https://github.com/airbytehq/airbyte/pull/30533) | Fix `json_schema` for stream `Leads` |
| `1.2.0` | 2023-06-26 | [27726](https://github.com/airbytehq/airbyte/pull/27726) | License Update: Elv2 |
| `1.1.0` | 2023-04-18 | [23956](https://github.com/airbytehq/airbyte/pull/23956) | Add `Segmentations` Stream |
| `1.0.4` | 2023-04-25 | [25481](https://github.com/airbytehq/airbyte/pull/25481) | Minor fix for bug caused by `<=` producing additional API call when there is a single date slice |
| `1.0.3` | 2023-02-13 | [22938](https://github.com/airbytehq/airbyte/pull/22938) | Specified date formatting in specification |
| `1.0.2` | 2023-02-01 | [22203](https://github.com/airbytehq/airbyte/pull/22203) | Handle Null cursor values |
| `1.0.1` | 2023-01-31 | [22015](https://github.com/airbytehq/airbyte/pull/22015) | Set `AvailabilityStrategy` for streams explicitly to `None` |
| `1.0.0` | 2023-01-25 | [21790](https://github.com/airbytehq/airbyte/pull/21790) | Fix `activities_*` stream schemas |
| `0.1.12` | 2023-01-19 | [20973](https://github.com/airbytehq/airbyte/pull/20973) | Fix encoding error (note: this change is not in version 1.0.0, but is in later versions |
| `0.1.11` | 2022-09-30 | [17445](https://github.com/airbytehq/airbyte/pull/17445) | Do not use temporary files for memory optimization |
| `0.1.10` | 2022-09-30 | [17445](https://github.com/airbytehq/airbyte/pull/17445) | Optimize memory consumption |
| `0.1.9` | 2022-09-28 | [17304](https://github.com/airbytehq/airbyte/pull/17304) | Migrate to per-stream sate. |
| `0.1.7` | 2022-08-23 | [15817](https://github.com/airbytehq/airbyte/pull/15817) | Improved unit test coverage |
| `0.1.6` | 2022-08-21 | [15824](https://github.com/airbytehq/airbyte/pull/15824) | Fix semi incremental streams: do not ignore start date, make one api call instead of multiple |
| `0.1.5` | 2022-08-16 | [15683](https://github.com/airbytehq/airbyte/pull/15683) | Retry failed creation of a job instead of skipping it |
| `0.1.4` | 2022-06-20 | [13930](https://github.com/airbytehq/airbyte/pull/13930) | Process failing creation of export jobs |
| `0.1.3` | 2021-12-10 | [8429](https://github.com/airbytehq/airbyte/pull/8578) | Updated titles and descriptions |
| `0.1.2` | 2021-12-03 | [8483](https://github.com/airbytehq/airbyte/pull/8483) | Improve field conversion to conform schema |
| `0.1.1` | 2021-11-29 | [0000](https://github.com/airbytehq/airbyte/pull/0000) | Fix timestamp value format issue |
| `0.1.0` | 2021-09-06 | [5863](https://github.com/airbytehq/airbyte/pull/5863) | Release Marketo CDK Connector |
| Version | Date | Pull Request | Subject |
|:---------|:-----------|:---------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------|
| `1.2.4` | 2024-01-08 | [33999](https://github.com/airbytehq/airbyte/pull/33999) | Fix for `Export daily quota exceeded` |
| `1.2.3` | 2023-08-02 | [28999](https://github.com/airbytehq/airbyte/pull/28999) | Fix for ` _csv.Error: line contains NUL` |
| `1.2.2` | 2023-10-19 | [31599](https://github.com/airbytehq/airbyte/pull/31599) | Base image migration: remove Dockerfile and use the python-connector-base image |
| `1.2.1` | 2023-09-18 | [30533](https://github.com/airbytehq/airbyte/pull/30533) | Fix `json_schema` for stream `Leads` |
| `1.2.0` | 2023-06-26 | [27726](https://github.com/airbytehq/airbyte/pull/27726) | License Update: Elv2 |
| `1.1.0` | 2023-04-18 | [23956](https://github.com/airbytehq/airbyte/pull/23956) | Add `Segmentations` Stream |
| `1.0.4` | 2023-04-25 | [25481](https://github.com/airbytehq/airbyte/pull/25481) | Minor fix for bug caused by `<=` producing additional API call when there is a single date slice |
| `1.0.3` | 2023-02-13 | [22938](https://github.com/airbytehq/airbyte/pull/22938) | Specified date formatting in specification |
| `1.0.2` | 2023-02-01 | [22203](https://github.com/airbytehq/airbyte/pull/22203) | Handle Null cursor values |
| `1.0.1` | 2023-01-31 | [22015](https://github.com/airbytehq/airbyte/pull/22015) | Set `AvailabilityStrategy` for streams explicitly to `None` |
| `1.0.0` | 2023-01-25 | [21790](https://github.com/airbytehq/airbyte/pull/21790) | Fix `activities_*` stream schemas |
| `0.1.12` | 2023-01-19 | [20973](https://github.com/airbytehq/airbyte/pull/20973) | Fix encoding error (note: this change is not in version 1.0.0, but is in later versions |
| `0.1.11` | 2022-09-30 | [17445](https://github.com/airbytehq/airbyte/pull/17445) | Do not use temporary files for memory optimization |
| `0.1.10` | 2022-09-30 | [17445](https://github.com/airbytehq/airbyte/pull/17445) | Optimize memory consumption |
| `0.1.9` | 2022-09-28 | [17304](https://github.com/airbytehq/airbyte/pull/17304) | Migrate to per-stream sate. |
| `0.1.7` | 2022-08-23 | [15817](https://github.com/airbytehq/airbyte/pull/15817) | Improved unit test coverage |
| `0.1.6` | 2022-08-21 | [15824](https://github.com/airbytehq/airbyte/pull/15824) | Fix semi incremental streams: do not ignore start date, make one api call instead of multiple |
| `0.1.5` | 2022-08-16 | [15683](https://github.com/airbytehq/airbyte/pull/15683) | Retry failed creation of a job instead of skipping it |
| `0.1.4` | 2022-06-20 | [13930](https://github.com/airbytehq/airbyte/pull/13930) | Process failing creation of export jobs |
| `0.1.3` | 2021-12-10 | [8429](https://github.com/airbytehq/airbyte/pull/8578) | Updated titles and descriptions |
| `0.1.2` | 2021-12-03 | [8483](https://github.com/airbytehq/airbyte/pull/8483) | Improve field conversion to conform schema |
| `0.1.1` | 2021-11-29 | [0000](https://github.com/airbytehq/airbyte/pull/0000) | Fix timestamp value format issue |
| `0.1.0` | 2021-09-06 | [5863](https://github.com/airbytehq/airbyte/pull/5863) | Release Marketo CDK Connector |