Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🚨🚨🎉Source Pinterest: migrate to low-code #37698

Merged
merged 25 commits into from
May 28, 2024
Merged

🚨🚨🎉Source Pinterest: migrate to low-code #37698

Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
7f61757
Source Pinterest: migrate to low-code
Apr 30, 2024
77f31c5
Add custom authenticator
May 3, 2024
0e33e81
Update docs
May 7, 2024
d2eb57c
Merge branch 'master' into source-pinterest-migrate-to-low-code
May 7, 2024
ec0ff82
Fix auth
May 7, 2024
aea7e7b
Add semi incremental streams
May 7, 2024
fbb0d93
Add streams with nested parent streams
May 8, 2024
7d35cb0
Add analytics streams
May 8, 2024
d25651d
Merge branch 'master' into source-pinterest-migrate-to-low-code
May 10, 2024
f1d5def
Add analytics streams
May 10, 2024
f9cb4c0
Fix semi-incremental sync
May 13, 2024
b3ed284
Code cleanup
May 14, 2024
6f92640
Move schemas to manifest
May 14, 2024
780aece
Move spec to manifest
May 15, 2024
1fc9819
Update docs
May 15, 2024
64ab753
Add comments
May 16, 2024
b15cf3c
Remove unused component
May 21, 2024
333ee0f
Update error handlers
May 21, 2024
96baba6
Merge branch 'master' into source-pinterest-migrate-to-low-code
May 22, 2024
665adc8
Merge branch 'master' into source-pinterest-migrate-to-low-code
May 22, 2024
d8ed1ff
Merge branch 'master' into source-pinterest-migrate-to-low-code
May 23, 2024
f4eade0
Small updates
May 23, 2024
7ac3feb
Fix tests
May 23, 2024
ec17817
Update metadata
May 23, 2024
3b4b4f2
Update migration guide
May 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-pinterest/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ See `sample_files/sample_config.json` for a sample config file.
poetry run source-pinterest spec
poetry run source-pinterest check --config secrets/config.json
poetry run source-pinterest discover --config secrets/config.json
poetry run source-pinterest read --config secrets/config.json --catalog sample_files/configured_catalog.json
poetry run source-pinterest read --config secrets/config.json --catalog integration_tests/configured_catalog.json
```

### Running unit tests
Expand Down
121 changes: 114 additions & 7 deletions airbyte-integrations/connectors/source-pinterest/integration_tests/abnormal_state.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,12 @@
"type": "STREAM",
"stream": {
"stream_state": {
"DATE": "3021-06-09"
"states": [
{
"partition": { "id": "549761668032", "parent_slice": {} },
"cursor": { "date": "3021-06-09" }
}
]
},
"stream_descriptor": {
"name": "ad_account_analytics"
Expand All @@ -25,7 +30,12 @@
"type": "STREAM",
"stream": {
"stream_state": {
"updated_time": 9628867244
"states": [
{
"partition": { "id": "549761668032", "parent_slice": {} },
"cursor": { "updated_time": 9628867244 }
}
]
},
"stream_descriptor": {
"name": "campaigns"
Expand All @@ -36,7 +46,12 @@
"type": "STREAM",
"stream": {
"stream_state": {
"updated_time": 9628867244
"states": [
{
"partition": { "id": "549761668032", "parent_slice": {} },
"cursor": { "updated_time": 9628867244 }
}
]
},
"stream_descriptor": {
"name": "ad_groups"
Expand All @@ -47,7 +62,12 @@
"type": "STREAM",
"stream": {
"stream_state": {
"updated_time": 9628867244
"states": [
{
"partition": { "id": "549761668032", "parent_slice": {} },
"cursor": { "updated_time": 9628867244 }
}
]
},
"stream_descriptor": {
"name": "ads"
Expand All @@ -58,7 +78,36 @@
"type": "STREAM",
"stream": {
"stream_state": {
"DATE": "3021-06-09"
"states": [
{
"partition": {
"id": "626744128956",
"parent_slice": { "id": "549761668032" }
},
"cursor": { "DATE": "3021-06-09" }
},
{
"partition": {
"id": "626744128962",
"parent_slice": { "id": "549761668032" }
},
"cursor": { "DATE": "3021-06-09" }
},
{
"partition": {
"id": "626744128966",
"parent_slice": { "id": "549761668032" }
},
"cursor": { "DATE": "3021-06-09" }
},
{
"partition": {
"id": "626744128982",
"parent_slice": { "id": "549761668032" }
},
"cursor": { "DATE": "3021-06-09" }
}
]
},
"stream_descriptor": {
"name": "campaign_analytics"
Expand All @@ -69,7 +118,36 @@
"type": "STREAM",
"stream": {
"stream_state": {
"DATE": "3021-06-09"
"states": [
{
"partition": {
"id": "2680068678965",
"parent_slice": { "id": "549761668032" }
},
"cursor": { "DATE": "3021-06-09" }
},
{
"partition": {
"id": "2680068678971",
"parent_slice": { "id": "549761668032" }
},
"cursor": { "DATE": "3021-06-09" }
},
{
"partition": {
"id": "2680068678975",
"parent_slice": { "id": "549761668032" }
},
"cursor": { "DATE": "3021-06-09" }
},
{
"partition": {
"id": "2680068678993",
"parent_slice": { "id": "549761668032" }
},
"cursor": { "DATE": "3021-06-09" }
}
]
},
"stream_descriptor": {
"name": "ad_group_analytics"
Expand All @@ -80,7 +158,36 @@
"type": "STREAM",
"stream": {
"stream_state": {
"DATE": "3021-06-09"
"states": [
{
"partition": {
"id": "687218400118",
"parent_slice": { "id": "549761668032" }
},
"cursor": { "DATE": "3021-06-09" }
},
{
"partition": {
"id": "687218400125",
"parent_slice": { "id": "549761668032" }
},
"cursor": { "DATE": "3021-06-09" }
},
{
"partition": {
"id": "687218400132",
"parent_slice": { "id": "549761668032" }
},
"cursor": { "DATE": "3021-06-09" }
},
{
"partition": {
"id": "687218400210",
"parent_slice": { "id": "549761668032" }
},
"cursor": { "DATE": "3021-06-09" }
}
]
},
"stream_descriptor": {
"name": "ad_analytics"
Expand Down
17 changes: 15 additions & 2 deletions airbyte-integrations/connectors/source-pinterest/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ data:
connectorSubtype: api
connectorType: source
definitionId: 5cb7e5fe-38c2-11ec-8d3d-0242ac130003
dockerImageTag: 1.3.3
dockerImageTag: 2.0.0
dockerRepository: airbyte/source-pinterest
connectorBuildOptions:
baseImage: docker.io/airbyte/python-connector-base:1.1.0@sha256:bd98f6505c6764b1b5f99d3aedc23dfc9e9af631a62533f60eb32b1d3dbab20c
Expand All @@ -26,6 +26,19 @@ data:
releaseStage: generally_available
releases:
breakingChanges:
2.0.0:
message: "This release introduces updated format of state for incremental streams. Users will need to reset affected streams after upgrading."
askarpets marked this conversation as resolved.
Show resolved Hide resolved
upgradeDeadline: "2024-06-03"
scopedImpact:
- scopeType: stream
impactedScopes:
- ad_account_analytics
- ad_analytics
- ad_group_analytics
- ad_groups
- ads
- campaign_analytics
- campaigns
1.0.0:
message: "This release updates the date-time fields to use the Airbyte format `timestamp_without_timezone`. This change affects all streams where date-time fields are present, ensuring more accurate and standardized time representations: BoardPins, BoardSectionPins, Boards, Catalogs, and CatalogFeeds. Additionally, the stream names AdvertizerReport and AdvertizerTargetingReport have been renamed to AdvertiserReport and AdvertiserTargetingReport, respectively. Users will need to refresh the source schema and reset affected streams after upgrading."
upgradeDeadline: "2023-12-14"
Expand All @@ -43,7 +56,7 @@ data:
documentationUrl: https://docs.airbyte.com/integrations/sources/pinterest
tags:
- language:python
- cdk:python
- cdk:low-code
ab_internal:
sl: 200
ql: 400
Expand Down
Loading
Loading