Skip to content

Commit

Permalink
Source Amazon SP: Update reports streams logics. (#5311)
Browse files Browse the repository at this point in the history
* Update check connection method

* #5796 silence printing full config when config validation fails (#5879)

* - #5796 silence printing full config when config validation fails

* fix unit tests after config validation check changes

Co-authored-by: Marcos Eliziario Santos <marcos@coremarcos.com>

* Format google-search-console schemas (#6047)

* Update ads_insights.json (#5946)

fix ads_insights schema according to [facebook docs](https://developers.facebook.com/docs/marketing-api/reference/adgroup/insights/) and my own data

* Bump connectors version + update docs (#6060)

* 🐛 Source Facebook Marketing: Convert values' types according to schema types (#4978)

* Convert values' types according to schema types

* Put streams back to `configured_catalog.json`

Put back `ads_insights` and `ads_insights_age_and_gender` streams.

* Pickup changes from #5946

* Implement change request + fix previous PR

* Update schema

* Remove items_type from convert_to_schema_types()

* Bump connectors version

* add oauth to connector_base dependencies (#6064)

* use spec when persisting source configs (#6036)

* switch most usages of writing sources to using specs

* fix other usages

* fix test

* only wait on the server in the scheduler, not the worker

* fix

* rephrase sanity check and remove stdout

* 🎉 Source Stripe: Add `PaymentIntents` stream (#6004)

* Add `PaymentIntents` stream

* Update docs

* Implement change request + few updates

Split `source.py` file into `source.py` and `streams.py` files.
Update `payment_intents.json` file.

* Bump connectors version + update docs

* Add skeleton for databricks destination (#5629)

Co-authored-by: Liren Tu <tuliren.git@outlook.com>
Co-authored-by: LiRen Tu <tuliren@gmail.com>

* Revert "Add skeleton for databricks destination (#5629)" (#6066)

This reverts commit 79256c4.

* 🎉 New Destination: Databricks (#5998)

Implement new destination connector for databricks delta lake.
Resolves #2075.

Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Source PostHog: add support for self-hosted instances (#6058)

* publish #6058 (#6059)

* Destination Kafka: correct spec json and data types in config (#6040)

* correct spec json and data types in config

* bump version

* correct tests

* correct config parser NPE

* format files

Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local>

* Fix or delete broken links (#6069)

* Fix more doc issues (#6072)

* 🎉 Added optional platform flag for build image script (#6000)

* Fix dependabot security alert. (#6073)

* Pin set value to greater than 4.0.1 to fix security warning.

* Format the rest of the connectors.

* add coverage report (#6045)

Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>

* Fix the format of the data returned by Google Ads oauth to match the config accepted by the connector (#6032)

* update salesforce docs (#6081)

* 🎉 Source Github: add caching for all streams (#5949)

* Source Github: add checking for all streams

* bump version, update changelogs

* Disable automatic migration acceptance test (#5988)

- The automatic migration acceptance test no longer works because of the new Flyway migration system.
- The file-based migration system is being deprecated.

* 🎉 CDK: Add requests native authenticator support (#5731)

* Add requests native auth class

* Update init file.
Update type annotations.
Bump version.

* Update TokenAuthenticator implementation.
Update Oauth2Authenticator implemetation.
Add CHANGELOG.md record.

* Update Oauth2Authenticator default value setting.
Update CHANGELOG.md

* Add requests native authenticator tests

* Add CDK requests native __call__ method tests.
Update CHANGELOG.md

* Add outdated auth deprication messages

* Update requests native auth __call__ method tests

* Bump CDK version to 0.1.20

* Interface changes to support separating secrets from the config (#6065)

* Interface changes to support separating secrets from the config
* Cleanup from PR comments and whitespace

* Update log message for empty env variable (#6115)

Co-authored-by: Jared Rhizor <jared@dataline.io>

* Bump Airbyte version from 0.29.17-alpha to 0.29.18-alpha (#6125)

Co-authored-by: davinchia <davinchia@users.noreply.github.com>

* return auth spec in the API when getting definition specification (#6121)

* Ignore python test coverage files (#6144)

* CDK: support nested refs resolving (#6044)

Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>

* feat: path for nested fields (#6130)

* feat: path for nested fields

* fix: clipRule error

* fix: remove field name

* Fix request middleware for ConnectionService (#6148)

* Jamakase/update onboarding flow (#5656)

* Doc explains normalization full-refresh implications (#6097)

* update docs

* add info in quickstart connection page

* update abhi comments

Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local>

* Fix migration validation issue (#6154)

Resolves #6151.

* Bump Airbyte version from 0.29.18-alpha to 0.29.19-alpha (#6156)

Co-authored-by: tuliren <tuliren@users.noreply.github.com>

* Add information on which destinations support Incremental - Deduped History in their docs (#6031)

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* Update Airbyte Spec acknowledgements. (#6155)

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* Update new integration request

* Add back the migration acceptance test (#6163)

* 🎉 Create a Helm Chart For Airbyte (#5891)

See number #1868. This creates an initial helm chart for installing Airbyte in Kubernetes to make it easier for users who are more familiar with helm. It also includes GitHub actions to help continually test that the chart works in the most basic case.
All of the templates are based off of the kustomize folder, but minio and postgres have been removed in favor of adding the bitnami helm charts as dependencies since they have an active community and allow easily tweaking their install.

* Fix OAuth Summary strings (#6143)

Co-authored-by: Marcos Eliziario Santos <eliziario@users.noreply.github.com>
Co-authored-by: Marcos Eliziario Santos <marcos@coremarcos.com>
Co-authored-by: oleh.zorenko <19872253+Zirochkaa@users.noreply.github.com>
Co-authored-by: Mauro <35332423+m-ronchi@users.noreply.github.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Jared Rhizor <jared@dataline.io>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Liren Tu <tuliren.git@outlook.com>
Co-authored-by: LiRen Tu <tuliren@gmail.com>
Co-authored-by: coeurdestenebres <90490546+coeurdestenebres@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local>
Co-authored-by: Harsha Teja Kanna <h7kanna@users.noreply.github.com>
Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: Dmytro <avida.d3@gmail.com>
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
Co-authored-by: Yevhenii <34103125+yevhenii-ldv@users.noreply.github.com>
Co-authored-by: Jenny Brown <85510829+airbyte-jenny@users.noreply.github.com>
Co-authored-by: davinchia <davinchia@users.noreply.github.com>
Co-authored-by: Iakov Salikov <36078770+isalikov@users.noreply.github.com>
Co-authored-by: Artem Astapenko <3767150+Jamakase@users.noreply.github.com>
Co-authored-by: tuliren <tuliren@users.noreply.github.com>
Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
Co-authored-by: Jonathan Stacks <jonstacks@users.noreply.github.com>
Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
  • Loading branch information
27 people committed Sep 17, 2021
1 parent b49e49c commit 092d54d
Show file tree
Hide file tree
Showing 289 changed files with 7,959 additions and 1,472 deletions.
2 changes: 1 addition & 1 deletion .bumpversion.cfg
@@ -1,5 +1,5 @@
[bumpversion]
current_version = 0.29.17-alpha
current_version = 0.29.19-alpha
commit = False
tag = False
parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)(\-[a-z]+)?
Expand Down
2 changes: 1 addition & 1 deletion .env
@@ -1,4 +1,4 @@
VERSION=0.29.17-alpha
VERSION=0.29.19-alpha

# Airbyte Internal Job Database, see https://docs.airbyte.io/operator-guides/configuring-airbyte-db
DATABASE_USER=docker
Expand Down
3 changes: 2 additions & 1 deletion .github/ISSUE_TEMPLATE/new-integration-request.md
Expand Up @@ -12,9 +12,10 @@ assignees: ''
* Do you need a specific version of the underlying data source e.g: you specifically need support for an older version of the API or DB?

## Describe the context around this new connector
* Which team in your company wants this integration, what for? This helps us understand the use case.
* Why do you need this integration? How does your team intend to use the data? This helps us understand the use case.
* How often do you want to run syncs?
* If this is an API source connector, which entities/endpoints do you need supported?
* If the connector is for a paid service, can we name you as a mutual user when we subscribe for an account? Which company should we name?

## Describe the alternative you are considering or using
What are you considering doing if you don’t have this integration through Airbyte?
Expand Down
74 changes: 74 additions & 0 deletions .github/workflows/helm.yaml
@@ -0,0 +1,74 @@
name: Helm
on:
push:
paths:
- '.github/workflows/helm.yaml'
- 'charts/**'
pull_request:
paths:
- '.github/workflows/helm.yaml'
- 'charts/**'
jobs:
lint:
name: Lint
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v2
- name: Setup Kubectl
uses: azure/setup-kubectl@v1
- name: Setup Helm
uses: azure/setup-helm@v1
with:
version: '3.6.3'
- name: Lint Chart
working-directory: ./charts/airbyte
run: ./ci.sh lint

generate-docs:
name: Generate Docs Parameters
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v2
- name: Checkout bitnami-labs/readme-generator-for-helm
uses: actions/checkout@v2
with:
repository: 'bitnami-labs/readme-generator-for-helm'
ref: '55cab5dd2191c4ffa7245cfefa428d4d9bb12730'
path: readme-generator-for-helm
- name: Install readme-generator-for-helm dependencies
working-directory: readme-generator-for-helm
run: npm install -g
- name: Test can update README with generated parameters
working-directory: charts/airbyte
run: ./ci.sh check-docs-updated

install:
name: Install
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v2
- name: Setup Kubectl
uses: azure/setup-kubectl@v1
- name: Setup Helm
uses: azure/setup-helm@v1
with:
version: '3.6.3'
- name: Setup Kind Cluster
uses: helm/kind-action@v1.2.0
with:
version: "v0.11.1"
image: "kindest/node:v1.21.1"
- name: Install airbyte chart
working-directory: ./charts/airbyte
run: ./ci.sh install
- if: always()
name: Print diagnostics
working-directory: ./charts/airbyte
run: ./ci.sh diagnostics
- if: success()
name: Test airbyte chart
working-directory: ./charts/airbyte
run: ./ci.sh test
1 change: 1 addition & 0 deletions .github/workflows/publish-command.yml
Expand Up @@ -178,6 +178,7 @@ jobs:
SOURCE_CLOSE_COM_CREDS: ${{ secrets.SOURCE_CLOSE_COM_CREDS }}
SOURCE_BAMBOO_HR_CREDS: ${{ secrets.SOURCE_BAMBOO_HR_CREDS }}
SOURCE_BIGCOMMERCE_CREDS: ${{ secrets.SOURCE_BIGCOMMERCE_CREDS }}
DESTINATION_DATABRICKS_CREDS: ${{ secrets.DESTINATION_DATABRICKS_CREDS }}
- run: |
echo "$SPEC_CACHE_SERVICE_ACCOUNT_KEY" > spec_cache_key_file.json && docker login -u airbytebot -p ${DOCKER_PASSWORD}
./tools/integrations/manage.sh publish airbyte-integrations/${{ github.event.inputs.connector }} ${{ github.event.inputs.run-tests }} --publish_spec_to_cache
Expand Down
12 changes: 12 additions & 0 deletions .github/workflows/test-command.yml
Expand Up @@ -173,6 +173,7 @@ jobs:
SOURCE_CLOSE_COM_CREDS: ${{ secrets.SOURCE_CLOSE_COM_CREDS }}
SOURCE_BAMBOO_HR_CREDS: ${{ secrets.SOURCE_BAMBOO_HR_CREDS }}
SOURCE_BIGCOMMERCE_CREDS: ${{ secrets.SOURCE_BIGCOMMERCE_CREDS }}
DESTINATION_DATABRICKS_CREDS: ${{ secrets.DESTINATION_DATABRICKS_CREDS }}
- run: |
./tools/bin/ci_integration_test.sh ${{ github.event.inputs.connector }}
name: test ${{ github.event.inputs.connector }}
Expand All @@ -194,6 +195,16 @@ jobs:
**/normalization_test_output/**/build/compiled/airbyte_utils/**
**/normalization_test_output/**/build/run/airbyte_utils/**
**/normalization_test_output/**/models/generated/**
- name: Test coverage reports artifacts
if: github.event.inputs.comment-id && success()
uses: actions/upload-artifact@v2
with:
name: test-reports
path: |
**/${{ github.event.inputs.connector }}/htmlcov/**
retention-days: 3

- name: Report Status
if: github.ref == 'refs/heads/master' && always()
run: ./tools/status/report.sh ${{ github.event.inputs.connector }} ${{github.repository}} ${{github.run_id}} ${{steps.test.outcome}}
Expand All @@ -208,6 +219,7 @@ jobs:
comment-id: ${{ github.event.inputs.comment-id }}
body: |
> :white_check_mark: ${{github.event.inputs.connector}} https://github.com/${{github.repository}}/actions/runs/${{github.run_id}}
${{env.PYTHON_UNITTEST_COVERAGE_REPORT}}
- name: Add Failure Comment
if: github.event.inputs.comment-id && failure()
uses: peter-evans/create-or-update-comment@v1
Expand Down
18 changes: 18 additions & 0 deletions .gitignore
Expand Up @@ -10,6 +10,9 @@ data
.project
.settings

# Logs
acceptance_tests_logs/

# Secrets
secrets
!airbyte-integrations/connector-templates/**/secrets
Expand All @@ -26,6 +29,21 @@ __pycache__
.ipynb_checkpoints
.pytest_

# Python unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# dbt
profiles.yml

Expand Down
38 changes: 36 additions & 2 deletions airbyte-api/src/main/openapi/config.yaml
Expand Up @@ -1234,7 +1234,7 @@ paths:
post:
tags:
- oauth
summary: Given a source def ID and optional workspaceID generate an access/refresh token etc.
summary: Given a source def ID generate an access/refresh token etc.
operationId: completeSourceOAuth
requestBody:
content:
Expand Down Expand Up @@ -1280,7 +1280,7 @@ paths:
post:
tags:
- oauth
summary:
summary: Given a destination def ID generate an access/refresh token etc.
operationId: completeDestinationOAuth
requestBody:
content:
Expand Down Expand Up @@ -1906,6 +1906,34 @@ components:
description: The specification for what values are required to configure the sourceDefinition.
type: object
example: { user: { type: string } }
SourceAuthSpecification:
$ref: "#/components/schemas/AuthSpecification"
AuthSpecification:
type: object
properties:
auth_type:
type: string
enum: ["oauth2.0"] # Future auth types should be added here
oauth2Specification:
"$ref": "#/components/schemas/OAuth2Specification"
OAuth2Specification:
description: An object containing any metadata needed to describe this connector's Oauth flow
type: object
properties:
oauthFlowInitParameters:
description:
"Pointers to the fields in the ConnectorSpecification which are needed to obtain the initial refresh/access tokens for the OAuth flow.
Each inner array represents the path in the ConnectorSpecification of the referenced field.
For example.
Assume the ConnectorSpecification contains params 'app_secret', 'app_id' which are needed to get the initial refresh token.
If they are not nested in the config, then the array would look like this [['app_secret'], ['app_id']]
If they are nested inside, say, an object called 'auth_params' then this array would be [['auth_params', 'app_secret'], ['auth_params', 'app_id']]"
type: array
items:
description: A list of strings which describes each parameter's path inside the ConnectionSpecification
type: array
items:
type: string
SourceDefinitionSpecificationRead:
type: object
required:
Expand All @@ -1918,6 +1946,8 @@ components:
type: string
connectionSpecification:
$ref: "#/components/schemas/SourceDefinitionSpecification"
authSpecification:
$ref: "#/components/schemas/SourceAuthSpecification"
jobInfo:
$ref: "#/components/schemas/SynchronousJobRead"
# SOURCE
Expand Down Expand Up @@ -2018,6 +2048,8 @@ components:
DestinationDefinitionId:
type: string
format: uuid
DestinationAuthSpecification:
$ref: "#/components/schemas/AuthSpecification"
DestinationDefinitionIdRequestBody:
type: object
required:
Expand Down Expand Up @@ -2101,6 +2133,8 @@ components:
type: string
connectionSpecification:
$ref: "#/components/schemas/DestinationDefinitionSpecification"
authSpecification:
$ref: "#/components/schemas/DestinationAuthSpecification"
jobInfo:
$ref: "#/components/schemas/SynchronousJobRead"
supportedDestinationSyncModes:
Expand Down
11 changes: 11 additions & 0 deletions airbyte-cdk/python/CHANGELOG.md
@@ -1,5 +1,16 @@
# Changelog

## 0.1.21
Resolve nested schema references and move external references to single schema definitions.

## 0.1.20
- Allow using `requests.auth.AuthBase` as authenticators instead of custom CDK authenticators.
- Implement Oauth2Authenticator, MultipleTokenAuthenticator and TokenAuthenticator authenticators.
- Add support for both legacy and requests native authenticator to HttpStream class.

## 0.1.19
No longer prints full config files on validation error to prevent exposing secrets to log file: https://github.com/airbytehq/airbyte/pull/5879

## 0.1.18
Fix incremental stream not saved state when internal limit config set.

Expand Down
Expand Up @@ -26,7 +26,10 @@
from abc import ABC, abstractmethod
from typing import Any, Mapping

from deprecated import deprecated


@deprecated(version="0.1.20", reason="Use requests.auth.AuthBase instead")
class HttpAuthenticator(ABC):
"""
Base abstract class for various HTTP Authentication strategies. Authentication strategies are generally
Expand All @@ -40,6 +43,7 @@ def get_auth_header(self) -> Mapping[str, Any]:
"""


@deprecated(version="0.1.20", reason="Set `authenticator=None` instead")
class NoAuth(HttpAuthenticator):
def get_auth_header(self) -> Mapping[str, Any]:
return {}
Expand Up @@ -27,10 +27,12 @@

import pendulum
import requests
from deprecated import deprecated

from .core import HttpAuthenticator


@deprecated(version="0.1.20", reason="Use airbyte_cdk.sources.streams.http.requests_native_auth.Oauth2Authenticator instead")
class Oauth2Authenticator(HttpAuthenticator):
"""
Generates OAuth2.0 access tokens from an OAuth2.0 refresh token and client credentials.
Expand Down
Expand Up @@ -26,9 +26,12 @@
from itertools import cycle
from typing import Any, List, Mapping

from deprecated import deprecated

from .core import HttpAuthenticator


@deprecated(version="0.1.20", reason="Use airbyte_cdk.sources.streams.http.requests_native_auth.TokenAuthenticator instead")
class TokenAuthenticator(HttpAuthenticator):
def __init__(self, token: str, auth_method: str = "Bearer", auth_header: str = "Authorization"):
self.auth_method = auth_method
Expand All @@ -39,6 +42,7 @@ def get_auth_header(self) -> Mapping[str, Any]:
return {self.auth_header: f"{self.auth_method} {self._token}"}


@deprecated(version="0.1.20", reason="Use airbyte_cdk.sources.streams.http.requests_native_auth.MultipleTokenAuthenticator instead")
class MultipleTokenAuthenticator(HttpAuthenticator):
def __init__(self, tokens: List[str], auth_method: str = "Bearer", auth_header: str = "Authorization"):
self.auth_method = auth_method
Expand Down
11 changes: 9 additions & 2 deletions airbyte-cdk/python/airbyte_cdk/sources/streams/http/http.py
Expand Up @@ -29,6 +29,7 @@
import requests
from airbyte_cdk.models import SyncMode
from airbyte_cdk.sources.streams.core import Stream
from requests.auth import AuthBase

from .auth.core import HttpAuthenticator, NoAuth
from .exceptions import DefaultBackoffException, RequestBodyException, UserDefinedBackoffException
Expand All @@ -46,10 +47,16 @@ class HttpStream(Stream, ABC):
source_defined_cursor = True # Most HTTP streams use a source defined cursor (i.e: the user can't configure it like on a SQL table)
page_size = None # Use this variable to define page size for API http requests with pagination support

def __init__(self, authenticator: HttpAuthenticator = NoAuth()):
self._authenticator = authenticator
# TODO: remove legacy HttpAuthenticator authenticator references
def __init__(self, authenticator: Union[AuthBase, HttpAuthenticator] = None):
self._session = requests.Session()

self._authenticator = NoAuth()
if isinstance(authenticator, AuthBase):
self._session.auth = authenticator
elif authenticator:
self._authenticator = authenticator

@property
@abstractmethod
def url_base(self) -> str:
Expand Down
@@ -0,0 +1,32 @@
#
# MIT License
#
# Copyright (c) 2020 Airbyte
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
#

from .oauth import Oauth2Authenticator
from .token import MultipleTokenAuthenticator, TokenAuthenticator

__all__ = [
"Oauth2Authenticator",
"TokenAuthenticator",
"MultipleTokenAuthenticator",
]

0 comments on commit 092d54d

Please sign in to comment.