diff --git a/airbyte-integrations/connectors/source-public-apis/Dockerfile b/airbyte-integrations/connectors/source-public-apis/Dockerfile index 9dc3b03c8017..88ad494451f2 100644 --- a/airbyte-integrations/connectors/source-public-apis/Dockerfile +++ b/airbyte-integrations/connectors/source-public-apis/Dockerfile @@ -1,4 +1,4 @@ -FROM python:3.9.13-alpine3.15 as base +FROM python:3.9.11-alpine3.15 as base # build and load all requirements FROM base as builder @@ -34,5 +34,5 @@ COPY source_public_apis ./source_public_apis ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py" ENTRYPOINT ["python", "/airbyte/integration_code/main.py"] -LABEL io.airbyte.version=0.1.0 +LABEL io.airbyte.version=0.2.0 LABEL io.airbyte.name=airbyte/source-public-apis diff --git a/airbyte-integrations/connectors/source-public-apis/README.md b/airbyte-integrations/connectors/source-public-apis/README.md index 7b6d5c541361..0cded4c3d65e 100644 --- a/airbyte-integrations/connectors/source-public-apis/README.md +++ b/airbyte-integrations/connectors/source-public-apis/README.md @@ -1,35 +1,10 @@ # Public Apis Source -This is the repository for the Public Apis source connector, written in Python. -For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/public-apis). +This is the repository for the Public Apis configuration based source connector. +For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/sources/public-apis). ## Local development -### Prerequisites -**To iterate on this connector, make sure to complete this prerequisites section.** - -#### Minimum Python version required `= 3.9.0` - -#### Build & Activate Virtual Environment and install dependencies -From this connector directory, create a virtual environment: -``` -python -m venv .venv -``` - -This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your -development environment of choice. To activate it from the terminal, run: -``` -source .venv/bin/activate -pip install -r requirements.txt -pip install '.[tests]' -``` -If you are in an IDE, follow your IDE's instructions to activate the virtualenv. - -Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is -used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`. -If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything -should work as you expect. - #### Building via Gradle You can also build the connector in Gradle. This is typically used in CI and not needed for your development workflow. @@ -39,7 +14,7 @@ To build using Gradle, from the Airbyte repository root, run: ``` #### Create credentials -**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/public-apis) +**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/public-apis) to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_public_apis/spec.yaml` file. Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information. See `integration_tests/sample_config.json` for a sample config file. @@ -47,14 +22,6 @@ See `integration_tests/sample_config.json` for a sample config file. **If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source public-apis test creds` and place them into `secrets/config.json`. -### Locally running the connector -``` -python main.py spec -python main.py check --config secrets/config.json -python main.py discover --config secrets/config.json -python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json -``` - ### Locally running the connector docker image #### Build @@ -79,32 +46,15 @@ docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-public-apis:dev discov docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-public-apis:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json ``` ## Testing -Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named. -First install test dependencies into your virtual environment: -``` -pip install .[tests] -``` -### Unit Tests -To run unit tests locally, from the connector directory run: -``` -python -m pytest unit_tests -``` -### Integration Tests -There are two types of integration tests: Acceptance Tests (Airbyte's test suite for all source connectors) and custom integration tests (which are specific to this connector). -#### Custom Integration tests -Place custom tests inside `integration_tests/` folder, then, from the connector root, run -``` -python -m pytest integration_tests -``` #### Acceptance Tests -Customize `acceptance-test-config.yml` file to configure tests. See [Connector Acceptance Tests](https://docs.airbyte.io/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information. +Customize `acceptance-test-config.yml` file to configure tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information. If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py. -To run your integration tests with acceptance tests, from the connector root, run + +To run your integration tests with Docker, run: ``` -python -m pytest integration_tests -p integration_tests.acceptance +./acceptance-test-docker.sh ``` -To run your integration tests with docker ### Using gradle to run tests All commands should be run from airbyte project root. diff --git a/airbyte-integrations/connectors/source-public-apis/unit_tests/__init__.py b/airbyte-integrations/connectors/source-public-apis/__init__.py similarity index 100% rename from airbyte-integrations/connectors/source-public-apis/unit_tests/__init__.py rename to airbyte-integrations/connectors/source-public-apis/__init__.py diff --git a/airbyte-integrations/connectors/source-public-apis/acceptance-test-config.yml b/airbyte-integrations/connectors/source-public-apis/acceptance-test-config.yml index 1628949d5750..8049d87c0eaa 100644 --- a/airbyte-integrations/connectors/source-public-apis/acceptance-test-config.yml +++ b/airbyte-integrations/connectors/source-public-apis/acceptance-test-config.yml @@ -10,7 +10,7 @@ acceptance_tests: - config_path: "secrets/config.json" status: "succeed" - config_path: "integration_tests/invalid_config.json" - status: "failed" + status: "succeed" discovery: tests: - config_path: "secrets/config.json" @@ -19,7 +19,7 @@ acceptance_tests: - config_path: "secrets/config.json" configured_catalog_path: "integration_tests/configured_catalog.json" empty_streams: [] - incremental: + incremental: bypass_reason: "This connector does not implement incremental sync" full_refresh: tests: diff --git a/airbyte-integrations/connectors/source-public-apis/acceptance-test-docker.sh b/airbyte-integrations/connectors/source-public-apis/acceptance-test-docker.sh old mode 100644 new mode 100755 index 5797d20fe9a7..b6d65deeccb4 --- a/airbyte-integrations/connectors/source-public-apis/acceptance-test-docker.sh +++ b/airbyte-integrations/connectors/source-public-apis/acceptance-test-docker.sh @@ -1,2 +1,3 @@ #!/usr/bin/env sh + source "$(git rev-parse --show-toplevel)/airbyte-integrations/bases/connector-acceptance-test/acceptance-test-docker.sh" diff --git a/airbyte-integrations/connectors/source-public-apis/bootstrap.md b/airbyte-integrations/connectors/source-public-apis/bootstrap.md deleted file mode 100644 index 57d2529e8494..000000000000 --- a/airbyte-integrations/connectors/source-public-apis/bootstrap.md +++ /dev/null @@ -1,19 +0,0 @@ -## Streams - -[Public APIs](https://api.publicapis.org/) is a REST API without authentication. Connector has the following streams, and none of them support incremental refresh. - -* [Services](https://api.publicapis.org#get-entries) -* [Categories](https://api.publicapis.org#get-categories) - - -## Pagination - -[Public APIs](https://api.publicapis.org/) uses NO pagination. - -## Properties - -The connector configuration includes NO properties as this is a publi API without authentication. - -## Authentication - -[Public APIs](https://api.publicapis.org/) is a REST API without authentication. \ No newline at end of file diff --git a/airbyte-integrations/connectors/source-public-apis/integration_tests/abnormal_state.json b/airbyte-integrations/connectors/source-public-apis/integration_tests/abnormal_state.json new file mode 100644 index 000000000000..52b0f2c2118f --- /dev/null +++ b/airbyte-integrations/connectors/source-public-apis/integration_tests/abnormal_state.json @@ -0,0 +1,5 @@ +{ + "todo-stream-name": { + "todo-field-name": "todo-abnormal-value" + } +} diff --git a/airbyte-integrations/connectors/source-public-apis/integration_tests/sample_state.json b/airbyte-integrations/connectors/source-public-apis/integration_tests/sample_state.json new file mode 100644 index 000000000000..3587e579822d --- /dev/null +++ b/airbyte-integrations/connectors/source-public-apis/integration_tests/sample_state.json @@ -0,0 +1,5 @@ +{ + "todo-stream-name": { + "todo-field-name": "value" + } +} diff --git a/airbyte-integrations/connectors/source-public-apis/metadata.yaml b/airbyte-integrations/connectors/source-public-apis/metadata.yaml index 8056a2a39b58..1ea225d61bbc 100644 --- a/airbyte-integrations/connectors/source-public-apis/metadata.yaml +++ b/airbyte-integrations/connectors/source-public-apis/metadata.yaml @@ -1,24 +1,29 @@ data: + allowedHosts: + hosts: + - "*" + registries: + cloud: + enabled: true + oss: + enabled: true ab_internal: ql: 200 sl: 100 connectorSubtype: api connectorType: source definitionId: a4617b39-3c14-44cd-a2eb-6e720f269235 - dockerImageTag: 0.1.0 + dockerImageTag: 0.2.0 dockerRepository: airbyte/source-public-apis documentationUrl: https://docs.airbyte.com/integrations/sources/public-apis githubIssueLabel: source-public-apis - icon: publicapi.svg + icon: public-apis.svg license: MIT - name: Public APIs - registries: - cloud: - enabled: true - oss: - enabled: true + name: Public Apis + releaseDate: TODO releaseStage: alpha supportLevel: community + documentationUrl: https://docs.airbyte.com/integrations/sources/public-apis tags: - - language:python + - language:lowcode metadataSpecVersion: "1.0" diff --git a/airbyte-integrations/connectors/source-public-apis/sample_files/config.json b/airbyte-integrations/connectors/source-public-apis/sample_files/config.json deleted file mode 100644 index 0967ef424bce..000000000000 --- a/airbyte-integrations/connectors/source-public-apis/sample_files/config.json +++ /dev/null @@ -1 +0,0 @@ -{} diff --git a/airbyte-integrations/connectors/source-public-apis/sample_files/configured_catalog.json b/airbyte-integrations/connectors/source-public-apis/sample_files/configured_catalog.json deleted file mode 100644 index 414979f92e93..000000000000 --- a/airbyte-integrations/connectors/source-public-apis/sample_files/configured_catalog.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "streams": [ - { - "stream": { - "name": "services", - "json_schema": {}, - "supported_sync_modes": ["full_refresh"] - }, - "sync_mode": "full_refresh", - "destination_sync_mode": "overwrite" - }, - { - "stream": { - "name": "categories", - "json_schema": {}, - "supported_sync_modes": ["full_refresh"] - }, - "sync_mode": "full_refresh", - "destination_sync_mode": "overwrite" - } - ] -} diff --git a/airbyte-integrations/connectors/source-public-apis/sample_files/invalid_config.json b/airbyte-integrations/connectors/source-public-apis/sample_files/invalid_config.json deleted file mode 100644 index 5a47475c1995..000000000000 --- a/airbyte-integrations/connectors/source-public-apis/sample_files/invalid_config.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "invalid": "config_key" -} diff --git a/airbyte-integrations/connectors/source-public-apis/setup.py b/airbyte-integrations/connectors/source-public-apis/setup.py index d4c33672b789..18d040a4e259 100644 --- a/airbyte-integrations/connectors/source-public-apis/setup.py +++ b/airbyte-integrations/connectors/source-public-apis/setup.py @@ -6,13 +6,14 @@ from setuptools import find_packages, setup MAIN_REQUIREMENTS = [ - "airbyte-cdk~=0.2", + "airbyte-cdk~=0.1", ] TEST_REQUIREMENTS = [ "requests-mock~=1.9.3", - "pytest~=6.1", + "pytest~=6.2", "pytest-mock~=3.6.1", + "connector-acceptance-test", ] setup( diff --git a/airbyte-integrations/connectors/source-public-apis/source_public_apis/components.py b/airbyte-integrations/connectors/source-public-apis/source_public_apis/components.py new file mode 100644 index 000000000000..22303d468013 --- /dev/null +++ b/airbyte-integrations/connectors/source-public-apis/source_public_apis/components.py @@ -0,0 +1,12 @@ +from typing import Any, Iterable, Mapping +from airbyte_cdk.sources.streams.http import HttpStream +from airbyte_cdk.sources.declarative.extractors.record_extractor import RecordExtractor +from typing import Any, List +import requests + + +class CustomExtractor(RecordExtractor): + + def extract_records(self, response: requests.Response, **kwargs) -> List[Mapping[str, Any]]: + + return [{"name": cat} for cat in response.json()["categories"]] diff --git a/airbyte-integrations/connectors/source-public-apis/source_public_apis/manifest.yaml b/airbyte-integrations/connectors/source-public-apis/source_public_apis/manifest.yaml new file mode 100644 index 000000000000..9a853522606b --- /dev/null +++ b/airbyte-integrations/connectors/source-public-apis/source_public_apis/manifest.yaml @@ -0,0 +1,58 @@ +version: "0.29.0" + +definitions: + selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: ["entries"] + + selector_categories: + type: RecordSelector + extractor: + class_name: source_public_apis.components.CustomExtractor + requester: + type: HttpRequester + url_base: "https://api.publicapis.org/" + http_method: "GET" + authenticator: + type: NoAuth + retriever: + type: SimpleRetriever + record_selector: + $ref: "#/definitions/selector" + paginator: + type: NoPagination + requester: + $ref: "#/definitions/requester" + base_stream: + type: DeclarativeStream + retriever: + $ref: "#/definitions/retriever" + + categories_stream: + $ref: "#/definitions/base_stream" + name: "categories" + primary_key: "name" + retriever: + $ref: "#/definitions/retriever" + record_selector: + $ref: "#/definitions/selector_categories" + $parameters: + path: "/categories" + + services_stream: + $ref: "#/definitions/base_stream" + name: "services" + primary_key: "API" + $parameters: + path: "/entries" + +streams: + - "#/definitions/categories_stream" + - "#/definitions/services_stream" + +check: + type: CheckStream + stream_names: + - "categories" diff --git a/airbyte-integrations/connectors/source-public-apis/source_public_apis/source.py b/airbyte-integrations/connectors/source-public-apis/source_public_apis/source.py index e1aa4a26ec9e..b9925483338d 100644 --- a/airbyte-integrations/connectors/source-public-apis/source_public_apis/source.py +++ b/airbyte-integrations/connectors/source-public-apis/source_public_apis/source.py @@ -2,84 +2,17 @@ # Copyright (c) 2023 Airbyte, Inc., all rights reserved. # +from airbyte_cdk.sources.declarative.yaml_declarative_source import YamlDeclarativeSource -from abc import ABC -from typing import Any, Iterable, List, Mapping, MutableMapping, Optional, Tuple +""" +This file provides the necessary constructs to interpret a provided declarative YAML configuration file into +source connector. -import requests -from airbyte_cdk.sources import AbstractSource -from airbyte_cdk.sources.streams import Stream -from airbyte_cdk.sources.streams.http import HttpStream -from airbyte_cdk.sources.streams.http.auth import NoAuth +WARNING: Do not modify this file. +""" -class PublicApisStream(HttpStream, ABC): - url_base = "https://api.publicapis.org/" - - def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]: - return None - - def request_params( - self, stream_state: Mapping[str, Any], stream_slice: Mapping[str, any] = None, next_page_token: Mapping[str, Any] = None - ) -> MutableMapping[str, Any]: - return {} - - def parse_response(self, response: requests.Response, **kwargs) -> Iterable[Mapping]: - yield {} - - -class Categories(PublicApisStream): - primary_key = "name" - - def path( - self, stream_state: Mapping[str, Any] = None, stream_slice: Mapping[str, Any] = None, next_page_token: Mapping[str, Any] = None - ) -> str: - return "categories" - - def parse_response( - self, - response: requests.Response, - stream_state: Mapping[str, Any], - stream_slice: Mapping[str, Any] = None, - next_page_token: Mapping[str, Any] = None, - ) -> Iterable[Mapping]: - return [{"name": cat} for cat in response.json()["categories"]] - - -class Services(PublicApisStream): - primary_key = "API" - - def path( - self, stream_state: Mapping[str, Any] = None, stream_slice: Mapping[str, Any] = None, next_page_token: Mapping[str, Any] = None - ) -> str: - return "entries" - - def parse_response( - self, - response: requests.Response, - stream_state: Mapping[str, Any], - stream_slice: Mapping[str, Any] = None, - next_page_token: Mapping[str, Any] = None, - ) -> Iterable[Mapping]: - return response.json()["entries"] - - -# Source -class SourcePublicApis(AbstractSource): - def check_connection(self, logger, config) -> Tuple[bool, any]: - """ - :param config: the user-input config object conforming to the connector's spec.yaml - :param logger: logger object - :return Tuple[bool, any]: (True, None) if the input config can be used to connect to the API successfully, (False, error) otherwise. - """ - if len(config) == 0: - return True, None - else: - return False, None - - def streams(self, config: Mapping[str, Any]) -> List[Stream]: - """ - :param config: A Mapping of the user input configuration as defined in the connector spec. - """ - auth = NoAuth() - return [Services(authenticator=auth), Categories(authenticator=auth)] +# Declarative Source +class SourcePublicApis(YamlDeclarativeSource): + def __init__(self): + super().__init__(**{"path_to_yaml": "manifest.yaml"}) diff --git a/airbyte-integrations/connectors/source-public-apis/unit_tests/test_source.py b/airbyte-integrations/connectors/source-public-apis/unit_tests/test_source.py deleted file mode 100644 index 5ede02ae6007..000000000000 --- a/airbyte-integrations/connectors/source-public-apis/unit_tests/test_source.py +++ /dev/null @@ -1,21 +0,0 @@ -# -# Copyright (c) 2023 Airbyte, Inc., all rights reserved. -# - -from unittest.mock import MagicMock - -from source_public_apis.source import SourcePublicApis - - -def test_check_connection(mocker): - source = SourcePublicApis() - logger_mock, config_mock = MagicMock(), MagicMock() - assert source.check_connection(logger_mock, config_mock) == (True, None) - - -def test_streams(mocker): - source = SourcePublicApis() - config_mock = MagicMock() - streams = source.streams(config_mock) - expected_streams_number = 2 - assert len(streams) == expected_streams_number diff --git a/airbyte-integrations/connectors/source-public-apis/unit_tests/test_streams.py b/airbyte-integrations/connectors/source-public-apis/unit_tests/test_streams.py deleted file mode 100644 index e17d2e8df843..000000000000 --- a/airbyte-integrations/connectors/source-public-apis/unit_tests/test_streams.py +++ /dev/null @@ -1,75 +0,0 @@ -# -# Copyright (c) 2023 Airbyte, Inc., all rights reserved. -# - -from http import HTTPStatus -from unittest.mock import MagicMock - -import pytest -from source_public_apis.source import PublicApisStream - - -@pytest.fixture -def patch_base_class(mocker): - # Mock abstract methods to enable instantiating abstract class - mocker.patch.object(PublicApisStream, "path", "v0/example_endpoint") - mocker.patch.object(PublicApisStream, "primary_key", "test_primary_key") - mocker.patch.object(PublicApisStream, "__abstractmethods__", set()) - - -def test_request_params(patch_base_class): - stream = PublicApisStream() - inputs = {"stream_slice": None, "stream_state": None, "next_page_token": None} - expected_params = {} - assert stream.request_params(**inputs) == expected_params - - -def test_next_page_token(patch_base_class): - stream = PublicApisStream() - inputs = {"response": MagicMock()} - expected_token = None - assert stream.next_page_token(**inputs) == expected_token - - -def test_parse_response(patch_base_class): - stream = PublicApisStream() - inputs = {"response": MagicMock()} - expected_parsed_object = {} - assert next(stream.parse_response(**inputs)) == expected_parsed_object - - -def test_request_headers(patch_base_class): - stream = PublicApisStream() - inputs = {"stream_slice": None, "stream_state": None, "next_page_token": None} - expected_headers = {} - assert stream.request_headers(**inputs) == expected_headers - - -def test_http_method(patch_base_class): - stream = PublicApisStream() - # TODO: replace this with your expected http request method - expected_method = "GET" - assert stream.http_method == expected_method - - -@pytest.mark.parametrize( - ("http_status", "should_retry"), - [ - (HTTPStatus.OK, False), - (HTTPStatus.BAD_REQUEST, False), - (HTTPStatus.TOO_MANY_REQUESTS, True), - (HTTPStatus.INTERNAL_SERVER_ERROR, True), - ], -) -def test_should_retry(patch_base_class, http_status, should_retry): - response_mock = MagicMock() - response_mock.status_code = http_status - stream = PublicApisStream() - assert stream.should_retry(response_mock) == should_retry - - -def test_backoff_time(patch_base_class): - response_mock = MagicMock() - stream = PublicApisStream() - expected_backoff_time = None - assert stream.backoff_time(response_mock) == expected_backoff_time diff --git a/docs/integrations/sources/public-apis.md b/docs/integrations/sources/public-apis.md index c4c9ebaff643..220e20405465 100644 --- a/docs/integrations/sources/public-apis.md +++ b/docs/integrations/sources/public-apis.md @@ -43,4 +43,5 @@ This source requires no setup. | Version | Date | Pull Request | Subject | | :--- | :--- | :--- | :--- | -| 0.1.0 | 2022-10-28 | [18471](https://github.com/airbytehq/airbyte/pull/18471) | Initial Release | +| 0.2.0 | 2023-06-15 | [29391](https://github.com/airbytehq/airbyte/pull/29391) | Migrated to Low Code | +| 0.1.0 | 2022-10-28 | [18471](https://github.com/airbytehq/airbyte/pull/18471) | Initial Release |