Skip to content

Commit

Permalink
✨Source News Api: Make Connector Compatible with Builder (#38635)
Browse files Browse the repository at this point in the history
  • Loading branch information
pabloescoder committed Jun 13, 2024
1 parent 09a5939 commit fb680ac
Show file tree
Hide file tree
Showing 14 changed files with 409 additions and 511 deletions.
6 changes: 0 additions & 6 deletions airbyte-integrations/connectors/source-news-api/.dockerignore

This file was deleted.

28 changes: 20 additions & 8 deletions airbyte-integrations/connectors/source-news-api/README.md
Original file line number Diff line number Diff line change
@@ -1,45 +1,52 @@
# News-Api source connector


This is the repository for the News-Api source connector, written in Python.
This is the repository for the News Api configuration based source connector.
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/sources/news-api).

## Local development

### Prerequisites
* Python (~=3.9)
* Poetry (~=1.7) - installation instructions [here](https://python-poetry.org/docs/#installation)

* Python (`^3.9`)
* Poetry (`^1.7`) - installation instructions [here](https://python-poetry.org/docs/#installation)



### Installing the connector

From this connector directory, run:
```bash
poetry install --with dev
```


### Create credentials

**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/news-api)
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_news_api/spec.yaml` file.
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `src/source_news_api/spec.yaml` file.
Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
See `sample_files/sample_config.json` for a sample config file.


### Locally running the connector

```
poetry run source-news-api spec
poetry run source-news-api check --config secrets/config.json
poetry run source-news-api discover --config secrets/config.json
poetry run source-news-api read --config secrets/config.json --catalog sample_files/configured_catalog.json
```

### Running unit tests
To run unit tests locally, from the connector directory run:
### Running tests

To run tests locally, from the connector directory run:

```
poetry run pytest unit_tests
poetry run pytest tests
```

### Building the docker image

1. Install [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md)
2. Run the following command to build the docker image:
```bash
Expand All @@ -50,6 +57,7 @@ An image will be available on your host with the tag `airbyte/source-news-api:de


### Running as a docker container

Then run any of the connector commands as follows:
```
docker run --rm airbyte/source-news-api:dev spec
Expand All @@ -59,16 +67,19 @@ docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integrat
```

### Running our CI test suite

You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
```bash
airbyte-ci connectors --name=source-news-api test
```

### Customizing acceptance Tests

Customize `acceptance-test-config.yml` file to configure acceptance tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.

### Dependency Management

All of your dependencies should be managed via Poetry.
To add a new dependency, run:
```bash
Expand All @@ -78,6 +89,7 @@ poetry add <package-name>
Please commit the changes to `pyproject.toml` and `poetry.lock` files.

## Publishing a new version of the connector

You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-news-api test`
2. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)):
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
# Copyright (c) 2024 Airbyte, Inc., all rights reserved.
#

This file was deleted.

2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-news-api/main.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
# Copyright (c) 2024 Airbyte, Inc., all rights reserved.
#

from source_news_api.run import run
Expand Down
32 changes: 18 additions & 14 deletions airbyte-integrations/connectors/source-news-api/metadata.yaml
Original file line number Diff line number Diff line change
@@ -1,31 +1,37 @@
data:
registries:
oss:
enabled: true
cloud:
enabled: false
remoteRegistries:
pypi:
enabled: true
packageName: airbyte-source-news-api
connectorBuildOptions:
# Please update to the latest version of the connector base image.
# https://hub.docker.com/r/airbyte/python-connector-base
# Please use the full address with sha256 hash to guarantee build reproducibility.
baseImage: docker.io/airbyte/python-connector-base:1.2.2@sha256:57703de3b4c4204bd68a7b13c9300f8e03c0189bffddaffc796f1da25d2dbea0
connectorSubtype: api
connectorType: source
definitionId: df38991e-f35b-4af2-996d-36817f614587
dockerImageTag: 0.1.3
dockerImageTag: 0.1.4
dockerRepository: airbyte/source-news-api
githubIssueLabel: source-news-api
icon: newsapi.svg
license: MIT
name: News API
remoteRegistries:
pypi:
enabled: true
packageName: airbyte-source-news-api
registries:
cloud:
enabled: false
oss:
enabled: true
name: News Api
releaseDate: 2022-10-21
releaseStage: alpha
supportLevel: community
documentationUrl: https://docs.airbyte.com/integrations/sources/news-api
tags:
- language:python
- cdk:low-code
ab_internal:
sl: 100
ql: 100
supportLevel: community
# Disabling acceptance tests for now
# - They are not passing
# - The connector has low/no Cloud usage
Expand All @@ -38,6 +44,4 @@ data:
# secretStore:
# type: GSM
# alias: airbyte-connector-testing-secret-store
connectorBuildOptions:
baseImage: docker.io/airbyte/python-connector-base:1.2.2@sha256:57703de3b4c4204bd68a7b13c9300f8e03c0189bffddaffc796f1da25d2dbea0
metadataSpecVersion: "1.0"
Loading

0 comments on commit fb680ac

Please sign in to comment.