🎉 New Source: Newsdata [low-code CDK] #18576

Xabilahu · 2022-10-27T23:12:20Z

What

New Source: Newsdata API. newsdata.io

Notes

I'm using a free API key, click here to get one for CI integration tests.

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Xabilahu · 2022-10-27T23:25:55Z

secrets/config.json being used for integ tests is:

{
  "api_key": "XXX",
  "country": ["fr"],
  "category": ["health"],
  "language": ["en"]
}

marcosmarxm · 2022-10-27T23:36:05Z

Thanks for the contribution @Xabilahu !

marcosmarxm

Some comments.

airbyte-integrations/connectors/source-newsdata/source_newsdata/spec.yaml

marcosmarxm · 2022-10-28T10:47:11Z

airbyte-integrations/connectors/source-newsdata/source_newsdata/newsdata.yaml

+    paginator:
+      type: NoPagination


@Xabilahu can you check the pagination for this connector?

Xabilahu · 2022-10-28T15:40:30Z

airbyte-integrations/connectors/source-newsdata/source_newsdata/newsdata.yaml

+      type: "CursorPagination"
+      cursor_value: "{{ response['nextPage'] }}"
+      page_size: 10
+      # page_size: !!int "{% if config['is_free_tier'] %}{{ 10 }}{% else %}{{ 50 }}{% endif %}"


The API sets the page_size dependent on the subscription you have.

Free tier gets page_size = 10

Any paid tier gets page_size = 50

The jinja interpolation works, but in this case, as it interpolates after !!int is executed, it fails to convert the string to an int. We need to cast it to int, as it is the type of page_size property. Any hint?

@girarda any advice here?

I would suggest setting the page_size in the spec instead of the is_free_tier field. I'll create an issue for the pagination strategy to handle dynamic page_size

issue: https://github.com/airbytehq/airbyte/issues/18783

@girarda, but wouldn't we still need string interpolation if handled in the spec?

yes, you're right. There's no way to dynamically set the page size today

Okey, so for now I will remove is_free_tier from spec.yaml, and set the page_size = 10, so that this connector works for both, free and paid tiers.

We will leave the dynamic page_size for a follow up.

@Xabilahu agree.

…supported

marcosmarxm · 2022-11-01T22:21:46Z

airbyte-integrations/connectors/source-newsdata/source_newsdata/newsdata.yaml

+  base_requester:
+    url_base: "https://newsdata.io/api/1"
+    http_method: "GET"
+    authenticator:
+      type: ApiKeyAuthenticator
+      header: "X-ACCESS-KEY"
+      api_token: "{{ config['api_key'] }}"


From the API documentation:

To obtain an API key you have to create an account. You get a free API key while you are in development. When making REST API requests, your API key has to be provided as a parameter in the URL (e.g. apikey=YOUR_API_KEY).

I tried to read data but no records were read.

{"type": "LOG", "log": {"level": "INFO", "message": "Read 0 records from latest stream"}}

Please check the authentication.

If the request would not have been authenticated, you would have gotten a 401 - Unauthorized response.

From Newsdata API documentation:

Your Newsdata.io API Key. Alternatively you can provide this via the X-ACCESS-KEY HTTP header.

Could you share your config.json?

marcosmarxm

@Xabilahu reading the API documentation I saw one endpoint News Archive where you can read all news from a specific date. This can be really valuable to users and you can implement a Incremental sync.

Xabilahu · 2022-11-02T09:57:44Z

@Xabilahu reading the API documentation I saw one endpoint News Archive where you can read all news from a specific date. This can be really valuable to users and you can implement a Incremental sync.

After taking a look, I remembered why I didn't implement support for that endpoint. It is because it is only accessible to paid tiers, which I don't have access to, and I wouldn't be able to test my implementation :( Same happens for the Crypto News endpoint...

Should I implement them anyways?

marcosmarxm · 2022-11-02T20:15:30Z

Not a problem!

marcosmarxm

Thanks @Xabilahu I'll publish later this week.

marcosmarxm · 2022-11-04T08:32:40Z

Hello! Your PR is approved but didn't have the time to publish and merge it this week. As you can check in Chris' comment all PRs submitted before 2-nov are eligible to win the prize. I'll be out of the office on Friday and return Monday to start publishing your contribution. Any question you can send a message in #hacktoberfest-2022 channel in Airbyte's Slack.

Have a good weekend and thank you for this amazing contribution for Hacktoberfest 🎉

Xabilahu · 2022-11-04T08:38:35Z

Hello! Your PR is approved but didn't have the time to publish and merge it this week. As you can check in Chris' comment all PRs submitted before 2-nov are eligible to win the prize. I'll be out of the office on Friday and return Monday to start publish it. Any question you can send a message in #hacktoberfest-2022 in Airbyte's Slack.

Have a good weekend and thank you for this amazing contribution for Hacktoberfest tada

More than happy to contribute ❤️ Have a nice weekend! 🥳

marcosmarxm · 2022-11-10T19:52:12Z

/test connector=connectors/source-newsdata

🕑 connectors/source-newsdata https://github.com/airbytehq/airbyte/actions/runs/3439969771
✅ connectors/source-newsdata https://github.com/airbytehq/airbyte/actions/runs/3439969771
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              345    110    68%   53, 64-72, 77-84, 88-89, 93-94, 178, 216-233, 242-250, 254-259, 265, 298-303, 341-348, 391-393, 396, 461-469, 481-484, 489, 545-546, 552, 555, 591-601, 614-639
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1508    348    77%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: This connector does not implement incremental sync
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:65: The previous connector image could not be retrieved.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:243: The previous connector image could not be retrieved.
======================== 24 passed, 3 skipped in 24.65s ========================

marcosmarxm · 2022-11-10T20:08:13Z

/publish connector=connectors/source-newsdata

🕑 Publishing the following connectors:
connectors/source-newsdata
https://github.com/airbytehq/airbyte/actions/runs/3440066583

Connector	Did it publish?	Were definitions generated?
connectors/source-newsdata	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

Xabilahu · 2022-11-10T20:29:36Z

Thanks for all the help @marcosmarxm 🥳😁

* Initial Newsdata source connector implementation * Update docs, source_definitions and upload icon * Set single input for `sources` stream * Remove unused definition * Update `spec.yaml` config * Support Pagination * Remove `is_free_tier`, because as of now, dynamic `page_size` is not supported * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

Xabilahu added 3 commits October 27, 2022 23:32

Initial Newsdata source connector implementation

c46d2e6

Merge branch 'airbytehq:master' into newsdata

24dd7c6

Merge branch 'airbytehq:master' into newsdata

81eccba

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Oct 27, 2022

octavia-squidington-iv added bounty community connectors/source/newsdata labels Oct 27, 2022

Xabilahu mentioned this pull request Oct 27, 2022

New Source: Newsdata airbytehq/connector-contest#205

Closed

Update docs, source_definitions and upload icon

802a0ce

marcosmarxm added the hacktober label Oct 28, 2022

Set single input for sources stream

8f29c2f

marcosmarxm requested changes Oct 28, 2022

View reviewed changes

Xabilahu added 2 commits October 28, 2022 12:44

Remove unused definition

9310d49

Merge branch 'master' into newsdata

ea1bbf5

marcosmarxm requested changes Oct 28, 2022

View reviewed changes

Xabilahu added 2 commits October 28, 2022 12:54

Update spec.yaml config

ffc999c

Support Pagination

eb19152

Xabilahu commented Oct 28, 2022

View reviewed changes

Xabilahu requested a review from marcosmarxm October 28, 2022 19:34

Remove is_free_tier, because as of now, dynamic page_size is not …

fecbe1e

…supported

marcosmarxm requested changes Nov 1, 2022

View reviewed changes

marcosmarxm approved these changes Nov 2, 2022

View reviewed changes

Merge branch 'master' into newsdata

7d94c18

auto-bump connector version

12225f1

marcosmarxm approved these changes Nov 10, 2022

View reviewed changes

Merge branch 'master' into newsdata

fd823ad

marcosmarxm merged commit c2cd11d into airbytehq:master Nov 10, 2022

Xabilahu deleted the newsdata branch November 10, 2022 20:29

RealChrisSean added the low-code label Nov 15, 2022

github-actions bot mentioned this pull request Nov 15, 2022

Bump helm chart version reference to 0.40.43 #19453

Closed

octavia-squidington-iii mentioned this pull request Nov 18, 2022

Bump Airbyte version from 0.40.18 to 0.40.19 #19579

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New Source: Newsdata [low-code CDK] #18576

🎉 New Source: Newsdata [low-code CDK] #18576

Xabilahu commented Oct 27, 2022

Xabilahu commented Oct 27, 2022

marcosmarxm commented Oct 27, 2022

marcosmarxm left a comment

marcosmarxm Oct 28, 2022

Xabilahu Oct 28, 2022 •

edited

marcosmarxm Nov 1, 2022

girarda Nov 1, 2022 •

edited

Xabilahu Nov 1, 2022

girarda Nov 1, 2022

Xabilahu Nov 1, 2022

marcosmarxm Nov 1, 2022

marcosmarxm Nov 1, 2022

marcosmarxm Nov 1, 2022

Xabilahu Nov 1, 2022 •

edited

marcosmarxm left a comment

Xabilahu commented Nov 2, 2022

marcosmarxm commented Nov 2, 2022

marcosmarxm left a comment

marcosmarxm commented Nov 4, 2022 •

edited

Xabilahu commented Nov 4, 2022

marcosmarxm commented Nov 10, 2022 •

edited by github-actions bot

marcosmarxm commented Nov 10, 2022 •

edited by github-actions bot

Xabilahu commented Nov 10, 2022

🎉 New Source: Newsdata [low-code CDK] #18576

🎉 New Source: Newsdata [low-code CDK] #18576

Conversation

Xabilahu commented Oct 27, 2022

What

Notes

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

Xabilahu commented Oct 27, 2022

marcosmarxm commented Oct 27, 2022

marcosmarxm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xabilahu Oct 28, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

girarda Nov 1, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xabilahu Nov 1, 2022 • edited

Choose a reason for hiding this comment

marcosmarxm left a comment

Choose a reason for hiding this comment

Xabilahu commented Nov 2, 2022

marcosmarxm commented Nov 2, 2022

marcosmarxm left a comment

Choose a reason for hiding this comment

marcosmarxm commented Nov 4, 2022 • edited

Xabilahu commented Nov 4, 2022

marcosmarxm commented Nov 10, 2022 • edited by github-actions bot

Build Passed

marcosmarxm commented Nov 10, 2022 • edited by github-actions bot

Xabilahu commented Nov 10, 2022

Xabilahu Oct 28, 2022 •

edited

girarda Nov 1, 2022 •

edited

Xabilahu Nov 1, 2022 •

edited

marcosmarxm commented Nov 4, 2022 •

edited

marcosmarxm commented Nov 10, 2022 •

edited by github-actions bot

marcosmarxm commented Nov 10, 2022 •

edited by github-actions bot