🎉 New Source: Strava #7151

terencecho · 2021-10-19T06:52:14Z

What

Add new source: Strava

How

Add two streams for this source:

Athlete Stats
Activities

Tests

Unit Tests

(.venv) ❰tcho❙~/code/forks/airbyte/airbyte-integrations/connectors/source-strava(git✱source-strava)❱✔≻ python3 -m pytest -s unit_tests
Test session starts (platform: darwin, Python 3.9.7, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/tcho/code/forks/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, mock-3.6.1, timeout-1.4.2
collecting ...
 ../../airbyte/airbyte-integrations/connectors/source-strava/unit_tests/test_incremental_streams.py::test_cursor_field ✓                                                                                                         7% ▋
 ../../airbyte/airbyte-integrations/connectors/source-strava/unit_tests/test_incremental_streams.py::test_get_updated_state ✓                                                                                                   13% █▍
 ../../airbyte/airbyte-integrations/connectors/source-strava/unit_tests/test_incremental_streams.py::test_stream_slices ✓                                                                                                       20% ██
 ../../airbyte/airbyte-integrations/connectors/source-strava/unit_tests/test_incremental_streams.py::test_supports_incremental ✓                                                                                                27% ██▋
 ../../airbyte/airbyte-integrations/connectors/source-strava/unit_tests/test_incremental_streams.py::test_source_defined_cursor ✓                                                                                               33% ███▍
 ../../airbyte/airbyte-integrations/connectors/source-strava/unit_tests/test_incremental_streams.py::test_stream_checkpoint_interval ✓                                                                                          40% ████
 airbyte-integrations/connectors/source-strava/unit_tests/test_streams.py::test_request_params ✓                                                                                                                                47% ████▋
 airbyte-integrations/connectors/source-strava/unit_tests/test_streams.py::test_next_page_token ✓                                                                                                                               53% █████▍
 airbyte-integrations/connectors/source-strava/unit_tests/test_streams.py::test_request_headers ✓                                                                                                                               60% ██████
 airbyte-integrations/connectors/source-strava/unit_tests/test_streams.py::test_http_method ✓                                                                                                                                   67% ██████▋
 airbyte-integrations/connectors/source-strava/unit_tests/test_streams.py::test_should_retry[HTTPStatus.OK-False] ✓                                                                                                             73% ███████▍
 airbyte-integrations/connectors/source-strava/unit_tests/test_streams.py::test_should_retry[HTTPStatus.BAD_REQUEST-False] ✓                                                                                                    80% ████████
 airbyte-integrations/connectors/source-strava/unit_tests/test_streams.py::test_should_retry[HTTPStatus.TOO_MANY_REQUESTS-True] ✓                                                                                               87% ████████▋
 airbyte-integrations/connectors/source-strava/unit_tests/test_streams.py::test_should_retry[HTTPStatus.INTERNAL_SERVER_ERROR-True] ✓                                                                                           93% █████████▍
 airbyte-integrations/connectors/source-strava/unit_tests/test_streams.py::test_backoff_time ✓                                                                                                                                 100% ██████████
============================================================================================================== warnings summary ==============================================================================================================
airbyte-integrations/connectors/source-strava/unit_tests/test_incremental_streams.py: 6 warnings
airbyte-integrations/connectors/source-strava/unit_tests/test_streams.py: 9 warnings
  /Users/tcho/code/forks/airbyte/airbyte-integrations/connectors/source-strava/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py:37: DeprecationWarning: Call to deprecated class NoAuth. (Set `authenticator=None` instead) -- Deprecated since version 0.1.20.
    self._authenticator = NoAuth()

airbyte-integrations/connectors/source-strava/unit_tests/test_incremental_streams.py: 6 warnings
airbyte-integrations/connectors/source-strava/unit_tests/test_streams.py: 9 warnings
  /Users/tcho/code/forks/airbyte/airbyte-integrations/connectors/source-strava/.venv/lib/python3.9/site-packages/deprecated/classic.py:173: DeprecationWarning: Call to deprecated class HttpAuthenticator. (Use requests.auth.AuthBase instead) -- Deprecated since version 0.1.20.
    return old_new1(cls, *args, **kwargs)

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (0.25s):
      15 passed

Integration Tests

(.venv) ❰tcho❙~/code/forks/airbyte/airbyte-integrations/connectors/source-strava(git✱source-strava)❱✔≻ python -m pytest -p integration_tests.acceptance
Test session starts (platform: darwin, Python 3.9.7, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/tcho/code/forks/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, mock-3.6.1, timeout-1.4.2
collecting ...
 ../../airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_match_expected[inputs0] ✓                                                                              7% ▊
 ../../airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_required[inputs0] ✓                                                                                   14% █▌
 ../../airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_optional[inputs0] ✓                                                                                   21% ██▎
 ../../airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_has_secret[inputs0] ✓                                                                                 29% ██▉
 ../../airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_secret_never_in_the_output[inputs0] ✓                                                                 36% ███▋
 ../../airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_oauth_flow_parameters[inputs0] ✓                                                                      43% ████▍
 ../../airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs0] ✓                                                                                50% █████
 ../../airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs1] ✓                                                                                57% █████▊
 ../../airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_discover[inputs0] ✓                                                                              64% ██████▌
 ../../airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_cursors_exist_in_schema[inputs0] ✓                                                       71% ███████▎
 ../../airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs0] ✓                                                                                  79% ███████▉
 ../../airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py::TestFullRefresh.test_sequential_reads[inputs0] ✓                                                            86% ████████▋
 ../../airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_two_sequential_reads[inputs0] ✓                                                         93% █████████▍
 ../../airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_state_with_abnormally_large_values[inputs0] s                                          100% ██████████
========================================================================================================== short test summary info ===========================================================================================================
SKIPPED [1] ../../../../../airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py:21: `future_state_path` not specified, skipping

Results (27.68s):
      13 passed
       1 skipped

Pre-merge Checklist

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

CLAassistant · 2021-10-19T06:52:17Z

All committers have signed the CLA.

terencecho · 2021-10-20T01:43:38Z

Am I supposed to generate a sourceDefinitionId to add to the connector index myself, in accordance to the last item on the pre-merge checklist for contributors? The documentation linked only refers to updating <uuid>.json files that already exists when updating connectors.

marcosmarxm · 2021-10-20T02:46:04Z

Am I supposed to generate a sourceDefinitionId to add to the connector index myself, in accordance to the last item on the pre-merge checklist for contributors? The documentation linked only refers to updating <uuid>.json files that already exists when updating connectors.

Yes.

airbyte-integrations/connectors/source-strava/source_strava/source.py

airbyte-integrations/connectors/source-strava/source_strava/spec.json

airbyte-integrations/connectors/source-strava/source_strava/source.py

marcosmarxm

Hi @terencecho I made some comments about the contribution, is it possible to share the integration credentials to test (you can send to me in Slack). I request to another member to review your code in more details.

If possible add the endpoint docs in each stream.

…n check

terencecho · 2021-10-20T03:50:29Z

@marcosmarxm Thank you for the comments! I will address your commends and share the integration credentials to you through slack.

If possible add the endpoint docs in each stream.

I added endpoint docs in comments underneath each stream class such that they look like:

class Activities(IncrementalStravaStream):
    """
    Returns the activities of an athlete.
    API Docs: https://developers.strava.com/docs/reference/#api-Activities-getLoggedInAthleteActivities
    Endpoint: https://www.strava.com/api/v3/athlete/activities
    """

Is there another place I should add that information to as well?

airbyte-config/init/src/main/resources/seed/source_definitions.yaml

airbyte-integrations/connectors/source-strava/acceptance-test-config.yml

airbyte-integrations/connectors/source-strava/source_strava/source.py

marcosmarxm · 2021-10-26T01:35:42Z

airbyte-integrations/connectors/source-strava/source_strava/schemas/activities.json

+    "athlete": {
+      "type": "object",
+      "properties": {
+        "MetaAthlete": {


remove this, is not saving the correct data where only need id/resource_state field.

I think you need to correct to all nested objects from this schema.

I can make those changes. The reason I added them in the first place is because the output of the api is an "array of SummaryActivity objects" along with the other nested objects.

airbyte-integrations/connectors/source-strava/source_strava/schemas/activities.json

marcosmarxm · 2021-10-26T02:16:09Z

Overall after those changes the connector is ready to merge 🎉

airbyte-integrations/connectors/source-strava/source_strava/schemas/TODO.md

…into source-strava

terencecho · 2021-10-26T02:30:28Z

@marcosmarxm Thanks for the extra comments. They should all be addressed now. Let me know if I understood any of your comments incorrectly, especially regarding the schema json files.

terencecho · 2021-10-26T02:48:05Z

All unit/integration tests are passing and got it running locally with no problems. should be good for final look over.

airbyte-integrations/connectors/source-strava/source_strava/source.py

marcosmarxm

LGTM awesome work @terencecho

marcosmarxm · 2021-10-26T02:58:00Z

@terencecho just to confirm, run ./gradlew format to format all files and check flake/linting.

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

terencecho · 2021-10-26T04:11:46Z

@marcosmarxm thanks for your help. I decided to mark most of the schema in activities.json to be optionally null, since it wasn't clear through the api documentation which ones were nullable. I also ran ./gradlew format and pushed all the formatting changes, so I think everything is good to go!

* Initial commit for new source: Strava * Update readme and invalid_config * Update PR number in docs * Update docs/integrations/README * Annotae spec with airbyte_secret * Change starting_after input to start_date and add try/catch connection check * Add connector to connector index * Address PR commends and separate streams * Set start_date as a required field * Remove nexted objects from schema * Remove schema TODO.md * Fix schema * Edit activites schema to allow for null values * Fix formatting Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> * Fix formatting * Update source_definitions.yaml Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

terencecho added 2 commits October 18, 2021 23:20

Initial commit for new source: Strava

9f8994b

Update readme and invalid_config

8eb4a3a

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Oct 19, 2021

octavia-squidington-iii added the community label Oct 19, 2021

terencecho added 2 commits October 19, 2021 00:26

Update PR number in docs

1ee9b70

Update docs/integrations/README

b665696

marcosmarxm added the hacktober label Oct 19, 2021

Annotae spec with airbyte_secret

8ab1747

terencecho marked this pull request as ready for review October 20, 2021 01:38

terencecho mentioned this pull request Oct 20, 2021

Source: Strava airbytehq/connector-contest#77

Closed

marcosmarxm added the zzm label Oct 20, 2021

marcosmarxm reviewed Oct 20, 2021

View reviewed changes

airbyte-integrations/connectors/source-strava/source_strava/source.py Outdated Show resolved Hide resolved

marcosmarxm reviewed Oct 20, 2021

View reviewed changes

airbyte-integrations/connectors/source-strava/source_strava/spec.json Outdated Show resolved Hide resolved

marcosmarxm reviewed Oct 20, 2021

View reviewed changes

airbyte-integrations/connectors/source-strava/source_strava/source.py Outdated Show resolved Hide resolved

marcosmarxm requested changes Oct 20, 2021

View reviewed changes

Change starting_after input to start_date and add try/catch connectio…

6269ab9

…n check

Add connector to connector index

451b941

gaart reviewed Oct 20, 2021

View reviewed changes

terencecho added 2 commits October 20, 2021 19:28

Address PR commends and separate streams

2a87b32

Set start_date as a required field

00b7b7f

terencecho requested review from marcosmarxm and gaart October 21, 2021 02:40

Merge branch 'master' into source-strava

6b3d759

marcosmarxm reviewed Oct 26, 2021

View reviewed changes

marcosmarxm requested changes Oct 26, 2021

View reviewed changes

airbyte-integrations/connectors/source-strava/source_strava/schemas/activities.json Outdated Show resolved Hide resolved

marcosmarxm reviewed Oct 26, 2021

View reviewed changes

airbyte-integrations/connectors/source-strava/source_strava/schemas/TODO.md Outdated Show resolved Hide resolved

terencecho added 3 commits October 25, 2021 19:23

Remove nexted objects from schema

a0eaa10

Merge branch 'source-strava' of https://github.com/terencecho/airbyte …

c6157bd

…into source-strava

Remove schema TODO.md

0465abf

terencecho requested a review from marcosmarxm October 26, 2021 02:29

Fix schema

88310cd

marcosmarxm reviewed Oct 26, 2021

View reviewed changes

airbyte-integrations/connectors/source-strava/source_strava/source.py Outdated Show resolved Hide resolved

marcosmarxm approved these changes Oct 26, 2021

View reviewed changes

terencecho and others added 3 commits October 25, 2021 20:28

Edit activites schema to allow for null values

fb57f16

Fix formatting

5173edd

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

Fix formatting

c3e5141

marcosmarxm added 2 commits October 26, 2021 16:59

Merge branch 'master' into source-strava

f862a39

Update source_definitions.yaml

927192c

marcosmarxm merged commit 1b1b52d into airbytehq:master Oct 26, 2021

terencecho deleted the source-strava branch October 27, 2021 07:10

jrhizor mentioned this pull request Nov 1, 2021

Bump Airbyte version from 0.30.23-alpha to 0.30.24-alpha #7532

Merged

marcosmarxm removed the hacktober label Oct 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New Source: Strava #7151

🎉 New Source: Strava #7151

terencecho commented Oct 19, 2021 •

edited

Loading

CLAassistant commented Oct 19, 2021 •

edited

Loading

terencecho commented Oct 20, 2021

marcosmarxm commented Oct 20, 2021

marcosmarxm left a comment

terencecho commented Oct 20, 2021

marcosmarxm Oct 26, 2021

marcosmarxm Oct 26, 2021

terencecho Oct 26, 2021

marcosmarxm commented Oct 26, 2021

terencecho commented Oct 26, 2021

terencecho commented Oct 26, 2021

marcosmarxm left a comment

marcosmarxm commented Oct 26, 2021

terencecho commented Oct 26, 2021

🎉 New Source: Strava #7151

🎉 New Source: Strava #7151

Conversation

terencecho commented Oct 19, 2021 • edited Loading

What

How

Recommended reading order

Tests

Pre-merge Checklist

Community member or Airbyter

Airbyter

CLAassistant commented Oct 19, 2021 • edited Loading

terencecho commented Oct 20, 2021

marcosmarxm commented Oct 20, 2021

marcosmarxm left a comment

Choose a reason for hiding this comment

terencecho commented Oct 20, 2021

marcosmarxm Oct 26, 2021

Choose a reason for hiding this comment

marcosmarxm Oct 26, 2021

Choose a reason for hiding this comment

terencecho Oct 26, 2021

Choose a reason for hiding this comment

marcosmarxm commented Oct 26, 2021

terencecho commented Oct 26, 2021

terencecho commented Oct 26, 2021

marcosmarxm left a comment

Choose a reason for hiding this comment

marcosmarxm commented Oct 26, 2021

terencecho commented Oct 26, 2021

terencecho commented Oct 19, 2021 •

edited

Loading

CLAassistant commented Oct 19, 2021 •

edited

Loading