🎉 Source GitHub: Add option to pull commits from user-specified branches #6223

yevhenii-ldv · 2021-09-17T16:52:25Z

What

copy of #5931.

How

Describe the solution

Pre-merge Checklist

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

…branch-specific value

…u/commits

yevhenii-ldv · 2021-09-17T16:52:47Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1246289836
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1246289836
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        45     45     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              158    109    31%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     25    39%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py      75     11    85%
	 ------------------------------------------------------------------------
	 TOTAL                                                  776    430    45%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                        Stmts   Miss  Cover
	 -----------------------------------------------
	 source_github/__init__.py       2      0   100%
	 source_github/source.py        68     48    29%
	 source_github/streams.py      373    207    45%
	 -----------------------------------------------
	 TOTAL                         443    255    42%

Zirochkaa

lgtm

airbyte-integrations/connectors/source-github/source_github/streams.py

airbyte-integrations/connectors/source-github/source_github/source.py

yevhenii-ldv · 2021-09-20T13:12:35Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1253661549
❌ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1253661549
🐛 https://gradle.com/s/s7hh2j2ulyuv2

yevhenii-ldv · 2021-09-20T13:20:48Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1253690285
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1253690285
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        45     45     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              158    109    31%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     25    39%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py      75     11    85%
	 ------------------------------------------------------------------------
	 TOTAL                                                  776    430    45%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                        Stmts   Miss  Cover
	 -----------------------------------------------
	 source_github/__init__.py       2      0   100%
	 source_github/source.py        68     48    29%
	 source_github/streams.py      367    210    43%
	 -----------------------------------------------
	 TOTAL                         437    258    41%

davinchia · 2021-09-21T10:16:26Z

airbyte-integrations/connectors/source-github/source_github/source.py

+            default_branches.update(
+                {
+                    repo_stats["full_name"]: repo_stats["default_branch"]
+                    for repo_stats in repository_stats_stream.read_records(sync_mode=SyncMode.full_refresh, stream_slice=stream_slice)


sorry can you explain what this for loop is doing?

We get the default branch for each of the repositories, so that if nothing is specified in the branches parameter in spec.json, then use it.

davinchia · 2021-09-21T10:35:16Z

airbyte-integrations/connectors/source-github/source_github/streams.py

+        if current_stream_state.get(current_repository):
+            repository_commits_state = current_stream_state[current_repository]
+            if repository_commits_state.get(self.cursor_field):
+                # transfer state from old source version to per-branch version
+                if current_branch == self.default_branches[current_repository]:
+                    state_value = max(latest_cursor_value, repository_commits_state[self.cursor_field])
+                    del repository_commits_state[self.cursor_field]
+            elif repository_commits_state.get(current_branch, {}).get(self.cursor_field):
+                state_value = max(latest_cursor_value, repository_commits_state[current_branch][self.cursor_field])
+        if current_repository not in current_stream_state:
+            current_stream_state[current_repository] = {}
+        current_stream_state[current_repository][current_branch] = {self.cursor_field: state_value}
+        return current_stream_state


Suggested change

if current_stream_state.get(current_repository):

repository_commits_state = current_stream_state[current_repository]

if repository_commits_state.get(self.cursor_field):

# transfer state from old source version to per-branch version

if current_branch == self.default_branches[current_repository]:

state_value = max(latest_cursor_value, repository_commits_state[self.cursor_field])

del repository_commits_state[self.cursor_field]

elif repository_commits_state.get(current_branch, {}).get(self.cursor_field):

state_value = max(latest_cursor_value, repository_commits_state[current_branch][self.cursor_field])

if current_repository not in current_stream_state:

current_stream_state[current_repository] = {}

current_stream_state[current_repository][current_branch] = {self.cursor_field: state_value}

return current_stream_state

if current_stream_state.get(current_repository):

repository_commits_state = current_stream_state[current_repository]

if repository_commits_state.get(self.cursor_field):

# transfer state from old source version to per-branch version

if current_branch == self.default_branches[current_repository]:

state_value = max(latest_cursor_value, repository_commits_state[self.cursor_field])

del repository_commits_state[self.cursor_field]

elif repository_commits_state.get(current_branch, {}).get(self.cursor_field):

state_value = max(latest_cursor_value, repository_commits_state[current_branch][self.cursor_field])

if current_repository not in current_stream_state:

current_stream_state[current_repository] = {}

current_stream_state[current_repository][current_branch] = {self.cursor_field: state_value}

return current_stream_state

davinchia · 2021-09-21T10:36:24Z

@yevhenii-ldv nice! I think you might be missing a dockerfile update, otherwise looks good.

yevhenii-ldv · 2021-09-22T08:42:28Z

/publish connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1261036511
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1261036511

cjwooo and others added 7 commits September 8, 2021 13:56

Add option to pull commits from user-specified branches

66e5508

Address comments part 1

f427d41

Fix Repositories stream error when repo is not part of an org

a0290da

Make compatible with old state version and fix request_params to use …

ccfaa2e

…branch-specific value

update contributors code

bde1bb1

Merge branch 'master' of https://github.com/airbytehq/airbyte into cw…

858f290

…u/commits

update changelogs

403acf3

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Sep 17, 2021

format code

02259b9

yevhenii-ldv temporarily deployed to more-secrets September 17, 2021 16:56 Inactive

yevhenii-ldv requested review from Zirochkaa, avida and htrueman September 17, 2021 17:07

yevhenii-ldv added this to the Connectors, October 1st 2021 milestone Sep 17, 2021

Zirochkaa approved these changes Sep 19, 2021

View reviewed changes

update code after review

e32fcda

yevhenii-ldv temporarily deployed to more-secrets September 20, 2021 13:07 Inactive

Merge master

99227b1

yevhenii-ldv temporarily deployed to more-secrets September 20, 2021 13:13 Inactive

jrhizor temporarily deployed to more-secrets September 20, 2021 13:14 Inactive

jrhizor temporarily deployed to more-secrets September 20, 2021 13:23 Inactive

yevhenii-ldv requested review from sherifnada, davinchia, tuliren and Phlair and removed request for tuliren September 20, 2021 13:28

davinchia approved these changes Sep 21, 2021

View reviewed changes

bump version, update changelog

9bbdd8d

yevhenii-ldv temporarily deployed to more-secrets September 22, 2021 08:43 Inactive

jrhizor temporarily deployed to more-secrets September 22, 2021 08:44 Inactive

update docs for Github

65c0289

yevhenii-ldv temporarily deployed to more-secrets September 22, 2021 08:55 Inactive

update docs

342ff84

yevhenii-ldv merged commit 1644f90 into master Sep 22, 2021

yevhenii-ldv deleted the cwu/commits branch September 22, 2021 08:56

yevhenii-ldv mentioned this pull request Sep 22, 2021

Source GitHub: Add option to pull commits from user-specified branches #5931

Closed

38 tasks

yevhenii-ldv temporarily deployed to more-secrets September 22, 2021 08:57 Inactive

yevhenii-ldv self-assigned this Sep 23, 2021

jrhizor mentioned this pull request Sep 25, 2021

Bump Airbyte version from 0.29.21-alpha to 0.29.22-alpha #6450

Merged

igrankova added connectors/sources-api connectors/source/github labels Jan 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source GitHub: Add option to pull commits from user-specified branches #6223

🎉 Source GitHub: Add option to pull commits from user-specified branches #6223

yevhenii-ldv commented Sep 17, 2021

yevhenii-ldv commented Sep 17, 2021 •

edited by github-actions bot

Zirochkaa left a comment

yevhenii-ldv commented Sep 20, 2021 •

edited by github-actions bot

yevhenii-ldv commented Sep 20, 2021 •

edited by github-actions bot

davinchia Sep 21, 2021

yevhenii-ldv Sep 22, 2021

davinchia Sep 21, 2021

davinchia commented Sep 21, 2021

yevhenii-ldv commented Sep 22, 2021 •

edited by github-actions bot

🎉 Source GitHub: Add option to pull commits from user-specified branches #6223

🎉 Source GitHub: Add option to pull commits from user-specified branches #6223

Conversation

yevhenii-ldv commented Sep 17, 2021

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

yevhenii-ldv commented Sep 17, 2021 • edited by github-actions bot

Zirochkaa left a comment

Choose a reason for hiding this comment

yevhenii-ldv commented Sep 20, 2021 • edited by github-actions bot

yevhenii-ldv commented Sep 20, 2021 • edited by github-actions bot

davinchia Sep 21, 2021

Choose a reason for hiding this comment

yevhenii-ldv Sep 22, 2021

Choose a reason for hiding this comment

davinchia Sep 21, 2021

Choose a reason for hiding this comment

davinchia commented Sep 21, 2021

yevhenii-ldv commented Sep 22, 2021 • edited by github-actions bot

yevhenii-ldv commented Sep 17, 2021 •

edited by github-actions bot

yevhenii-ldv commented Sep 20, 2021 •

edited by github-actions bot

yevhenii-ldv commented Sep 20, 2021 •

edited by github-actions bot

yevhenii-ldv commented Sep 22, 2021 •

edited by github-actions bot