TDL-5961 Support of custom domain by prijendev · Pull Request #172 · singer-io/tap-github

prijendev · 2022-07-15T09:16:00Z

Description of change

Added support of custom domain.
Added unit test case for the changes.

Note: Build is failing because config parameter base_url is missing.

Manual QA steps

Verify that if a custom domain is provided in the config then that domain is being used by tap to call the REST API.
Verify that if a custom domain is not provided in the config then that default domain is being used by tap to call the REST API.

Risks

Rollback steps

revert this branch

dbshah1212 · 2022-07-19T06:59:19Z

tap_github/streams.py

        """

        # build full url
+        self.url = client.base_url


It seems meaning less assignment, we can pass base_url variable in build_url function.

Removed url from stream class and passed client.base_url directly instead of self.url.

savan-chovatiya · 2022-07-19T08:57:55Z

tap_github/streams.py

        bookmark_time = singer.utils.strptime_to_utc(min_bookmark_value)

        # Build full url
+        self.url = client.domain


I think the client does not have any attribute name called domain

Removed unnecessary domain attribute from the class.

dbshah1212 · 2022-07-20T07:24:43Z

tap_github/client.py

        self.config = config
        self.session = requests.Session()
-        self.base_url = "https://api.github.com"
+        self.base_url = config.get('api_endpoint', DEFAULT_DOMAIN)


Suggested change

self.base_url = config.get('api_endpoint', DEFAULT_DOMAIN)

self.base_url = config.get('base_url', DEFAULT_DOMAIN)

Also need to update the readMe file as well.

Updated readme file and client class attribute self.base_url.

…ed' of https://github.com/singer-io/tap-github into TDL-5961-support-custom-domain

kspeer825

Could you add a basic tap-tester test that creates a connection with this property provided, then runs check and sync jobs.

prijendev · 2022-07-29T05:24:03Z

Could you add a basic tap-tester test that creates a connection with this property provided, then runs check and sync jobs.

As per your suggestion, I have updated the basic tap tester test to run sync jobs with the base_url(parameter for custom domain) property in config. Currently, the build is failing because of the missing property base_url.

tap_github/client.py

tap_github/streams.py

…ed' into TDL-5961-support-custom-domain

…ed' of https://github.com/singer-io/tap-github into TDL-5961-support-custom-domain

dbshah1212 · 2022-08-05T05:43:10Z

tests/test_github_sync.py

    def name():
        return "tap_tester_github_sync_test"

+    def get_properties(self):


Add code comment here

…inger-io/tap-github into TDL-5961-support-custom-domain

…ed' into TDL-5961-support-custom-domain

* added backoff for 5xx error and fixed indexerror when only passed org in config * added missing test cases and updated test cases according to code change * removed tags stream as it is not yet added * Updated inturrupted sync assertion * Removed bookmark conversion to utc * updated inerrupted sync assertions * resolved review comments * updated sreams in parent-child test case * Initial commit. * Removed unnecessary unit test cases. * updated code comments * Resolved pylint errors. * disabled pylint for get_child_records. * Added unittest cases. * Added filter params for commits and comments stream. * Updated unit test cases. * resolved review comments * resolved review comments * Updated start_date test * updated date format conversion in test * Updated bookmark logic for parent child streams. * Updated credentials check logic. * Updated unittest case. * Resolved PR comments and handled error for / in repository in config * added unittests for only / in config * Updated client.py to raise error for wrong org. * Added back creds check logic during object creation time. * handled the /repo and raised exception with a list * unskipped incremental child streams from assertion * Make pk of child automatic in parent stream. * updated unittests * Added logger message to print url for all streams. * updated base.py * Skipped 404 error for all streams. * Removed unnecessary __exit__ method from client * Updated doc link for each stream. * removed duplicates from extract_repos_from_config() and thrown warning message for duplicate repos * resolved PR comments and fixed pylint * Added __exit__ method in the Client object. * Added new line in the end of client.py * Updated schema for pr_commits. * resolved PR comments * Give warning message for invalid repo. * Added comments in the code. * resolved PR comments * resolved review comments * Raised error for invalid organization. * Updated logger message in client.py * Updated unittest case to use parameterized. * added a new TooManyRequests class error for 429 * resolved review comment * update variable name * resolved review comment and updated print - LOGGER * TDL 16573 sync teams at organization level (#173) * Initial commit. * Updated unittest case and resolved pylints. * Defined orgs streams in constant. * Updated extract_repo_from_config method. * Updated unit test cases. * Added back unittest cases. * TDL-19555 added missing test cases and updated test cases according to code change (#167) * added missing test cases and updated test cases according to code change * removed tags stream as it is not yet added * Updated inturrupted sync assertion * Removed bookmark conversion to utc * updated inerrupted sync assertions * resolved review comments * updated sreams in parent-child test case * resolved review comments * resolved review comments * Updated start_date test * updated date format conversion in test * unskipped incremental child streams from assertion * updated base.py * resolved review comments * resolved review comment * update variable name * resolved review comment and updated print - LOGGER * Reverted back changes of all field test case. Co-authored-by: NevilParikh14 <92399024+NevilParikh14@users.noreply.github.com> * TDL-5961 Support of custom domain (#172) * Initial commit. * Removed url from the stream class. * Updated readme and sample config. * Updated sync test to utilize custom domain config parameter `base_url`. * Updated test_stream unit test. * Removed duplicate comment. * Added support of empty string in custom domain. Co-authored-by: namrata270998 <namrata.brahmbhatt@crestdatasys.com> * TDL-12323 Implemented currently syncing for repos and streams (#171) * TDL-12323 Implemented currently syncing for repos and streams * resolved PR review comments * TDL-19980 fixed the bug * fixed the bug when any stream not selected, but the state writes currenctly_syncing * resolved PR comments * resolved pylint * fixed cci issue * fixed cci issues * fixing cci issue * resolved cci issues * fixed cci issue * made cci happy * updated one assertion and comments * Updated verify_access_for_repo method in client. * Resolved pylint. * Updated currently syncing for teams streams. (#174) Co-authored-by: namrata270998 <namrata.brahmbhatt@crestdatasys.com> Co-authored-by: NevilParikh14 <nevil.parikh@crestdatasys.com> Co-authored-by: NevilParikh14 <92399024+NevilParikh14@users.noreply.github.com> Co-authored-by: namrata270998 <75604662+namrata270998@users.noreply.github.com>

* added backoff for 5xx error and fixed indexerror when only passed org in config * added missing test cases and updated test cases according to code change * removed tags stream as it is not yet added * Updated inturrupted sync assertion * Removed bookmark conversion to utc * updated inerrupted sync assertions * resolved review comments * updated sreams in parent-child test case * Initial commit. * Removed unnecessary unit test cases. * updated code comments * Resolved pylint errors. * disabled pylint for get_child_records. * Added unittest cases. * Added filter params for commits and comments stream. * Updated unit test cases. * resolved review comments * resolved review comments * Updated start_date test * updated date format conversion in test * added missing fields * Updated bookmark logic for parent child streams. * Updated credentials check logic. * Updated unittest case. * Resolved PR comments and handled error for / in repository in config * added unittests for only / in config * Updated client.py to raise error for wrong org. * Added back creds check logic during object creation time. * removed extra formatting * improved formatting * handled the /repo and raised exception with a list * unskipped incremental child streams from assertion * Make pk of child automatic in parent stream. * updated unittests * Added logger message to print url for all streams. * updated base.py * Skipped 404 error for all streams. * Removed unnecessary __exit__ method from client * Updated doc link for each stream. * removed duplicates from extract_repos_from_config() and thrown warning message for duplicate repos * resolved PR comments and fixed pylint * resolved pr comments * Added __exit__ method in the Client object. * added more fields to shared schema * removed additionalproperties as duplicate * Added new line in the end of client.py * added user to shared schema * updated indentation * Resolved PR review comments * resolved pr comments * added more references * added ref for assignees * removed description * added ref schema to reactions * fixed type of total count in reactions * removed indentation from team_members and commits * Updated schema for pr_commits. * removed _sdc_repo from users.json * removed commits from shared schema and added the original schema back * resolved PR comments * Give warning message for invalid repo. * Added comments in the code. * updated indentation * updated ref * resolved PR comments * resolved review comments * Raised error for invalid organization. * Updated logger message in client.py * Updated unittest case to use parameterized. * added missing fields * removed shared schema for some fields * added missing fields * added missing fields * reverted the type of some fields * removed extra ',' from schema * added missing fields in the KNOWN_MISSING_FIELDS for all_fields * added a new TooManyRequests class error for 429 * added missing fields in the schema * added draft in missing fields * updated indentation * updated indentation level * removed extra fields from commit->committer * added author and stats in pr_commits * added misisng field in all_fields * updated the string schema type to obj as per api doc * resolved review comment * update variable name * added string in changed object type for backup * resolved review comment and updated print - LOGGER * used shared schema for performed_via_github_app and added ref back inissue_events stream * fixed typo * TDL-14359 added properties to issue assignees (#169) * added properties to issue assignees * updated issues schema * resolved review comments * added assignees in shared schema * updated assignee to user as same schema * deleted shared assignees * reverted back to number for some schemas * removed extra',' from schema * copied the base branch fields back to assignees schema * removed additionalProperties from user.json * updated indentation level * updated indentation level * TDL 16573 sync teams at organization level (#173) * Initial commit. * Updated unittest case and resolved pylints. * Defined orgs streams in constant. * Updated extract_repo_from_config method. * Updated unit test cases. * Added back unittest cases. * TDL-19555 added missing test cases and updated test cases according to code change (#167) * added missing test cases and updated test cases according to code change * removed tags stream as it is not yet added * Updated inturrupted sync assertion * Removed bookmark conversion to utc * updated inerrupted sync assertions * resolved review comments * updated sreams in parent-child test case * resolved review comments * resolved review comments * Updated start_date test * updated date format conversion in test * unskipped incremental child streams from assertion * updated base.py * resolved review comments * resolved review comment * update variable name * resolved review comment and updated print - LOGGER * Reverted back changes of all field test case. Co-authored-by: NevilParikh14 <92399024+NevilParikh14@users.noreply.github.com> * TDL-5961 Support of custom domain (#172) * Initial commit. * Removed url from the stream class. * Updated readme and sample config. * Updated sync test to utilize custom domain config parameter `base_url`. * Updated test_stream unit test. * Removed duplicate comment. * Added support of empty string in custom domain. Co-authored-by: namrata270998 <namrata.brahmbhatt@crestdatasys.com> * TDL-12323 Implemented currently syncing for repos and streams (#171) * TDL-12323 Implemented currently syncing for repos and streams * resolved PR review comments * TDL-19980 fixed the bug * fixed the bug when any stream not selected, but the state writes currenctly_syncing * resolved PR comments * resolved pylint * fixed cci issue * fixed cci issues * fixing cci issue * resolved cci issues * fixed cci issue * made cci happy * updated one assertion and comments * Updated verify_access_for_repo method in client. * Resolved pylint. * Updated currently syncing for teams streams. (#174) * made schema type in a single line * fixed cci issue * added date-time format for closed_at in issue_events.json Co-authored-by: NevilParikh14 <nevil.parikh@crestdatasys.com> Co-authored-by: prijendev <prijen.khokhani@crestdatasys.com> Co-authored-by: Prijen Khokhani <88327452+prijendev@users.noreply.github.com> Co-authored-by: NevilParikh14 <92399024+NevilParikh14@users.noreply.github.com>

* TDL-19555 added missing test cases and updated test cases according to code change (#167) * added missing test cases and updated test cases according to code change * removed tags stream as it is not yet added * Updated inturrupted sync assertion * Removed bookmark conversion to utc * updated inerrupted sync assertions * resolved review comments * updated sreams in parent-child test case * resolved review comments * resolved review comments * Updated start_date test * updated date format conversion in test * unskipped incremental child streams from assertion * updated base.py * resolved review comments * resolved review comment * update variable name * resolved review comment and updated print - LOGGER * TDL 19530 update dict based implementation to class based (#168) * added backoff for 5xx error and fixed indexerror when only passed org in config * added missing test cases and updated test cases according to code change * removed tags stream as it is not yet added * Updated inturrupted sync assertion * Removed bookmark conversion to utc * updated inerrupted sync assertions * resolved review comments * updated sreams in parent-child test case * Initial commit. * Removed unnecessary unit test cases. * updated code comments * Resolved pylint errors. * disabled pylint for get_child_records. * Added unittest cases. * Added filter params for commits and comments stream. * Updated unit test cases. * resolved review comments * resolved review comments * Updated start_date test * updated date format conversion in test * Updated bookmark logic for parent child streams. * Updated credentials check logic. * Updated unittest case. * Resolved PR comments and handled error for / in repository in config * added unittests for only / in config * Updated client.py to raise error for wrong org. * Added back creds check logic during object creation time. * handled the /repo and raised exception with a list * unskipped incremental child streams from assertion * Make pk of child automatic in parent stream. * updated unittests * Added logger message to print url for all streams. * updated base.py * Skipped 404 error for all streams. * Removed unnecessary __exit__ method from client * Updated doc link for each stream. * removed duplicates from extract_repos_from_config() and thrown warning message for duplicate repos * resolved PR comments and fixed pylint * Added __exit__ method in the Client object. * Added new line in the end of client.py * Updated schema for pr_commits. * resolved PR comments * Give warning message for invalid repo. * Added comments in the code. * resolved PR comments * resolved review comments * Raised error for invalid organization. * Updated logger message in client.py * Updated unittest case to use parameterized. * added a new TooManyRequests class error for 429 * resolved review comment * update variable name * resolved review comment and updated print - LOGGER * TDL 16573 sync teams at organization level (#173) * Initial commit. * Updated unittest case and resolved pylints. * Defined orgs streams in constant. * Updated extract_repo_from_config method. * Updated unit test cases. * Added back unittest cases. * TDL-19555 added missing test cases and updated test cases according to code change (#167) * added missing test cases and updated test cases according to code change * removed tags stream as it is not yet added * Updated inturrupted sync assertion * Removed bookmark conversion to utc * updated inerrupted sync assertions * resolved review comments * updated sreams in parent-child test case * resolved review comments * resolved review comments * Updated start_date test * updated date format conversion in test * unskipped incremental child streams from assertion * updated base.py * resolved review comments * resolved review comment * update variable name * resolved review comment and updated print - LOGGER * Reverted back changes of all field test case. Co-authored-by: NevilParikh14 <92399024+NevilParikh14@users.noreply.github.com> * TDL-5961 Support of custom domain (#172) * Initial commit. * Removed url from the stream class. * Updated readme and sample config. * Updated sync test to utilize custom domain config parameter `base_url`. * Updated test_stream unit test. * Removed duplicate comment. * Added support of empty string in custom domain. Co-authored-by: namrata270998 <namrata.brahmbhatt@crestdatasys.com> * TDL-12323 Implemented currently syncing for repos and streams (#171) * TDL-12323 Implemented currently syncing for repos and streams * resolved PR review comments * TDL-19980 fixed the bug * fixed the bug when any stream not selected, but the state writes currenctly_syncing * resolved PR comments * resolved pylint * fixed cci issue * fixed cci issues * fixing cci issue * resolved cci issues * fixed cci issue * made cci happy * updated one assertion and comments * Updated verify_access_for_repo method in client. * Resolved pylint. * Updated currently syncing for teams streams. (#174) Co-authored-by: namrata270998 <namrata.brahmbhatt@crestdatasys.com> Co-authored-by: NevilParikh14 <nevil.parikh@crestdatasys.com> Co-authored-by: NevilParikh14 <92399024+NevilParikh14@users.noreply.github.com> Co-authored-by: namrata270998 <75604662+namrata270998@users.noreply.github.com> * Tdl 20302 add query param for pull requests (#175) * Added query param for pull_requests streams. * Updated unit test for pull_requests. * Updated interrupted sync test case. * Updated tap-tester for interupted sync. * TDL-19526 added missing fields (#170) * added backoff for 5xx error and fixed indexerror when only passed org in config * added missing test cases and updated test cases according to code change * removed tags stream as it is not yet added * Updated inturrupted sync assertion * Removed bookmark conversion to utc * updated inerrupted sync assertions * resolved review comments * updated sreams in parent-child test case * Initial commit. * Removed unnecessary unit test cases. * updated code comments * Resolved pylint errors. * disabled pylint for get_child_records. * Added unittest cases. * Added filter params for commits and comments stream. * Updated unit test cases. * resolved review comments * resolved review comments * Updated start_date test * updated date format conversion in test * added missing fields * Updated bookmark logic for parent child streams. * Updated credentials check logic. * Updated unittest case. * Resolved PR comments and handled error for / in repository in config * added unittests for only / in config * Updated client.py to raise error for wrong org. * Added back creds check logic during object creation time. * removed extra formatting * improved formatting * handled the /repo and raised exception with a list * unskipped incremental child streams from assertion * Make pk of child automatic in parent stream. * updated unittests * Added logger message to print url for all streams. * updated base.py * Skipped 404 error for all streams. * Removed unnecessary __exit__ method from client * Updated doc link for each stream. * removed duplicates from extract_repos_from_config() and thrown warning message for duplicate repos * resolved PR comments and fixed pylint * resolved pr comments * Added __exit__ method in the Client object. * added more fields to shared schema * removed additionalproperties as duplicate * Added new line in the end of client.py * added user to shared schema * updated indentation * Resolved PR review comments * resolved pr comments * added more references * added ref for assignees * removed description * added ref schema to reactions * fixed type of total count in reactions * removed indentation from team_members and commits * Updated schema for pr_commits. * removed _sdc_repo from users.json * removed commits from shared schema and added the original schema back * resolved PR comments * Give warning message for invalid repo. * Added comments in the code. * updated indentation * updated ref * resolved PR comments * resolved review comments * Raised error for invalid organization. * Updated logger message in client.py * Updated unittest case to use parameterized. * added missing fields * removed shared schema for some fields * added missing fields * added missing fields * reverted the type of some fields * removed extra ',' from schema * added missing fields in the KNOWN_MISSING_FIELDS for all_fields * added a new TooManyRequests class error for 429 * added missing fields in the schema * added draft in missing fields * updated indentation * updated indentation level * removed extra fields from commit->committer * added author and stats in pr_commits * added misisng field in all_fields * updated the string schema type to obj as per api doc * resolved review comment * update variable name * added string in changed object type for backup * resolved review comment and updated print - LOGGER * used shared schema for performed_via_github_app and added ref back inissue_events stream * fixed typo * TDL-14359 added properties to issue assignees (#169) * added properties to issue assignees * updated issues schema * resolved review comments * added assignees in shared schema * updated assignee to user as same schema * deleted shared assignees * reverted back to number for some schemas * removed extra',' from schema * copied the base branch fields back to assignees schema * removed additionalProperties from user.json * updated indentation level * updated indentation level * TDL 16573 sync teams at organization level (#173) * Initial commit. * Updated unittest case and resolved pylints. * Defined orgs streams in constant. * Updated extract_repo_from_config method. * Updated unit test cases. * Added back unittest cases. * TDL-19555 added missing test cases and updated test cases according to code change (#167) * added missing test cases and updated test cases according to code change * removed tags stream as it is not yet added * Updated inturrupted sync assertion * Removed bookmark conversion to utc * updated inerrupted sync assertions * resolved review comments * updated sreams in parent-child test case * resolved review comments * resolved review comments * Updated start_date test * updated date format conversion in test * unskipped incremental child streams from assertion * updated base.py * resolved review comments * resolved review comment * update variable name * resolved review comment and updated print - LOGGER * Reverted back changes of all field test case. Co-authored-by: NevilParikh14 <92399024+NevilParikh14@users.noreply.github.com> * TDL-5961 Support of custom domain (#172) * Initial commit. * Removed url from the stream class. * Updated readme and sample config. * Updated sync test to utilize custom domain config parameter `base_url`. * Updated test_stream unit test. * Removed duplicate comment. * Added support of empty string in custom domain. Co-authored-by: namrata270998 <namrata.brahmbhatt@crestdatasys.com> * TDL-12323 Implemented currently syncing for repos and streams (#171) * TDL-12323 Implemented currently syncing for repos and streams * resolved PR review comments * TDL-19980 fixed the bug * fixed the bug when any stream not selected, but the state writes currenctly_syncing * resolved PR comments * resolved pylint * fixed cci issue * fixed cci issues * fixing cci issue * resolved cci issues * fixed cci issue * made cci happy * updated one assertion and comments * Updated verify_access_for_repo method in client. * Resolved pylint. * Updated currently syncing for teams streams. (#174) * made schema type in a single line * fixed cci issue * added date-time format for closed_at in issue_events.json Co-authored-by: NevilParikh14 <nevil.parikh@crestdatasys.com> Co-authored-by: prijendev <prijen.khokhani@crestdatasys.com> Co-authored-by: Prijen Khokhani <88327452+prijendev@users.noreply.github.com> Co-authored-by: NevilParikh14 <92399024+NevilParikh14@users.noreply.github.com> * TDL-20336 custom domain which are not created as a part of Github results into key error (#177) * Raise exception if `X-RateLimit-Reset` is not found in the header. * Updated comment in the unit test. * TDL 20339 handle empty bookmark if all repos unselected for next sync. (#178) * Handle empty bookmark if all existing repos are unselected. * Added detailed comments and unit test. * Added more comments. * Updated variable name and comments. * Updated comment. * Updated logic to return existing state. * Updated variable name and function name in unit test. * Removed duplicate field in the team_membership schema. Co-authored-by: NevilParikh14 <92399024+NevilParikh14@users.noreply.github.com> Co-authored-by: namrata270998 <namrata.brahmbhatt@crestdatasys.com> Co-authored-by: NevilParikh14 <nevil.parikh@crestdatasys.com> Co-authored-by: namrata270998 <75604662+namrata270998@users.noreply.github.com>

* TDL-19555 added missing test cases and updated test cases according to code change (singer-io#167) * added missing test cases and updated test cases according to code change * removed tags stream as it is not yet added * Updated inturrupted sync assertion * Removed bookmark conversion to utc * updated inerrupted sync assertions * resolved review comments * updated sreams in parent-child test case * resolved review comments * resolved review comments * Updated start_date test * updated date format conversion in test * unskipped incremental child streams from assertion * updated base.py * resolved review comments * resolved review comment * update variable name * resolved review comment and updated print - LOGGER * TDL 19530 update dict based implementation to class based (singer-io#168) * added backoff for 5xx error and fixed indexerror when only passed org in config * added missing test cases and updated test cases according to code change * removed tags stream as it is not yet added * Updated inturrupted sync assertion * Removed bookmark conversion to utc * updated inerrupted sync assertions * resolved review comments * updated sreams in parent-child test case * Initial commit. * Removed unnecessary unit test cases. * updated code comments * Resolved pylint errors. * disabled pylint for get_child_records. * Added unittest cases. * Added filter params for commits and comments stream. * Updated unit test cases. * resolved review comments * resolved review comments * Updated start_date test * updated date format conversion in test * Updated bookmark logic for parent child streams. * Updated credentials check logic. * Updated unittest case. * Resolved PR comments and handled error for / in repository in config * added unittests for only / in config * Updated client.py to raise error for wrong org. * Added back creds check logic during object creation time. * handled the /repo and raised exception with a list * unskipped incremental child streams from assertion * Make pk of child automatic in parent stream. * updated unittests * Added logger message to print url for all streams. * updated base.py * Skipped 404 error for all streams. * Removed unnecessary __exit__ method from client * Updated doc link for each stream. * removed duplicates from extract_repos_from_config() and thrown warning message for duplicate repos * resolved PR comments and fixed pylint * Added __exit__ method in the Client object. * Added new line in the end of client.py * Updated schema for pr_commits. * resolved PR comments * Give warning message for invalid repo. * Added comments in the code. * resolved PR comments * resolved review comments * Raised error for invalid organization. * Updated logger message in client.py * Updated unittest case to use parameterized. * added a new TooManyRequests class error for 429 * resolved review comment * update variable name * resolved review comment and updated print - LOGGER * TDL 16573 sync teams at organization level (singer-io#173) * Initial commit. * Updated unittest case and resolved pylints. * Defined orgs streams in constant. * Updated extract_repo_from_config method. * Updated unit test cases. * Added back unittest cases. * TDL-19555 added missing test cases and updated test cases according to code change (singer-io#167) * added missing test cases and updated test cases according to code change * removed tags stream as it is not yet added * Updated inturrupted sync assertion * Removed bookmark conversion to utc * updated inerrupted sync assertions * resolved review comments * updated sreams in parent-child test case * resolved review comments * resolved review comments * Updated start_date test * updated date format conversion in test * unskipped incremental child streams from assertion * updated base.py * resolved review comments * resolved review comment * update variable name * resolved review comment and updated print - LOGGER * Reverted back changes of all field test case. Co-authored-by: NevilParikh14 <92399024+NevilParikh14@users.noreply.github.com> * TDL-5961 Support of custom domain (singer-io#172) * Initial commit. * Removed url from the stream class. * Updated readme and sample config. * Updated sync test to utilize custom domain config parameter `base_url`. * Updated test_stream unit test. * Removed duplicate comment. * Added support of empty string in custom domain. Co-authored-by: namrata270998 <namrata.brahmbhatt@crestdatasys.com> * TDL-12323 Implemented currently syncing for repos and streams (singer-io#171) * TDL-12323 Implemented currently syncing for repos and streams * resolved PR review comments * TDL-19980 fixed the bug * fixed the bug when any stream not selected, but the state writes currenctly_syncing * resolved PR comments * resolved pylint * fixed cci issue * fixed cci issues * fixing cci issue * resolved cci issues * fixed cci issue * made cci happy * updated one assertion and comments * Updated verify_access_for_repo method in client. * Resolved pylint. * Updated currently syncing for teams streams. (singer-io#174) Co-authored-by: namrata270998 <namrata.brahmbhatt@crestdatasys.com> Co-authored-by: NevilParikh14 <nevil.parikh@crestdatasys.com> Co-authored-by: NevilParikh14 <92399024+NevilParikh14@users.noreply.github.com> Co-authored-by: namrata270998 <75604662+namrata270998@users.noreply.github.com> * Tdl 20302 add query param for pull requests (singer-io#175) * Added query param for pull_requests streams. * Updated unit test for pull_requests. * Updated interrupted sync test case. * Updated tap-tester for interupted sync. * TDL-19526 added missing fields (singer-io#170) * added backoff for 5xx error and fixed indexerror when only passed org in config * added missing test cases and updated test cases according to code change * removed tags stream as it is not yet added * Updated inturrupted sync assertion * Removed bookmark conversion to utc * updated inerrupted sync assertions * resolved review comments * updated sreams in parent-child test case * Initial commit. * Removed unnecessary unit test cases. * updated code comments * Resolved pylint errors. * disabled pylint for get_child_records. * Added unittest cases. * Added filter params for commits and comments stream. * Updated unit test cases. * resolved review comments * resolved review comments * Updated start_date test * updated date format conversion in test * added missing fields * Updated bookmark logic for parent child streams. * Updated credentials check logic. * Updated unittest case. * Resolved PR comments and handled error for / in repository in config * added unittests for only / in config * Updated client.py to raise error for wrong org. * Added back creds check logic during object creation time. * removed extra formatting * improved formatting * handled the /repo and raised exception with a list * unskipped incremental child streams from assertion * Make pk of child automatic in parent stream. * updated unittests * Added logger message to print url for all streams. * updated base.py * Skipped 404 error for all streams. * Removed unnecessary __exit__ method from client * Updated doc link for each stream. * removed duplicates from extract_repos_from_config() and thrown warning message for duplicate repos * resolved PR comments and fixed pylint * resolved pr comments * Added __exit__ method in the Client object. * added more fields to shared schema * removed additionalproperties as duplicate * Added new line in the end of client.py * added user to shared schema * updated indentation * Resolved PR review comments * resolved pr comments * added more references * added ref for assignees * removed description * added ref schema to reactions * fixed type of total count in reactions * removed indentation from team_members and commits * Updated schema for pr_commits. * removed _sdc_repo from users.json * removed commits from shared schema and added the original schema back * resolved PR comments * Give warning message for invalid repo. * Added comments in the code. * updated indentation * updated ref * resolved PR comments * resolved review comments * Raised error for invalid organization. * Updated logger message in client.py * Updated unittest case to use parameterized. * added missing fields * removed shared schema for some fields * added missing fields * added missing fields * reverted the type of some fields * removed extra ',' from schema * added missing fields in the KNOWN_MISSING_FIELDS for all_fields * added a new TooManyRequests class error for 429 * added missing fields in the schema * added draft in missing fields * updated indentation * updated indentation level * removed extra fields from commit->committer * added author and stats in pr_commits * added misisng field in all_fields * updated the string schema type to obj as per api doc * resolved review comment * update variable name * added string in changed object type for backup * resolved review comment and updated print - LOGGER * used shared schema for performed_via_github_app and added ref back inissue_events stream * fixed typo * TDL-14359 added properties to issue assignees (singer-io#169) * added properties to issue assignees * updated issues schema * resolved review comments * added assignees in shared schema * updated assignee to user as same schema * deleted shared assignees * reverted back to number for some schemas * removed extra',' from schema * copied the base branch fields back to assignees schema * removed additionalProperties from user.json * updated indentation level * updated indentation level * TDL 16573 sync teams at organization level (singer-io#173) * Initial commit. * Updated unittest case and resolved pylints. * Defined orgs streams in constant. * Updated extract_repo_from_config method. * Updated unit test cases. * Added back unittest cases. * TDL-19555 added missing test cases and updated test cases according to code change (singer-io#167) * added missing test cases and updated test cases according to code change * removed tags stream as it is not yet added * Updated inturrupted sync assertion * Removed bookmark conversion to utc * updated inerrupted sync assertions * resolved review comments * updated sreams in parent-child test case * resolved review comments * resolved review comments * Updated start_date test * updated date format conversion in test * unskipped incremental child streams from assertion * updated base.py * resolved review comments * resolved review comment * update variable name * resolved review comment and updated print - LOGGER * Reverted back changes of all field test case. Co-authored-by: NevilParikh14 <92399024+NevilParikh14@users.noreply.github.com> * TDL-5961 Support of custom domain (singer-io#172) * Initial commit. * Removed url from the stream class. * Updated readme and sample config. * Updated sync test to utilize custom domain config parameter `base_url`. * Updated test_stream unit test. * Removed duplicate comment. * Added support of empty string in custom domain. Co-authored-by: namrata270998 <namrata.brahmbhatt@crestdatasys.com> * TDL-12323 Implemented currently syncing for repos and streams (singer-io#171) * TDL-12323 Implemented currently syncing for repos and streams * resolved PR review comments * TDL-19980 fixed the bug * fixed the bug when any stream not selected, but the state writes currenctly_syncing * resolved PR comments * resolved pylint * fixed cci issue * fixed cci issues * fixing cci issue * resolved cci issues * fixed cci issue * made cci happy * updated one assertion and comments * Updated verify_access_for_repo method in client. * Resolved pylint. * Updated currently syncing for teams streams. (singer-io#174) * made schema type in a single line * fixed cci issue * added date-time format for closed_at in issue_events.json Co-authored-by: NevilParikh14 <nevil.parikh@crestdatasys.com> Co-authored-by: prijendev <prijen.khokhani@crestdatasys.com> Co-authored-by: Prijen Khokhani <88327452+prijendev@users.noreply.github.com> Co-authored-by: NevilParikh14 <92399024+NevilParikh14@users.noreply.github.com> * TDL-20336 custom domain which are not created as a part of Github results into key error (singer-io#177) * Raise exception if `X-RateLimit-Reset` is not found in the header. * Updated comment in the unit test. * TDL 20339 handle empty bookmark if all repos unselected for next sync. (singer-io#178) * Handle empty bookmark if all existing repos are unselected. * Added detailed comments and unit test. * Added more comments. * Updated variable name and comments. * Updated comment. * Updated logic to return existing state. * Updated variable name and function name in unit test. * Removed duplicate field in the team_membership schema. Co-authored-by: NevilParikh14 <92399024+NevilParikh14@users.noreply.github.com> Co-authored-by: namrata270998 <namrata.brahmbhatt@crestdatasys.com> Co-authored-by: NevilParikh14 <nevil.parikh@crestdatasys.com> Co-authored-by: namrata270998 <75604662+namrata270998@users.noreply.github.com>

Initial commit.

0b6c871

prijendev changed the title ~~Initial commit.~~ TDL- Support of custom domain Jul 15, 2022

prijendev changed the title ~~TDL- Support of custom domain~~ TDL-5961 Support of custom domain Jul 15, 2022

dbshah1212 reviewed Jul 19, 2022

View reviewed changes

savan-chovatiya reviewed Jul 19, 2022

View reviewed changes

Removed url from the stream class.

2476a6d

prijendev requested review from dbshah1212 and savan-chovatiya July 19, 2022 10:24

savan-chovatiya approved these changes Jul 19, 2022

View reviewed changes

dbshah1212 reviewed Jul 20, 2022

View reviewed changes

Updated readme and sample config.

4ce091a

prijendev requested a review from dbshah1212 July 20, 2022 09:18

dbshah1212 approved these changes Jul 21, 2022

View reviewed changes

Merge branch 'TDl-19530-update-dict-based-implementation-to-class-bas…

0ae68ec

…ed' of https://github.com/singer-io/tap-github into TDL-5961-support-custom-domain

prijendev requested review from KrisPersonal, RushiT0122 and kspeer825 July 26, 2022 12:30

kspeer825 suggested changes Jul 26, 2022

View reviewed changes

Updated sync test to utilize custom domain config parameter base_url.

ba73d8f

prijendev requested a review from kspeer825 July 29, 2022 05:24

RushiT0122 requested a review from vishalp-dev August 1, 2022 06:46

vishalp-dev reviewed Aug 1, 2022

View reviewed changes

tap_github/client.py Show resolved Hide resolved

vishalp-dev reviewed Aug 1, 2022

View reviewed changes

tap_github/streams.py Show resolved Hide resolved

prijendev added 3 commits August 1, 2022 18:34

Merge branch 'TDl-19530-update-dict-based-implementation-to-class-bas…

edb887f

…ed' into TDL-5961-support-custom-domain

Updated test_stream unit test.

106945c

Removed duplicate comment.

e2d0343

prijendev requested a review from vishalp-dev August 2, 2022 05:08

Merge branch 'TDl-19530-update-dict-based-implementation-to-class-bas…

10ff541

…ed' into TDL-5961-support-custom-domain

vishalp-dev approved these changes Aug 2, 2022

View reviewed changes

Merge branch 'TDl-19530-update-dict-based-implementation-to-class-bas…

5647e09

…ed' of https://github.com/singer-io/tap-github into TDL-5961-support-custom-domain

dbshah1212 reviewed Aug 5, 2022

View reviewed changes

tests/test_github_sync.py

def name():

return "tap_tester_github_sync_test"

def get_properties(self):

Copy link

dbshah1212 Aug 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add code comment here

prijendev added 3 commits August 5, 2022 13:17

Added support of empty string in custom domain.

54cf935

Merge branch 'TDL-5961-support-custom-domain' of https://github.com/s…

93276b4

…inger-io/tap-github into TDL-5961-support-custom-domain

Merge branch 'TDl-19530-update-dict-based-implementation-to-class-bas…

5191c02

…ed' into TDL-5961-support-custom-domain

prijendev merged commit 2b7cbd8 into TDl-19530-update-dict-based-implementation-to-class-based Aug 8, 2022

prijendev mentioned this pull request Aug 12, 2022

Crest master #176

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TDL-5961 Support of custom domain#172

TDL-5961 Support of custom domain#172
prijendev merged 13 commits intoTDl-19530-update-dict-based-implementation-to-class-basedfrom
TDL-5961-support-custom-domain

prijendev commented Jul 15, 2022 •

edited

Loading

Uh oh!

dbshah1212 Jul 19, 2022

Uh oh!

prijendev Jul 19, 2022

Uh oh!

savan-chovatiya Jul 19, 2022

Uh oh!

prijendev Jul 19, 2022

Uh oh!

dbshah1212 Jul 20, 2022

Uh oh!

dbshah1212 Jul 20, 2022

Uh oh!

prijendev Jul 20, 2022

Uh oh!

kspeer825 left a comment

Uh oh!

prijendev commented Jul 29, 2022 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

dbshah1212 Aug 5, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

	self.base_url = config.get('api_endpoint', DEFAULT_DOMAIN)
	self.base_url = config.get('base_url', DEFAULT_DOMAIN)

Conversation

prijendev commented Jul 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of change

Manual QA steps

Risks

Rollback steps

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kspeer825 left a comment

Choose a reason for hiding this comment

Uh oh!

prijendev commented Jul 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

prijendev commented Jul 15, 2022 •

edited

Loading

prijendev commented Jul 29, 2022 •

edited

Loading