Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hard to tell if API token is valid or not #222

Open
visch opened this issue Jul 15, 2023 · 1 comment
Open

Hard to tell if API token is valid or not #222

visch opened this issue Jul 15, 2023 · 1 comment

Comments

@visch
Copy link
Member

visch commented Jul 15, 2023

https://stackoverflow.com/questions/22438805/github-api-oauth-token-validation offers an option to check on the x-ratelimit-limit header to see if your API token is valid or not. Could be an interesting way to tell folks if their token is valid as right now it's not clear if the token doesn't have access to a specific endpoint or if the api token itself is just invalid.

From this slack thread https://meltano.slack.com/archives/C01TCRBBJD7/p1689349341108699
@visch
Copy link
Member Author

visch commented Mar 28, 2024 •
edited

Another example hit today.

To repeat

  1. Provide an invalid token
  2. You get an error like this
2024-03-28 09:33:10,071 | WARNING  | tap-github           | A token was dismissed. 401 Client Error: b'{"message":"Bad credentials","documentation_url":"https://docs.github.com/rest"}' (Reason: Unauthorized)
2024-03-28 09:33:10,071 | INFO     | tap-github           | Tap will run with 0 auth tokens
2024-03-28 09:33:10,072 | INFO     | tap-github           | No auth token detected. For higher rate limits, please specify `auth_token` in config.
2024-03-28 09:33:10,225 | INFO     | singer_sdk.metrics   | METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.153058, "tags": {"stream": "tempStream", "endpoint": "", "http_status_code": 403, "status": "failed"}}
2024-03-28 09:33:10,225 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "http_request_count", "value": 0, "tags": {"stream": "tempStream", "endpoint": "", "context": {}}}
2024-03-28 09:33:10,225 | ERROR    | tap-github           | An unhandled error occurred while syncing 'repositories'
Traceback (most recent call last):
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1179, in sync
    for _ in self._sync_records(context=context):
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1054, in _sync_records
    context_list = [context] if context is not None else self.partitions
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/repository_streams.py", line 175, in partitions
    augmented_repo_list += self.get_repo_ids(
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/repository_streams.py", line 119, in get_repo_ids
    for record in temp_stream.request_records({}):
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/rest.py", line 397, in request_records
    resp = decorated_request(prepared_request, context)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/rest.py", line 276, in _request
    self.validate_response(response)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/repository_streams.py", line 103, in validate_response
    super().validate_response(response)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/client.py", line 440, in validate_response
    super().validate_response(response)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/client.py", line 213, in validate_response
    self.authenticator.get_next_auth_token()
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/authenticator.py", line 214, in get_next_auth_token
    raise RuntimeError(
RuntimeError: All GitHub tokens have hit their rate limit. Stopping here.
Traceback (most recent call last):
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/bin/tap-github", line 8, in <module>
    sys.exit(cli())
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 500, in invoke
    tap.sync_all()
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 459, in sync_all
    stream.sync()
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1186, in sync
    raise ex
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1179, in sync
    for _ in self._sync_records(context=context):
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1054, in _sync_records
    context_list = [context] if context is not None else self.partitions
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/repository_streams.py", line 175, in partitions
    augmented_repo_list += self.get_repo_ids(
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/repository_streams.py", line 119, in get_repo_ids
    for record in temp_stream.request_records({}):
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/rest.py", line 397, in request_records
    resp = decorated_request(prepared_request, context)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/rest.py", line 276, in _request
    self.validate_response(response)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/repository_streams.py", line 103, in validate_response
    super().validate_response(response)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/client.py", line 440, in validate_response
    super().validate_response(response)
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/client.py", line 213, in validate_response
    self.authenticator.get_next_auth_token()
  File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/authenticator.py", line 214, in get_next_auth_token
    raise RuntimeError(
RuntimeError: All GitHub tokens have hit their rate limit. Stopping here.

This runtime error is highly misleading. The github token that got a 401 should most likely fail the tap (Unless there's some reason a 401 is given for something that's needed to retry). Maybe this is related to #243 ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
MeltanoLabs Overview
Status: No status
Milestone
No milestone
Development

No branches or pull requests
1 participant
@visch