-
Notifications
You must be signed in to change notification settings - Fork 30
Open
Labels
manifest-only feature gapsMissing capabilities that prevent connectors from being fully declarative (manifest-only)Missing capabilities that prevent connectors from being fully declarative (manifest-only)
Description
Problem
GitHub Apps provide higher rate limits than Personal Access Tokens (PATs), but implementing GitHub App authentication requires custom Python code. This prevents connectors from being fully declarative (manifest-only).
Rate Limit Comparison
- PAT: 5,000 requests/hour
- OAuth App: 5,000 requests/hour (15,000 if owned by Enterprise Cloud org)
- GitHub App (non-Enterprise): 5,000-12,500 requests/hour (scales with repos/users)
- GitHub App (Enterprise Cloud): 15,000 requests/hour (3x PAT limit)
Current Workaround
Connectors like source-github only support PAT and OAuth authentication. See source-github/spec.json:16-65 which only exposes OAuth and PAT options.
GitHub App authentication is not supported because it requires a two-step flow:
- Sign a JWT using App ID + private key
- Exchange JWT for installation access token via
POST /app/installations/{installation_id}/access_tokens - Cache and refresh the installation token before ~1 hour expiry
Proposed Solution
Option 1: Extend SessionTokenAuthenticator to support GitHub App flow
- Use
JwtAuthenticatorfor the JWT signing step - Configure
login_requesterto call the installation token endpoint with JWT header - Set
expiration_durationto ~1 hour for token caching
Option 2: Create dedicated GitHubAppAuthenticator component
- Accepts
app_id,installation_id, andprivate_key - Handles JWT signing and installation token exchange internally
- Automatically refreshes tokens before expiry
Example Configuration (Option 1)
authenticator:
type: SessionTokenAuthenticator
login_requester:
type: HttpRequester
url_base: "https://api.github.com"
path: "/app/installations/{{ config.installation_id }}/access_tokens"
http_method: "POST"
authenticator:
type: JwtAuthenticator
secret_key: "{{ config.private_key }}"
algorithm: "RS256"
token_duration: 600 # 10 minutes
jwt_payload:
iss: "{{ config.app_id }}"
iat: "{{ now_utc() }}"
exp: "{{ now_utc() + 600 }}"
session_token_path: ["token"]
expiration_duration: "PT1H" # 1 hour
request_authentication:
type: BearerAuthenticator
api_token: "{{ session_token }}"Example Configuration (Option 2)
authenticator:
type: GitHubAppAuthenticator
app_id: "{{ config.app_id }}"
installation_id: "{{ config.installation_id }}"
private_key: "{{ config.private_key }}"Impact
This would enable connectors like source-github, source-gitlab (which also supports GitLab Apps), and others to leverage higher rate limits without custom Python code.
Related
- Survey of Manifest-Only Connectors Using Custom Components (Feature Gaps) #714 - Survey showing 74.3% of certified manifest-only connectors require custom components
- Requested by @aaronsteers in https://airbytehq-team.slack.com/archives/C08BHPUMEPJ/p1762665961303909
- GitHub Docs: Rate limits for the REST API
Code References
source-github/spec.json:16-65- Current auth options (OAuth and PAT only)source-github/source.py:118-135-get_access_token()implementation- Declarative schema already has
JwtAuthenticatorandSessionTokenAuthenticatorwhich could potentially be composed for this use case
Metadata
Metadata
Assignees
Labels
manifest-only feature gapsMissing capabilities that prevent connectors from being fully declarative (manifest-only)Missing capabilities that prevent connectors from being fully declarative (manifest-only)