You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Nonetheless, the tap sets replication keys accordingly and creates huge state files (esp. for workflow_run_jobs) where every run_id seems to get its own partition.
In my pipelines this results in append only behavior where instead I should probably do full loads instead.
A possible solution here might be to use the use_fake_since_parameter but I haven't checked this yet and would appreciate if one of the experts of this tap could offer an insight
The text was updated successfully, but these errors were encountered:
So most
GitHubRestStream
descendants in the tap support incremental loading using a combination ofupdated_at
replication key and GH APIssince
parameter, e.g. repository issueshttps://docs.github.com/en/rest/issues/issues?apiVersion=2022-11-28#list-repository-issues
None of GitHub's APIs used for
workflow
,workflow_runs
andworkflow_run_jobs
streams however supports those parameters, see e.g. https://docs.github.com/en/rest/actions/workflows?apiVersion=2022-11-28#list-repository-workflowsNonetheless, the tap sets replication keys accordingly and creates huge state files (esp. for
workflow_run_jobs
) where everyrun_id
seems to get its own partition.In my pipelines this results in append only behavior where instead I should probably do full loads instead.
A possible solution here might be to use the
use_fake_since_parameter
but I haven't checked this yet and would appreciate if one of the experts of this tap could offer an insightThe text was updated successfully, but these errors were encountered: