Triggering Github Rate limits

[boydkr]

I'm getting a whole load of these when trying to run the changelog generator now:

#<struct Octokit::RateLimit limit=5000, remaining=4937, resets_at=2018-05-26 10:13:11 -0500, resets_in=2073>
RETRY - Octokit::AbuseDetected: 'GET https://api.github.com/repos/myorg/myrepo/commits/xxxxxxxxxxxxxxxxxxxxx: 403 - You have triggered an abuse detection mechanism. Please wait a few minutes before you try again. // See: https://developer.github.com/v3/#abuse-rate-limits'
8 tries in 10.358754 seconds and 1.0 seconds until the next try
Warning: Can't finish operation: GitHub API rate limit exceeded, change log may be missing some issues. You can limit the number of issues fetched using the `--max-issues NUM` argument.

We have been using this for several months, and it just started happening with our latest release

[antenore]

Same here

$ github_changelog_generator --version
Version: 1.14.3

Repository: FreeRDP/Remmina

It was already fixed in #406 but it looks like it's not a regression. More like @github limiting access

[antenore]

Wait... My version is toooooooo old.

[antenore]

ok, same issue with Version: 1.15.0-rc

[boydkr]

I see. Ideally the number of concurrent requests would be configurable, since you can't assume that Github will never change its rate limits

[olleolleolle]

@boydkr As a diagnostic question: are you using a GitHub personal token to run the program? https://github.com/github-changelog-generator/github-changelog-generator#github-token

[boydkr]

Yes, it blows up way sooner if you don't.

The changelog is being partially/mostly generated despite the rate limiting in our case. I think it varies depending on how quickly the rate limit is hit.

[Skeyelab]

Think this has anything to do with the Microsoft acquisition? Timing seems to correlate.

[Skeyelab]

I've noticed if I close the GitHub Desktop application and VS Code and run this from a basic terminal, it seems to work better. Maybe those clients are sending some calls to the API, causing overages?

[levithomason]

This is unrelated the MS acquisition. I've been using github_changelog_generator for a few years and have had this issue for quite some time. I've been dancing around it with options like:

--no-issues
--no-unreleased
--release-branch master
--since-tag $(git describe --abbrev=0 --tags $(git rev-parse HEAD~250))

Trying to really limit the requests to what I need (PRs merged into master since the last 250 commits). Though, even this has not proven successful.

It would be ideal if we could manually adjust the rate to avoid hitting the limits.

[huan]

I can confirm that #661 can solve this problem with my medium size repository https://github.com/Chatie/wechaty/

[soundstep]

Currently using 1.15.0.pre.rc, I'm also getting a lot of RETRY - Octokit::AbuseDetected.
The --max-issues parameters seems to have no effect at all.

bundle exec github_changelog_generator \
    -u $ORG -p $PROJECT -t $GITHUB_TOKEN \
    --max-issues 1 \
    --exclude-tags-regex '.*\+\d{1,}' \
    --no-unreleased \
    --release-branch master \
    --no-author \
    -o "CHANGELOG.md" \
    --date-format "%d-%m-%Y"
Using these options:
{:date_format=>"%d-%m-%Y",
 :output=>"CHANGELOG.md",
 :base=>"HISTORY.md",
 :issues=>true,
 :add_issues_wo_labels=>true,
 :add_pr_wo_labels=>true,
 :pulls=>true,
 :filter_issues_by_milestone=>true,
 :author=>false,
 :unreleased=>false,
 :unreleased_label=>"Unreleased",
 :compare_link=>true,
 :enhancement_labels=>["enhancement", "Enhancement", "Type: Enhancement"],
 :bug_labels=>["bug", "Bug", "Type: Bug"],
 :exclude_labels=>
  ["duplicate",
   "question",
   "invalid",
   "wontfix",
   "Duplicate",
   "Question",
   "Invalid",
   "Wontfix",
   "Meta: Exclude From Changelog"],
 :breaking_labels=>["backwards-incompatible", "breaking"],
 :issue_line_labels=>[],
 :max_issues=>1,
 :simple_list=>false,
 :ssl_ca_file=>nil,
 :verbose=>true,
 :header=>"# Change Log",
 :merge_prefix=>"**Merged pull requests:**",
 :issue_prefix=>"**Closed issues:**",
 :bug_prefix=>"**Fixed bugs:**",
 :enhancement_prefix=>"**Implemented enhancements:**",
 :breaking_prefix=>"**Breaking changes:**",
 :http_cache=>true,
 :require=>[],
 :user=>"ITV",
 :project=>"project",
 :token=>"hidden value",
 :exclude_tags_regex=>".*\\+\\d{1,}",
 :release_branch=>"master"}

Found 471 tags
Fetching tags dates: 471/471
Sorting tags...
Warning: unable to reject any tag, using regex /.*\+\d{1,}/ in --exclude-tags-regex option.
Received issues: 100
Pull Request count: 703
Filtered pull requests: 64
Filtered issues: 0
Fetching events for issues and PR: 64
RETRY - Octokit::AbuseDetected: 'GET https://api.github.com/repos/org/project/commits/7923ffb48b3caad6129e8c5a4dbe71d69adbe47a: 403 - You have triggered an abuse detection mechanism. Please wait a few minutes before you try again. // See: https://developer.github.com/v3/#abuse-rate-limits'

[iGormilhit]

I'm getting a similar issue: "Octokit::Forbidden: GET https://api.github.com/repos/rero/rero-ils/issues/420/events?per_page=100: 403 - You have exceeded a secondary rate limit. Please wait a few minutes before you try again. // See: https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#secondary-rate-limits"

❯ github_changelog_generator --version
Version: 1.16.4

Using the command:

github_changelog_generator --user XXX --project rero-ils -t ghp_XXX --exclude-label stale --release-branch dev --future-release v1.4.7 --since-tag v1.4.6 -b CHANGES.md -o CHANGES.md

And getting this output:

Using these options:
:date_format=>"%Y-%m-%d"
:output=>"CHANGES.md"
:base=>"CHANGES.md"
:issues=>true
:add_issues_wo_labels=>true
:add_pr_wo_labels=>true
:pulls=>true
:filter_issues_by_milestone=>true
:issues_of_open_milestones=>true
:author=>true
:unreleased=>true
:unreleased_label=>"Unreleased"
:compare_link=>true
:exclude_labels=>["stale"]
:summary_labels=>["Release summary", "release-summary", "Summary", "summary"]
:breaking_labels=>["backwards-incompatible", "Backwards incompatible", "breaking"]
:enhancement_labels=>["enhancement", "Enhancement", "Type: Enhancement"]
:bug_labels=>["bug", "Bug", "Type: Bug"]
:deprecated_labels=>["deprecated", "Deprecated", "Type: Deprecated"]
:removed_labels=>["removed", "Removed", "Type: Removed"]
:security_labels=>["security", "Security", "Type: Security"]
:configure_sections=>{}
:add_sections=>{}
:issue_line_labels=>[]
:max_issues=>nil
:simple_list=>false
:ssl_ca_file=>nil
:verbose=>true
:header=>"# Changelog"
:merge_prefix=>"**Merged pull requests:**"
:issue_prefix=>"**Closed issues:**"
:summary_prefix=>""
:breaking_prefix=>"**Breaking changes:**"
:enhancement_prefix=>"**Implemented enhancements:**"
:bug_prefix=>"**Fixed bugs:**"
:deprecated_prefix=>"**Deprecated:**"
:removed_prefix=>"**Removed:**"
:security_prefix=>"**Security fixes:**"
:http_cache=>true
:require=>[]
:config_file=>".github_changelog_generator"
:user=>"rero"
:project=>"rero-ils"
:token=>"hidden value"
:release_branch=>"dev"
:future_release=>"v1.4.7"
:since_tag=>"v1.4.6"

Found 60 tags                                                          
Fetching tags dates: 60/60
Sorting tags...
Received issues: 2123                                                  
Pull Request count: 1438                                               
Filtered pull requests: 1290
Filtered issues: 668
Fetching events for issues and PR: 1958                                
Warning: Can't fetch commit 59e4e8e9c6ee39d555b49b78ba633c4558d81a9c. It is probably referenced from another repo.
Warning: Can't fetch commit edbd36f91881cc4fc6ffd68f64a24b76f44ac4ef. It is probably referenced from another repo.
Octokit::Forbidden: GET https://api.github.com/repos/rero/rero-ils/commits/f7e6f9e9e2f8c30384b252f85519841fe61e045c: 403 - You have exceeded a secondary rate limit. Please wait a few minutes before you try again. // See: https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#secondary-rate-limits
Exceeded retry limit

[jaygajjar-huma]

It is still happing in 1.16.4