Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Throttle (concurrent) requests to Maven Central #1218

Merged
merged 1 commit into from Jan 5, 2020
Merged

Conversation

@fthomas
Copy link
Owner

fthomas commented Jan 5, 2020

Scala Steward has become so fast that it is now frequently being
blacklisted by Maven Central despite using Coursier's caching layer.
Previous attempts to prevent that (#1215 and #1216) were't successful
so I'm now trying to limit the rate of calls to Coursier that
potentially result in a call to Maven Central.

Scala Steward has become so fast that it is now frequently being
blacklisted by Maven Central despite using Coursier's caching layer.
Previous attempts to prevent that (#1215 and #1216) were't successful
so I'm now trying to limit the rate of calls to Coursier that
potentially result in a call to Maven Central.
@fthomas fthomas added the enhancement label Jan 5, 2020
@fthomas fthomas added this to the 0.6.0 milestone Jan 5, 2020
@codecov

This comment has been minimized.

Copy link

codecov bot commented Jan 5, 2020

Codecov Report

Merging #1218 into master will decrease coverage by 0.21%.
The diff coverage is 28.57%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1218      +/-   ##
==========================================
- Coverage   69.06%   68.84%   -0.22%     
==========================================
  Files          95       96       +1     
  Lines        1432     1438       +6     
  Branches       44       35       -9     
==========================================
+ Hits          989      990       +1     
- Misses        443      448       +5
Impacted Files Coverage Δ
...la/org/scalasteward/core/application/Context.scala 0% <0%> (ø) ⬆️
...scala/org/scalasteward/core/util/RateLimiter.scala 0% <0%> (ø)
...scala/org/scalasteward/core/update/UpdateAlg.scala 61.9% <100%> (+1.9%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4a6940a...b1a4387. Read the comment docs.

@fthomas fthomas merged commit 04cedd3 into master Jan 5, 2020
4 of 6 checks passed
4 of 6 checks passed
codecov/patch 28.57% of diff hit (target 69.06%)
Details
codecov/project 68.84% (-0.22%) compared to 4a6940a
Details
Codacy/PR Quality Review Up to standards. A positive pull request.
Details
Summary no rules match, no planned actions
Details
Travis CI - Branch Build Passed
Details
Travis CI - Pull Request Build Passed
Details
fthomas added a commit that referenced this pull request Jan 6, 2020
The same rationale as for #1218 applies here too. Our own versions cache
allows for a more fine-grained control of which calls to Coursier (and
therefore potentially to Maven Central) are rate limited. Since we are
using the same TTL for our cache and for Coursier's cache, our cache
should be as fresh as Coursier's.

The previous rate limiting could result in limited calls that hit
Coursier's cache and unlimited calls that missed Coursier's cache.
This won't happen with this implementation. So we're faster now and
reduce the chance of being blacklisted by Maven Central again.
fthomas added a commit that referenced this pull request Jan 6, 2020
* Add our own versions cache

The same rationale as for #1218 applies here too. Our own versions cache
allows for a more fine-grained control of which calls to Coursier (and
therefore potentially to Maven Central) are rate limited. Since we are
using the same TTL for our cache and for Coursier's cache, our cache
should be as fresh as Coursier's.

The previous rate limiting could result in limited calls that hit
Coursier's cache and unlimited calls that missed Coursier's cache.
This won't happen with this implementation. So we're faster now and
reduce the chance of being blacklisted by Maven Central again.

* Do not rely on the cached sort order
@fthomas fthomas deleted the topic/rate-limiter branch Jan 6, 2020
fthomas added a commit that referenced this pull request Jan 14, 2020
This renames VersionsCacheAlg to VersionsCacheFacade and changes it to
not store versions anymore. Versions are now always obtained from
Coursier. VersionsCacheFacade's job now is to keep track of when
versions where updated by Coursier and to rate limit calls to Coursier
that hit the network (because of #1218).

The big advantage of this change is that we don't need to build and
maintain our own versions cache (which would always be inferior to
Coursier's cache). For example, until now we've neglected the
repositories where dependencies are available in our cache which would
have lead to incorrect PRs with #1209.

VersionsCacheFacade still ignores resolvers when storing the timestamp
of the last update, but this will only lead to unlimited calls to
Coursier that might hit the network but not to incorrect data and
behaviour.
fthomas added a commit that referenced this pull request Jan 14, 2020
This renames VersionsCacheAlg to VersionsCacheFacade and changes it to
not store versions anymore. Versions are now always obtained from
Coursier. VersionsCacheFacade's job now is to keep track of when
versions where updated by Coursier and to rate limit calls to Coursier
that hit the network (because of #1218).

The big advantage of this change is that we don't need to build and
maintain our own versions cache (which would always be inferior to
Coursier's cache). For example, until now we've neglected the
repositories where dependencies are available in our cache which would
have lead to incorrect PRs with #1209.

VersionsCacheFacade still ignores resolvers when storing the timestamp
of the last update, but this will only lead to unlimited calls to
Coursier that might hit the network but not to incorrect data and
behaviour.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.