Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UBO-122 add RateLimitedHttpClient #123

Draft
wants to merge 1 commit into
base: develop
Choose a base branch
from

Conversation

yagee-de
Copy link
Member

Copy link
Member

@fluetze fluetze left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How would one use this? Of course, I can write code to use this special kind of http client.
But enrichment resolver is already designed to use complete URIs per data source and identifier type, e.g.

MCR.MODS.EnrichmentResolver.DataSource.Scopus.doi.URI=xslStyle:import/scopus2mods,import/genre2genre:%UBO.Scopus.API.URL%abstract/doi/{0}?apikey=%UBO.Scopus.API.Key%

So, from enrichment resolver view, this is one call which hides multiple steps, xsl transformation and so on, and as first stept a http get call. Might be something else, thought.

From my point of view, it'd be easier if we had an "range limiting resolver", e.g.

MCR.MODS.EnrichmentResolver.DataSource.Scopus.doi.URI=xslStyle:import/scopus2mods,import/genre2genre:rangeLimit:scopus:%UBO.Scopus.API.URL%abstract/doi/{0}?apikey=%UBO.Scopus.API.Key%

"rangeLimit:scopus" would mean there is a RangeLimitingResolver, call and use with configuration "scopus". That configuration might be defined by a property including the range limits for Scopus.
The RangeLimitingResolver would apply that and block/async... the actual call of any URI "below".
This would fit into existing configuration with no need to rewrite Java code. We'd just had to extend the properties for all import URIs.

Copy link
Member

@fluetze fluetze left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... and I guess Bucket4J has much more features we might need, for example there might be a limit of total request per day (or even per year), and another limit per second. Web of Science is very restricting, for example.

Bucket bucket = Bucket4j.builder()
// allows 1000 tokens per 1 minute
.addLimit(Bandwidth.simple(1000, Duration.ofMinutes(1)))
// but not often then 50 tokens per 1 second
.addLimit(Bandwidth.simple(50, Duration.ofSeconds(1)))

@kkrebs kkrebs changed the base branch from main to develop October 10, 2022 12:42
@kkrebs kkrebs marked this pull request as draft October 10, 2022 12:43
@fluetze
Copy link
Member

fluetze commented Jun 21, 2024

Propose to delete this branch and PR, not needed any more

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants