rate limit the API call

[rajnmithun]

It would be nice to have rate limiter before making a call to API if it is in the local cache.

[dsnet]

Hi, can you elaborate? Currently, the logic avoids any API calls over the network if the result is in the local cache.

Are you talking about a general rate-limiter for network traffic?

[rajnmithun]

I am talking about the network traffic in making the API call.
The scenario I am talking about is, if there are more than 10k unique partial hash hit and these are not in the local cache then we would make API call. At this point if the API key has a rate limit of 10K. It would exceed and get blocked.

[dsnet]

I see. In your usage of LookupURLs, are you passing in a batch of URLs every time you call it?

Before implementing any rate limiting logic, I would much rather see the TODO in that function be addressed. As it currently is, we make a network call for each URL, rather than batching them together.

On the other hand, if you have many goroutines calling LookupURLs independently with only a small handful of URLs, then some rate limiting mechanism may be needed.

[rajnmithun]

I am currently not batching the URLs to LookupURLs(which is on my TODO). Also I have a scenario where I have multiple goroutines/instance of the client running which would still need a mechanism for rate limiting across them.

[rajnmithun]

Batching of the URLs is going to be done any time soon or should I submit a PR?

[dsnet]

You're welcome to submit a PR (assuming you sign the CLA) and I can review it.

[rajnmithun]

I have a PR ready for the batching of the partial hashes. Can you please take a look at it @dsnet .

[rajnmithun]

@dsnet can you please let me know how to get this reviewed and merged in to master branch.

[alexwoz]

Hi @rajnmithun, I apologize for the delay. I will try to take a look at this as soon as possible!

[rajnmithun]

@alexwoz thank you.