-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement caching to avoid a ton of requests to crowdsec api #21
Comments
Hi, thanks for the suggestion. Yes, caching would definitely be a nice feature, plus the data are quite suitable to be cached. I'm more concerned about the cache duration, since the bouncer does not own the data, nor can be notified on change. About the cache location, I'm thinking of memory first: can't be faster and easier to garbage collect in case of bug. Lastly, disk IO are a pain that I don't want to launch myself into... What do you think? |
Hello! I leave what I think:
|
@fbonalair |
That is what I was thinking of using for cache lifetime. Though, I'm still worried about first offenders getting unlimited access for the cache duration.
Depending on the caching solution parameters, some will be available through environment variables. Thanks for the suggestion. |
Any updates to this issue? I had to shut down my traefik-crowdsec-bouncer. My server would randomly went unresponsive even over ssh because the bouncer-container got unstable and jamed up the cpu. My initial guess was also that this is caused by some sort of overload issue with to many requests and therefore to many calls to the crowdsec-LAPI via the bouncer-middleware. I read some of the decumentation over at crowdsec an found, that the official nginx-bouncer has two operation modes:
That sounds like a solid solution in my estimation. Wouldn't that be also beneficial for the traefik-bouncer, especially in a more demanding environment or with limited ressources. |
Hello, I've been following this project for a while and wanted to contribute somehow. I've implemented a local cache using the library go-cache It is configurable using 2 environnement variables:
When the cache is enabled, the first time an IP has to be checked, it is first looked up in the local cache.
Cache invalidation is provided by the library, a background job will remove from the cache every entry which are not valid anymore. I've got some idea on how to implement a redis configurable version as well to mix cache with the streaming mode which could greatly improve performance. What do you think about this ? |
I've just finished working on the streaming mode, it works pretty well. At the start it takes all known banned IP and cache it in local-cache, and then every minute local cache is updated with the new information only. It can also be configured with env variables:
I've took the liberty to enable it by default. |
I took the liberty only to review the #33 PR since it's written to be based on the #32. About the default mode, couple of thoughts:
To prepare for a redis cache or other caches, I would be nice to externalize the cache logic into it's own file / service / folder. And depending on user configuration, the right one would initialized in bouncer.go . It was my rough start in branch feat/cache. |
Currently every request gets forwarded to crowdsec one by one and it is slow and resource intensive. In my setup I have setup mariadb additionally and calling crowdsec on every request results in a call to the db. All this can be avoided with a single json file with cached ip addresses on the bouncers side. Similar to how cloudflare bouncer is caching them.
This also results in pretty big mariadb binary logs.
Simple cache mechanism would save space and increase performance by having less impact on the system. File based caching (like json) would be enough, but redis would be awesome.
The text was updated successfully, but these errors were encountered: