Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

common: improve caching in API/SQL enrichments #747

Closed
pondzix opened this issue Feb 16, 2023 · 0 comments
Closed

common: improve caching in API/SQL enrichments #747

pondzix opened this issue Feb 16, 2023 · 0 comments

Comments

@pondzix
Copy link
Contributor

pondzix commented Feb 16, 2023

If we get an error from external API/database (e.g. because it's down), such error is cached by enrich. It's cached because we don't want to overload API/database experiencing some issues.

TTL for errors stored in the cache is the same as for successful results. If this TTL is some big integer (which is often reasonable in case of successful/valid results) we can get stuck with errors in the cache for while. So even though external API, which originally produced errors, is now working correctly we don't know that cause we use errors stored in the cache.

Potential improvements:

  • Use different, smaller TTL for errors (make it configurable?).
  • For scenario like: if successful result is stored in cache => TTL expires => new value is fetched => we get an error. In such case store error in the cache (with smaller TTL), but return last known 'old' good value to further processing. This fallback would allow enrich to produce fewer badrows in case of 'getting stuck' with errors in the enrichment cache.

Those improvements can be implemented for both API and SQL enrichments.

It all resembles behaviour of circuit-breaker. Adding proper circuit-breaker implementation (most likely coming from some external lib) could be problematic as we still have to support Id monad, hence improvements for existing solution relying on errors caching.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant