feat: Add Redis caching for navigation pagination #488
Conversation
YuriZmytrakov
force-pushed
the
CAT-1382-2
branch
9 times, most recently
from
5c8d201 to
8991cfc
Compare
YuriZmytrakov
changed the title
YuriZmytrakov
force-pushed
the
CAT-1382-2
branch
2 times, most recently
from
b16cc88 to
8991cfc
Compare
|
|
||
| if redis_enable: | ||
| try: | ||
| redis = await connect_redis() |
There was a problem hiding this comment.
how does this work with connect_redis_sentinel?
There was a problem hiding this comment.
there is connect_redis and connect_redis_sentinel. I think maybe there is logic missing which should determine which one to use? Maybe you can explain how it works?
There was a problem hiding this comment.
we should also log the redis connection whether its successful or not?
There was a problem hiding this comment.
-
If a user wants to use standalone Redis (the most common case), they should use
connect_redis(). If they want to use Redis Sentinel (used by Cloudferro),connect_redis_sentinel()should be used. The connection is configured differently in each case, which is why we have two separate functions. The user needs to configure either theREDISorREDIS_SENTINELenvironment variables. -
The selection of which function to use should be done in redis_utils.py:
- For standalone Redis: redis_settings: BaseSettings = RedisSettings()
- For Redis Sentinel: redis_settings: BaseSettings = RedisSentinelSettings()
-
Logging has been added for users.
There was a problem hiding this comment.
Ok thanks for explaining - here you only use connect_redis but should you use connect_redis_sentinel as well?
There was a problem hiding this comment.
If the user configures Redis Sentinel, the connect_redis_sentinel function should be used instead of connect_redis. I could try to combine these two functions into one, which would choose the appropriate approach depending on whether the user wants standalone or Sentinel depending on config selected. This way, no one would need to manually replace function, only connect_redis would be called. Both approaches are fine with me. Just to note, Cloudferro will probably be the only users of Sentinel, which is why I added this function.
There was a problem hiding this comment.
Good idea, maybe even a helper function that calls the right function. If there are env options for sentinel in the codebase then we have to make it useable. Never say never.
| "href": prev_link, | ||
| }, | ||
| ) | ||
|
|
There was a problem hiding this comment.
can this code - the redis_enabled block - be put into a function? It is used with all_collections too
| "href": prev_link, | ||
| }, | ||
| ) | ||
|
|
There was a problem hiding this comment.
this code into a function
YuriZmytrakov
force-pushed
the
CAT-1382-2
branch
from
8991cfc to
e3e84cd
Compare
- Configure Redis image for tests of caching navigation - Update Make file Redis with test targets for ES and OS - Integrate Redis/Redis Sentinel client to cache navigation - Add Redis funcs Sentinel for navigation caching
Add tests for Redis pagination caching in search and collections endpoints, plus utility function tests.
YuriZmytrakov
force-pushed
the
CAT-1382-2
branch
3 times, most recently
from
c88193b to
ab22909
Compare
YuriZmytrakov
force-pushed
the
CAT-1382-2
branch
from
ab22909 to
c658023
Compare
2 participants
Related Issue(s):
Description:
Add Redis caching support for navigation pagination to enable proper
prev/nextlinks in STAC API responses.PR Checklist:
pre-commit run --all-files)make test)