Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use redis to cache results #972

Merged
merged 29 commits into from
Jun 15, 2020
Merged

Use redis to cache results #972

merged 29 commits into from
Jun 15, 2020

Conversation

EDsCODE
Copy link
Member

@EDsCODE EDsCODE commented Jun 9, 2020

Changes

implements #765 and #940

  • added a decorator you can use on functions that you want to cache
  • trends/funnel queries will be cached for 30 seconds by default (so that repeat visits don't keep recalculating queries)
  • clicking save funnel will force a refresh so you don't have stale metrics when you're trying to update the funnel
  • dashboard items will be cached for 15 minutes and worker will check for all cached dashboard items and refresh every 10 minutes. this should keep the cache perpetually updated
  • refactor remaining dashboard elements to use hooks and kea (funnel, pie, table)

Todo:

  • keep cache up to date with worker
  • delete cached item when dashboard item is deleted
  • make sure all items are cached properly (pie graph, line graph, table, funnel)
  • make sure last updated prompt is updated on click and looks readable
    - [ ] dashboard items out of order after refresh

Checklist

  • All querysets/queries filter by Team (if applicable)
  • Backend tests (if applicable)
  • Cypress E2E tests (if applicable)

@timgl timgl temporarily deployed to posthog-function-cachin-vec3ad June 9, 2020 18:03 Inactive
@timgl timgl temporarily deployed to posthog-function-cachin-vec3ad June 9, 2020 18:12 Inactive
@timgl timgl temporarily deployed to posthog-function-cachin-vec3ad June 9, 2020 20:01 Inactive
@timgl timgl temporarily deployed to posthog-function-cachin-vec3ad June 9, 2020 21:20 Inactive
@timgl timgl temporarily deployed to posthog-function-cachin-vec3ad June 10, 2020 14:11 Inactive
@timgl timgl temporarily deployed to posthog-function-cachin-vec3ad June 10, 2020 14:16 Inactive
@timgl timgl temporarily deployed to posthog-function-cachin-vec3ad June 10, 2020 14:23 Inactive

people = _calculate_people(events=filtered_events)
return Response([people])

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved everything out of the class because I needed the logic to be easily reusable by the worker

CACHES = {
"default": {
"BACKEND": "django_redis.cache.RedisCache",
"LOCATION": REDIS_URL,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reusing the redis we use for everything else. may consider having a dedicated instance because we are likely going to end up caching 100s -> 1000s of dashboard items indefinitely

@timgl timgl temporarily deployed to posthog-function-cachin-vec3ad June 10, 2020 14:49 Inactive
@EDsCODE EDsCODE marked this pull request as ready for review June 10, 2020 14:54
@timgl timgl temporarily deployed to posthog-function-cachin-vec3ad June 10, 2020 15:04 Inactive
@timgl timgl temporarily deployed to posthog-function-cachin-vec3ad June 10, 2020 17:34 Inactive
@timgl timgl temporarily deployed to posthog-function-cachin-vec3ad June 10, 2020 19:53 Inactive
@timgl timgl temporarily deployed to posthog-function-cachin-vec3ad June 10, 2020 20:23 Inactive
@timgl timgl temporarily deployed to posthog-function-cachin-vec3ad June 10, 2020 20:37 Inactive
TRENDS_ENDPOINT = 'Trends'
FUNNEL_ENDPOINT = 'Funnel'

def cached_function(cache_type: str, expiry=1):
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

somewhat inflexible decorator for the moment. adding this decorator to an endpoint will require an explicit condition to handle

@timgl timgl temporarily deployed to posthog-function-cachin-vec3ad June 10, 2020 22:47 Inactive
@timgl timgl temporarily deployed to posthog-function-cachin-vec3ad June 10, 2020 22:57 Inactive
@EDsCODE EDsCODE requested a review from fuziontech June 10, 2020 23:09
@timgl timgl temporarily deployed to posthog-function-cachin-vec3ad June 11, 2020 14:34 Inactive
TRENDS_ENDPOINT = 'Trends'
FUNNEL_ENDPOINT = 'Funnel'

def cached_function(cache_type: str, expiry=30):
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing to note is that if we have a default expiry of 30 seconds this will be nice for very heavily trafficked instances but for new instances, it might seem like the data is not refreshing/stuck when you're on the trends page. I'm open to limiting the caching for dashboart items only

@timgl timgl temporarily deployed to posthog-function-cachin-vec3ad June 12, 2020 22:07 Inactive
@timgl timgl temporarily deployed to posthog-function-cachin-vec3ad June 12, 2020 22:30 Inactive
Copy link
Collaborator

@timgl timgl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I love it. Some small comments.

posthog/tasks/update_cache.py Outdated Show resolved Hide resolved
frontend/src/scenes/dashboard/DashboardItem.js Outdated Show resolved Hide resolved
@timgl timgl temporarily deployed to posthog-function-cachin-vec3ad June 15, 2020 14:27 Inactive
@timgl timgl temporarily deployed to posthog-function-cachin-vec3ad June 15, 2020 14:39 Inactive
@timgl timgl temporarily deployed to posthog-function-cachin-vec3ad June 15, 2020 17:16 Inactive
@timgl timgl merged commit 063d739 into master Jun 15, 2020
@timgl timgl deleted the function-caching branch June 15, 2020 17:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants