Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace local memcached usage with redis #6932

Closed
ryanpetrello opened this issue May 5, 2020 · 2 comments
Closed

Replace local memcached usage with redis #6932

ryanpetrello opened this issue May 5, 2020 · 2 comments

Comments

@ryanpetrello
Copy link
Contributor

ryanpetrello commented May 5, 2020

Now that every AWX node has a local independent redis server, we can probably just get rid of our memcached usage (and just use redis instead).

In modern versions of AWX, memcached is used specifically for:

  • a local cache for AWX settings access (see: awx.conf.settings); any time django.conf.settings is accessed (e.g., settings.TOWER_URL_BASE), we first query memcached via Django's cache framework (https://docs.djangoproject.com/en/3.0/topics/cache/) to see if the value exists in memcached (if not, we pull the value from the database and then place it in the cache)
  • any time a setting value is changed via the API, we run a task on every AWX node, which removes the old value from that nodes' local memcached
  • there are various other places where AWX code interacts with memcached, but it generally does so using Django's cache interface i.e., cache.get(), cache.set()
  • AWX used to use memcached as a holding space for in-flight playbook event data, and for fact gathering, but we've removed this type of usage
PLAN OF ACTION

I highly suspect that all of the above interactions are implemented only through Django's cache interface (i.e., I don't think we've got code that directly accesses memcached using the python-memcached library), so replacing memcached with redis should be as simple as just swapping out the dependency and the Django caching configuration.

  • investigate https://github.com/jazzband/django-redis as an alternative cache plugin, and add it to the list of dependencies in ./requirements
  • remove the python-memcached library from our list of dependencies in ./requirements
  • update our definition of CACHES under awx.settings to point at the redis Unix domain socket instead of memcached
  • remove any and all installer code and configuration that provisions memcached containers
    • in the local Docker installer
    • in the k8s installer
  • remove any integration or functional tests that test/verify memcached connectivity at startup/install time
@kdelee
Copy link
Member

kdelee commented Jul 6, 2020

main thing I see as missing from our testing is this case:

"I update settings on Tower A and look to see if updated on Tower B"

going to check that on a cluster

@kdelee
Copy link
Member

kdelee commented Jul 6, 2020

Allright, tried that out and adding a test to catch in future. Going to close as verified

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants