[Performance] Investigate performance characteristics of settings.__getattr__ memcached access

[ryanpetrello]

Every conf.settings.__getattr__() everywhere in every AWX process incurs a memcached round trip lookup. This is fine for some use cases (within most HTTP requests), but for things like callback event processing where throughput is very high, these domain socket reads really add up (see: #5762)

As an extension to this, every logger.info() (or debug(), or error() or etc...) in AWX's logging implementation does per-log-line settings.__getattr__() lookups to determine external logging settings every time we log a message (so, it's not usual to do 10+ memcached round trips per log line).

I'd like to investigate some new ideas for managing in-process setting caching that involve less per-process overhead, especially for the callback receiver, and I'd like to rethink our current approach of cache-based "pull on demand".

[ryanpetrello]

cc @matburt @AlanCoding @chrismeyersfsu since we talked about this briefly

[matburt]

For things that have a limited level of context like processing events we could use local in-memory memoization as a shortcut, without having to make major caching changes.

[AlanCoding]

Is this still valid with Redis now?

[matburt]

Yep I think this is probably still a valid concern. I still maintain we could do some basic memoization and probably mitigate it.

[AlanCoding]

This still isn't clicking for me. We already have local caching on top of redis.

awx/awx/conf/settings.py

Line 400 in 4e97492

@cachetools.cached(cache=cachetools.TTLCache(maxsize=2048, ttl=SETTING_MEMORY_TTL))

Granted, we don't use this local caching for most things.

awx/awx/conf/settings.py

Line 34 in 4e97492

SETTING_MEMORY_TTL = 5 if 'callback_receiver' in ' '.join(sys.argv) else 0

So is this issue just saying "use a value other than 0 here"?

Seems important to note that the TTLCache there was added September 2020 by Ryan, whereas this issue was created in January 2020.

[AlanCoding]

I'm putting this into the Needs Refinement bucket, because we want to collect more actionable performance-enhancing items.

If we changed the value from 0 to 5, that would be likely to fail a lot of integration tests... which is fine, we would just need to add some logic to be more patient in those tests.

We can also get a lot of use from Django Debug Toolbar, which will show the cache hits, we want to have some specific numbers of how much this will improve things by.

[AlanCoding]

This is not something that goes through normal QE, so I'll close it directly here. I notified the perfscale team, but don't expect any particular response from them.

Capturing from the PR, some writeup from this:

https://gist.github.com/AlanCoding/3bff66c63db36ae7cb8c326df2c38e29