[TECH DEBT] explore issue with pillar_cache

[whytewolf]

Description of the tech debt to be addressed, include links and screenshots

lately we have had a few issues in the discord and slack about pillar_cache issues.

I'm going to start a deep dive into if this is actually a problem with pillar_cache or with documentation. and attempt to fix the issue.

[carsten-AEI]

As I was one of the Discord participants, I'm adding my current findings here:

I think I found the problem locally. In the git branch I was developing in, this cache setting was still active

pillar_cache: True
pillar_cache_ttl: 3600
pillar_cache_backend: memory

while in prod, we have recently switched over to

pillar_cache: True
pillar_cache_ttl: 86400
pillar_cache_backend: disk

after reading that caches are local per worker process. Given we run 64 in parallel, it seems we could not reliably invalidate all caches via saltutil.refresh_pillar nor salt-run cache.clear_pillar '*' - possibly because a salt minion saltutil.refresh_pillar and a subsequent salt minion pillar.get were served by different workers.

While writing up a larger reply here, I was not able to reproduce my problem with disk based caching active, i.e. every saltutil.refresh_pillar updated the disk cache accordingly and every worker seem to use that to render the minion's pillar correctly.

Therefore, I think we should be good and the only possible enhancement I would suggest adding to the docs that using memory as a pillar cache backend may be a bad idea.

I've saved my unsuccessful debugging steps trying to replicate this issue locally and I can add it here if you want (but I doubt it will help much).

[whytewolf]

@carsten-AEI thank you for the update. I do think memory cache is listed as experimental. but your experience does help as to locate a potential pain point with it.

[network-shark]

maybe the same issue ? #66292

[jtraub91]

@network-shark @whytewolf

maybe the same issue ? #66292

Not sure what was being discussed in the discord/slack but the issue referenced here is that inline pillar was being cached in certain scenarios, which I would think ought to never happen...

[whytewolf]

@network-shark @whytewolf

maybe the same issue ? #66292

Not sure what was being discussed in the discord/slack but the issue referenced here is that inline pillar was being cached in certain scenarios, which I would think ought to never happen...

you're right this wasn't the issue that was being discussed. the one that was he issue that was being discussed was the one that is memory based chacheing is dependent on each worker.

however i will take a look at this as you're right. inline pillar should not be cached. even in orchestration.