/root/.auth_ref.json caching can cause issue

[BjoernT]

We had disks filling up on compute nodes and that caused maas to store corrupt service catalog cache files (took some time to find that). In such cases, I would like to add a parsing of file this file prior to actually using it and if the parser fails, rebuilt it or better storing it under /run which is mounted as tmpfs. So it would not be affected by disks filling up.

[aaronsegura]

If the file is empty nova_service and neutron_linuxbridge maas plugins fail with "status error No JSON object could be decoded". Please remove the file if it cannot be written to.

[mancdaz]

Since it seems that this issue would manifest in failing plugins (and hence getting alarms), I'm not sure there is anything that needs to be done here.

[aaronsegura]

That's not an answer. This is something that can (and should) be handled automatically without human intervention.

[stevelle]

maas checks should recover when any legitimate underlying condition (such as a full disk) is cleared.
This seems like a legitimate issue. I'd like to reopen it (even knowing that means I will end up being responsible for a fix, per @sigmavirus24).

[BjoernT]

Yes agreed, this seems like a case of catching the authentication exception and removing the file. Should be a fairly simple hack inside the maas_common module

[git-harry]

This repo is no longer actively maintained. If this issue is still valid please open a new issue at https://github.com/rcbops/rpc-openstack/issues