Proposal: Decouple Rsyslog from web and task subsystem

[TheRealHaoLiu]

Proposal

What

This is a proposal for the change in the rsyslog subsystem and its integration with AWX to allow decoupling of rsyslog configuration management from the web and task subsystems.

Why

We are attempting to decouple the link between web container and task container within AWX. Decoupling the link between web and task containers will enable us to deploy and scale components of AWX in more flexible and resource-efficient ways.

In examining the interaction and connection between the web and task container we notice that the management of the rsyslog subsystem and its configuration is done in a way that requires a lot of shared files between the two containers.

This proposal will focus on discussing the linkage between web and task containers caused by the rsyslog subsystem and how we plan to sever it.

Context

In the current deployment of AWX, rsyslog is a process that runs in the web container via supervisord.

When a user changes the settings that are related to rsyslog, the web container commits the settings into the database and uses pg_notify to trigger the task subsystem to execute a task to generate new rsyslog configuration and restart rsyslog in the web container.

This design results in the following coupling:

• supervisord socket of the web container needs to be mounted into the task container.
• rsyslog config directory must be mounted by both the task and web container so that the task container can write to it and rsyslog that’s running in the web container can read from it.
• The rsyslog socket needs to be mounted into both web and task containers so logs can be written to rsyslog.

When deployed on Kubernetes, this coupling and sharing of files is done via emptyDir volumes. emptyDirs cannot be shared between containers in different deployments (pods) and therefore prevent task and web container to be deployed as separate deployments.

The following diagram shows the containers and emptyDir volumes within the AWX deployment with the rsyslog related emptyDir volumes highlighted in red.

Goals

• Eliminate the filesystem base coupling between web and task container caused by rsyslog
• Does not dramatically complicate VM/container base deployment

Non-Goals

• Couplings between web and task containers caused by other shared systems, such as redis. (This will be discussed in another follow up proposal)

How

Phase 1: rsyslog sidecar before splitting web and task

In order to eliminate the complicated cross interaction described in the context section we are proposing to create a new configwatcher process

Responsibility of the rsyslog configwatcher process:

• consumes pg_notify on configuration change
• handles the generation of rsyslog config
• reloads rsyslog when needed

With this new configwatcher process we can move rsyslog and the configwatcher process into a separate container, web and task process would no longer need to manage rsyslog’s configuration and lifecycle.

Only interaction between web/task container and rsyslog will be through the socket connection to write log to rsyslog.

Phase 2: rsyslog sidecar after splitting web and task

Once all emptyDir and filesystem coupling (including redis) between web and task container has been removed, the task and web container is able to be deployed in separate pods. We will be able to put a rsyslog sidecar in the independent web and task pod to retain the external logging capability of AWX

Select the relevant components

• UI
• API
• Docs
• Collection
• CLI
• Other

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.