How should Admin Console log ansible runs?

[tim-moody]

This will cause ansible to log to the standard iiab log even when run from Admin Console.

[jvonau]

What is the standard iiab log? Note export ANSIBLE_LOG_PATH= is used within iiab-install, iiab-network, and runrole to direct the logging to different log files.

[tim-moody]

I didn't realize runrole had a different log than iiab-install. In any event I am asking for iiab-install's log. The export variable takes precedence over the config file, so there will be no conflict with current scripts. I am unable to set the env variable.

[holta]

I am unable to set the env variable.

Why is that out of curiosity?

Note export ANSIBLE_LOG_PATH= is used within iiab-install, iiab-network, and runrole to direct the logging to different log files.

Good point.

Some might argue that ~4 log files is too many.

On the other hand, mashing everything together into 1 single log file (e.g. /opt/iiab/iiab/iiab-install.log) is a bit much.

Certainly this would makes iiab-diagnostics output less meaningful, as iiab-diagnostics takes the last 100 lines of each of these log files.

Ref:

• PR Install Services in Admin Console iiab-admin-console#475

[jvonau]

log_path=/var/log/iiab-ansible.log maybe as the fallback? Just one more log file to check for.

[jvonau]

To test this I broke the link of /opt/admin/cmdsrv/ansible.cfg and provided a new file with the above included, works nicely. The path and log file name is up for debate as this would be shared with runroles if the change is made in iiab without breaking the link.

[holta]

Followup:

• PR make ansible runs add to iiab install log iiab-admin-console#476
• Admin Console will not install: "Could not find or access 'ansible.cfg.j2'" #3181
• PR forgot to add file iiab-admin-console#477

[holta]

We should think about Admin Console recording to a distinct log file separate from the initial install's /opt/iiab/iiab/iiab-install.log (regardless what it is named!) — so as to illuminate more insights via iiab-diagnostics, which only records the last 100 lines of each such file:

• PR forgot to add file iiab-admin-console#477

[holta]

We should think about Admin Console recording to a distinct log file separate from the initial install's /opt/iiab/iiab/iiab-install.log (regardless what it is named!) — so as to illuminate more insights via iiab-diagnostics, which only records the last 100 lines of each such file:

• PR forgot to add file iiab-admin-console#477

A related issue was just solved thanks to @shanti-bhardwa and @tim-moody:

• PR avoid writing 'garbage' to iiab log iiab-admin-console#486

The above question of keeping diagnostics clear, by using distinct log files (especially for iiab-diagnostics which communicates the final 100 lines of each log file) remains important.

@jvonau also suggested using /var/log and logrotate, if it gets to the point where logs truly balloon in size.

[tim-moody]

no longer needed

[holta]

Quickly discussed on this week's http://minutes.iiab.io call:

iiab-diagnostics should offer context around what fails and why — in any file separate from iiab-install.log e.g. with this new functionality and similar:

• PR Install Services in Admin Console iiab-admin-console#475

Background: log file iiab-install.log for iiab-install has always been kept intentionally distinct from other Ansible log files in /opt/iiab/iiab, e.g.:

• iiab-debug.log for runrole
• iiab-network.log for iiab-network
• iiab-configure.log for iiab-configure

We discussed trying to find a way to include /etc/tmp-<JOB-NUMBER> in iiab-diagnostics — but very unfortunately that won't work — as these files in /tmp are deleted immediately upon completion of each job (and even if they weren't...all files in /tmp are deleted during reboot...so that would be a very fragile approach).

I strongly recommend this info be logged to any other filename (e.g. something like /opt/iiab/iiab/iiab-admin.log or any distinct filename).

@jvonau recommends this be saved to /var/log/* which is also a fine option if others prefer that.

[tim-moody]

These logs are not for the benefit of Admin Console, so I guess I don't really care where they are. In terms of /var/log I would keep them all together. Ansible writes to the log, so any rotation would have to be configured there and I don't know how.

I can offer the following:

journalctl -t IIAB-CMDSRV will tell you what cmdsrv is doing at a high level

we can turn on logging to the file declared in /opt/admin/cmdsrv/ansible.cfg

I think I can suppress logging of the setup -m to about 200 chars. This will still happen every time cmdsrv starts.

[holta]

journalctl -t IIAB-CMDSRV will tell you what cmdsrv is doing at a high level

Should iiab-diagnostics include that output?
Or do you/all think that's too coarse (insufficiently granular) to be of much use?

we can turn on logging to the file declared in /opt/admin/cmdsrv/ansible.cfg

Yes. Quick solution would be to save to /opt/iiab/iiab/iiab-admin.log alongside the current log files there. And then perhaps move all of these to /var/log if ongoing disk bloat is a real issue, using logrotate or similar? So far disk bloat from a few MB of IIAB log files is not an issue on today's massive disks & SD cards. But that could hypothetically change if we throw a lot more into log files on an ongoing basis?

I think I can suppress logging of the setup -m to about 200 chars. This will still happen every time cmdsrv starts.

@jvonau does that make sense? Or do you have another recommendation?

[tim-moody]

I think I can suppress logging of the setup -m to about 200 chars. This will still happen every time cmdsrv starts.

it occurred to me that the 200 chars can go to a separate log file in /tmp, since no one will be interested in it.

[jvonau]

These logs are not for the benefit of Admin Console, so I guess I don't really care where they are. In terms of /var/log I would keep them all together.

Why? The existing log files are generated with a process started by human intervention. During the initial install iiab-install.log is usually only written to one time. The proposed admin-console/cmdserv logging is tied to a daemon that will be running and modifying the install an unknown number of times and for a unknown length of service.

Ansible writes to the log, so any rotation would have to be configured there and I don't know how.

Drop a file into /etc/logrotate.d/ describing the actions needed.

I can offer the following:

journalctl -t IIAB-CMDSRV will tell you what cmdsrv is doing at a high level

we can turn on logging to the file declared in /opt/admin/cmdsrv/ansible.cfg

I think I can suppress logging of the setup -m to about 200 chars. This will still happen every time cmdsrv starts.

That would be a bonus, the main issue was the logging would go to the same log file for all ansible calls from cmdsrv. All the logging in iiab uses 'ANSIBLE_LOG_PATH=' perhaps iiab-cmdsrv.service might be able to add Environment="ANSIBLE_LOG_PATH=/some/throwaway/path/file.log" to keep the logging cleaner? Pressing 'ICO' would be a human user intervention, the logging defined in /opt/admin/cmdsrv/ansible.cfg could then point to /opt/iiab/iiab/iiab-debug.log, without the noted '-m setup info' being stored in the log file, as iiab-debug.log is already in use for runrole. Might want to add ANSIBLE_LOG_PATH= to 'runroles' also, I suggest iiab-debug.log.

[tim-moody]

yes. I discovered that if I suppress the logging the output has that suppression message, so the only way to suppress and get the ansible variables is to write to a throw away file in /tmp.

Drop a file into /etc/logrotate.d/ describing the actions needed

I don't notice any for the logs in /opt/iiab/iiab

the main issue was the logging would go to the same log file for all ansible calls from cmdsrv

apart from the cmdsrv init, I'm not clear why that would be more than the other iiab logs or why install and reinstall are somehow two different things. iiab-diags get clogged with stale data. Installing services with cmdsrv is a different flow than with a script, but I wouldn't expect it to be more frequent.

To be clear, cmdsrv already logs. The only change requested is to satisfy the needs of iiab-diags, in which btw, 10K chars is a trifle.

In any event, if the noted changes are desired I will make them.

[holta]

10K chars is a trifle

Each line may be only 10 KB, but these add up fast if an Admin Console implementer/operator initiates a lot of Ansible actions presumably?

The human intervention point is a good one, as these moments will (generally) correspond with the most useful diagnostics.

Question for Everyone: do any of those 10KB lines contain info of real diagnostic value? (I'm assuming not much of value there, but perhaps that's not the full truth??)

/opt/iiab/iiab/iiab-debug.log

Sounds very reasonable to me. Let's go with this now, if possible.

If /etc/logrotate.d etc are later needed to face overwhelming bloat, that's fine too.

[holta]

I haven't tried it, but fyi @tim-moody's PR looks great here:

• feed iiab ansible log sometimes iiab-admin-console#487

[holta]

I can offer the following:

journalctl -t IIAB-CMDSRV will tell you what cmdsrv is doing at a high level

Looks like an excellent idea. This needs to be added to iiab-diagnostics, as we just discussed on the weekly community call.

[holta]

Even better idea now that @tim-moody has redacted logging of passwords etc — great catch:

• PR don't log arguments for sensitive commands iiab-admin-console#488