Logging needs improvement #3373

Closed
madduck opened this Issue Jan 22, 2013 · 8 comments

Comments

Projects
None yet
4 participants
@madduck
Contributor

madduck commented Jan 22, 2013

The log output of salt-{master,minion,call} needs work. At the moment, there is a lot of noise and very little signal.

At the INFO level, the log output should really tell me what the script/program is doing at a high level. I don't care that it's running the ext_nodes script or that it received AES payload — those are interesting for debugging — but what I want to know is what states are being applied, what resources accessed, what actions taken, all succinct but with plenty of detail.

It seems that the master currently only really logs crypto-related stuff. Surely, there's much more interesting stuff happening.

salt-{minion,call}'s INFO logging is much better, albeit too verbose. I don't think I need a line telling me that a state is being executed, followed by a line about the result — just the result will do, with enough context. I don't need to know what commands are being executed or what's being loaded and sync'd (that is DEBUG information). I want to know what it's doing with all the details, one line per step (state).

Altogether, I think there ought to be more log calls at the DEBUG level and less at the INFO level. It would probably pay off for someone interested in learning the code base to go through the code and add consistent logging in one go. This might require a fair bit of coordination due to the amount of code that will be touched.

@thatch45

This comment has been minimized.

Show comment
Hide comment
@thatch45

thatch45 Jan 22, 2013

Member

I completely agree, I have been meaning to audit this for a while since I have been very lax on the logging messages.

Member

thatch45 commented Jan 22, 2013

I completely agree, I have been meaning to audit this for a while since I have been very lax on the logging messages.

@kfdm

This comment has been minimized.

Show comment
Hide comment
@kfdm

kfdm Jan 22, 2013

Contributor

Perhaps it's too simplified, but for the sake of discussion, perhaps

  • INFO would be for the state actions (executing state pkg.installed)
  • DEBUG would be the module actions (executing command rpm)
  • TRACE for things lower than that ?

At least from the perspective of salt-minion/salt-call

Some of the other highstate logging like "Fetching file 'salt://top.sls'" seem like DEBUG level stuff

Contributor

kfdm commented Jan 22, 2013

Perhaps it's too simplified, but for the sake of discussion, perhaps

  • INFO would be for the state actions (executing state pkg.installed)
  • DEBUG would be the module actions (executing command rpm)
  • TRACE for things lower than that ?

At least from the perspective of salt-minion/salt-call

Some of the other highstate logging like "Fetching file 'salt://top.sls'" seem like DEBUG level stuff

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Jan 22, 2013

I'm sure an audit would help. Right now you can actually set the logging level for modules, states, etc. all separately using log_granular_levels, and also the console & log file --all separately:
http://docs.saltstack.org/en/latest/ref/configuration/master.html#log-granular-levels

Even better: if you set your log file to file:///dev/log it lets rsyslog sort things out into /var/log/debug (for DEBUG), /var/log/messages and /var/log/syslog all by default (some adjustment may be required. In addition to that you can have rsyslog use a remote log host, of course, which gets the same great results.

ghost commented Jan 22, 2013

I'm sure an audit would help. Right now you can actually set the logging level for modules, states, etc. all separately using log_granular_levels, and also the console & log file --all separately:
http://docs.saltstack.org/en/latest/ref/configuration/master.html#log-granular-levels

Even better: if you set your log file to file:///dev/log it lets rsyslog sort things out into /var/log/debug (for DEBUG), /var/log/messages and /var/log/syslog all by default (some adjustment may be required. In addition to that you can have rsyslog use a remote log host, of course, which gets the same great results.

@jesusaurus

This comment has been minimized.

Show comment
Hide comment
@jesusaurus

jesusaurus Sep 20, 2013

Contributor

I agree that salt's logging needs to be audited in order to improve the signal-to-noise, but I don't agree with @kfdm about what the levels should reperesent. salt.state and salt.pillar should both be able log at INFO and DEBUG levels, but we do need a clear definition of what belongs at which level. I wish python had a PEP for logging levels, but the best we have is the second table on the logging HOWTO page. Perhaps we could draw a line based on what a salt module is doing versus how that salt module is doing it? @thatch45 do you still have plans to audit the logging levels?

Contributor

jesusaurus commented Sep 20, 2013

I agree that salt's logging needs to be audited in order to improve the signal-to-noise, but I don't agree with @kfdm about what the levels should reperesent. salt.state and salt.pillar should both be able log at INFO and DEBUG levels, but we do need a clear definition of what belongs at which level. I wish python had a PEP for logging levels, but the best we have is the second table on the logging HOWTO page. Perhaps we could draw a line based on what a salt module is doing versus how that salt module is doing it? @thatch45 do you still have plans to audit the logging levels?

@thatch45

This comment has been minimized.

Show comment
Hide comment
@thatch45

thatch45 Sep 21, 2013

Member

Yes, we need to do this, I think that the first step is to define what messages should be being sent were and write a spec on it.
I think that a generic document is the first step, one that could be like a pep8 for logging, then we can layer salt specifics on top of it

Member

thatch45 commented Sep 21, 2013

Yes, we need to do this, I think that the first step is to define what messages should be being sent were and write a spec on it.
I think that a generic document is the first step, one that could be like a pep8 for logging, then we can layer salt specifics on top of it

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Sep 21, 2013

I really like this thinking; I think there should be RFC's written to define the abstracted interfaces of Salt as data center standards. 'Requesting comments' in this way can only be good for Salt if only for circulating material that happens to double as excellent marketing collateral among the Internet engineering experts of the world.

On Sep 20, 2013, at 7:56 PM, Thomas S Hatch notifications@github.com wrote:

Yes, we need to do this, I think that the first step is to define what messages should be being sent were and write a spec on it.
I think that a generic document is the first step, one that could be like a pep8 for logging, then we can layer salt specifics on top of it


Reply to this email directly or view it on GitHub.

ghost commented Sep 21, 2013

I really like this thinking; I think there should be RFC's written to define the abstracted interfaces of Salt as data center standards. 'Requesting comments' in this way can only be good for Salt if only for circulating material that happens to double as excellent marketing collateral among the Internet engineering experts of the world.

On Sep 20, 2013, at 7:56 PM, Thomas S Hatch notifications@github.com wrote:

Yes, we need to do this, I think that the first step is to define what messages should be being sent were and write a spec on it.
I think that a generic document is the first step, one that could be like a pep8 for logging, then we can layer salt specifics on top of it


Reply to this email directly or view it on GitHub.

@madduck

This comment has been minimized.

Show comment
Hide comment
@madduck

madduck Sep 21, 2013

Contributor

I would just like to say that standards don't get defined, standard documents document existing best practice. But yes, we should "standardise" log levels, but please leave it up to the administrator to control, which messages at which level get logged to which channels.

Contributor

madduck commented Sep 21, 2013

I would just like to say that standards don't get defined, standard documents document existing best practice. But yes, we should "standardise" log levels, but please leave it up to the administrator to control, which messages at which level get logged to which channels.

@stale stale bot added the stale label May 13, 2017

@stale

This comment has been minimized.

Show comment
Hide comment
@stale

stale bot May 13, 2017

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

stale bot commented May 13, 2017

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

@stale stale bot closed this May 20, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment