New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logging & other output concerns #15

bitprophet opened this Issue Jul 6, 2012 · 1 comment


None yet
1 participant

bitprophet commented Jul 6, 2012

Turning this into a general "how to do logging and useful output controls for invoke and fabric 2" ticket.

Use cases

  • Display the task or tasks being invoked so users can follow the "flow" of nontrivial execution
    • plus parameters
    • plus dependency chain info
      • even skipped/satisfied dependencies
  • Display subprocess commands being called so users can tell what's actually being done on their behalf
    • plus which task is currently in play (optionally) in case the per-task info has already scrolled by
  • Display some sort of 'prefix', on a line-by-line basis, prepended to subprocess stdout/err - such as the command, task or task parameter that spawned it
    • Command itself less likely (esp since most command strings tend to be long) but task or task param is useful for things like parallel execution (#63) or Fabric per-host exec.
  • Probably more besides...
  • Must be able to turn a lot of this off or on at will
  • May require an entirely different stdout/err mirroring/display mode when prefixes are enabled, since otherwise they'd get in the way of bytewise output (think Unicode decoding, interactive sessions, etc)

Older thoughts

Thoughts for now, jumping off of the description of fabric/fabric#57 & current state of things in Invoke master:

  • Re: log lib to use:
    • May not do anything besides stdlib logging after all, it's still the most-used option, and the core problem of "what to log, when, and how" is not going to change drastically even if we use another lib. It's Not That Bad, Really™.
    • Structlog came around after that original ticket started, and actually looks like the best option if we want to add anything on top of stdlib (in part as it's the only major one that supports 2.6-3.x, has lots of good features, actually runs on top of stdlib or whatever you want, and I am mildly biased as I know the author :3)
    • Twiggy is 2.x only, so nope.
    • Logbook is 2.7-3.x, so until we're comfortable dropping 2.6 (not for a while...) it's also a no-go. Still a decent looking option otherwise.
  • Log a lot of things by default, mostly at DEBUG for, well debugging. E.g. logic paths used, etc - good for troubleshooting.
    • Invoke does this already; it simply offers invoke.utils.debug (w/ setup to add other levels easily) that logs to a package-level logger. Could probably be expanded some, it's mostly used in the parsing and config modules.
  • Log 'operational' level things as INFO or similar, sans any 'large' fields - i.e. log when run is called, on which host (in Fabric), with which command (tho this may want truncating...), run-time, etc - but not the stdout/stderr contents (though sizes of those, similar to HTTP log formats, is probably good).
    • Basically, the same data we expose in Result objects.
    • Re: this ticket's original aegis, DO we want to be logging the stdout/stderr ever, at any point? Or should we rely on users' ability to redirect those streams where they need?
    • They could, for example, just point both at their process' stdout/err, which would mix log output with command stdout/stderr well enough...
  • I was wondering whether to log both the initiation of, and completion of, actions, or just completion (as above). Seems like best is to log initiation as DEBUG or something else below INFO (since you mostly care when troubleshooting) and completion as INFO.
  • When I started fabric/fabric#57 way back, I was assuming we'd want ALL output to use the logging system, with an open question about how to handle bytewise output (as it's not line-based).
    • I no longer think this makes sense - simply log a lot of stuff, as above, and don't forward it to stdout/stderr by default.
    • Allow users to determine where their logs go (usually to a file).
    • When run via the CLI, default to a "Fabric 1-like" mode where basic info is printed to stdout/stderr - what task is run, what command is run, hook up its stdout/err, file transfer info, etc.
    • When NOT run via the CLI, probably default to no output at all - expect users to supply desired file-like objects to e.g. run's stream arguments, and to configure logging to taste.
      • Users wanting a 'hybrid' setup would do what I mentioned above, point both the runners' streams AND the logging framework to system pipes.
  • Regarding Fabric specifically, how to handle multi-host invocation?
    • Having line prefixes is still useful for some use cases (especially, but not only, for parallel execution), which also means, we probably do want to retain a line-buffer at some level
    • Having 'headers' is another useful mode I've wanted at times, where I don't want the clutter of line prefixes, but do want to be able to read output sequentially and tell what happened on which host.

Original/outdated description

Right now stdout/stderr is printed in our Popen subclass byte-by-byte to allow interaction etc. However, eventually we'll want real logging, meaning an ability to throw things line-by-line into a log library, meaning an extra buffer at the Popen level that does additional Things with lines as they are tallied up.

I suppose there should be an option to turn off the bytewise in favor of just linewise, but that could be a 2nd tier thing or saved for when we start layering Fab 2 on top of this.


This comment has been minimized.

Show comment
Hide comment

bitprophet Jul 5, 2018


Calling out #546 and #547 as recently filed tickets with specifically phrased use cases that fall under this aegis.


bitprophet commented Jul 5, 2018

Calling out #546 and #547 as recently filed tickets with specifically phrased use cases that fall under this aegis.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment