Support full logging to file

[bitprophet]

Description

Outside of any further modifications to the stdout/stderr output controls, it would be very handy to log everything to a file; this would give users another alternate channel for debugging their fabfiles without having to wrestle with what they see at runtime.

Main question is whether to output things as if debug were set, by default. Thinking to leave it off at first, and have a simple flag/option to turn it on, perhaps --log-debug.

Where to log to? By default I'd say user's cwd, though that can get annoying (like "pipturds", i.e. pip-log.txt files that pip drops everywhere.) Unfortunately there's no other great standard location, so perhaps turn logging-to-file off by default? Either way, allow override, say via --log-location or similar.

Unsure whether it makes any sense to utilize the logging module's concept of levels; the only place I can see it being useful at all is for the debug stuff, but we currently think of debugging as modifying output instead of adding to it -- which doesn't mesh with how logging works. So possibly best to just stick everything in, say, INFO for now, then tweak later if necessary.

---

Potential modules:

• stdlib logging
  • Pros: stdlib, well known/documented
  • Cons: kind of byzantine/complicated to use
• Logbook
  • Pros: Armin
• Twiggy
• Others?

---

Originally submitted by Jeff Forcier (bitprophet) on 2009-09-06 at 10:57am EDT

Relations

• Related to ANSI Color support #101: ANSI Color support
• Duplicated by All output should be sent through logging module #135: All output should be sent through logging module
• Related to Make fabfile print() statements controllable via output controls #151: Make fabfile print() statements controllable via output controls
• Related to Allow configuring (vs just toggling) v1's output line prefix #163: Formattable output prefix.
• Related to Add additional, verbose-only output when connecting #244: Add additional, verbose-only output when connecting
• Related to Consider adding "global"/per-task capture of run/sudo/local stdout/stderr #71: Consider adding "global"/per-task capture of run/sudo/local stdout/stderr
• Duplicated by Use logger() instead of print() #333: Use logger() instead of print()

[bitprophet]

Travis Swicegood (tswicegood) posted:

---

Carrying over the conversation from #135, the core logging module is what should be used for all output. That gets logging to file for free as a side-effect, but more importantly provides a uniform way to handle formatting output.

---

on 2010-01-24 at 09:12pm EST

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

See #333 for a pull request that does some basic work in this area (though ATM it does not pass tests.)

Keep in mind that I want to at least quickly compare the existing logging modules to see if the non-stdlib ones offer enough benefits (cleaner APIs etc) over the stdlib one to be worth using.

---

on 2011-04-19 at 12:43pm EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Some prior art using the stdlib module can be found in this GH commit (came from a pull request)

---

on 2011-07-07 at 03:09pm EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

And recent work by tswicegood, also using the stdlib module and wrapping it a bit, is here on Github

---

on 2011-07-07 at 03:38pm EDT

[bitprophet]

Adding Invoke label since Invoke will need to log stuff too, and Fabric will then ride on that to some degree (while obviously logging a lot of its own stuff too.)

[andresriancho]

+1

[ijstokes]

A big +1 from me.

Has this been added to fabric yet? A bit of a chore to remember to pass every fab command through tee.

[agroszer]

yes please, that's somehow a must

[thedotedge]

👍

[josegonzalez]

@bitprophet is there something we can help with to bring this feature to life?

[hodduc]

👍

[budlight]

+1

[chrisguidry]

👍

[loris]

+1

[josegonzalez]

The current hack I have for this is to set stdout and stderr to a fake logger class, which seems to work well enough for me:

https://github.com/josegonzalez/chef-solo-cup/blob/master/chef_solo_cup/commands/runner.py#L141

[josegonzalez]

Note: it seems like interactive shells - like asking for passwords - might break in my above implementation, so I still think it would be nice to have it in the core of fabric.

[dreampuf]

+1

I'm using a ad-hot decorator. But this decorator could not support execute function when the task was parallel running ( you can't get the stdout of multiprocess env )

class Buffered:
    def __init__(self, stream):
        self.stream = stream
        self.buf = []
    def write(self, data):
        self.buf.append(data)
        self.stream.write(data)
        self.stream.flush()
    def __getattr__(self, attr):
        return getattr(self.stream, attr)

def auto_comment(func):
    @wraps(func)
    def wraper(*args, **kws):
        _stdout = sys.stdout
        buffered = sys.stdout = Buffered(_stdout)
        ret = func(*args, **kws)
        if buffered.buf:
            # only pick last 500 lines is good enough
            echo_result = "".join(buffered.buf[-500:])
            # filter the colored results
            echo_result = re.sub("\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]", "", echo_result)
            # using the result as you want 
        sys.stdout = _stdout
        return ret
    return wraper

[thedotedge]

Is there any plan to include this in the core any time soon?

[fruch]

+1

[ghost]

+1

[ChrisMarinos]

+1

[mbra]

👍

[msabramo]

I also would love a solution to this.

[hamiltont]

+1

Also, this may help others trying to build a streamlogger

[yshao]

This feature would be nice to have!

[satels]

+1

[zodman]

+1

[marthjod]

+1

[zodman]

i offer 10bucks for this feature!

[bitprophet]

Rolling this into pyinvoke/invoke#15, updated its desc with a bunch of thoughts too.

[guhuajun]

Sorry for replying a closed issue.
I am using https://github.com/algrebe/python-tee to redirect stdout to a file.
Hope this project can help others.