New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix mixed output in parallel mode #965

Closed
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
2 participants
@akitada

akitada commented Aug 26, 2013

No description provided.

@bitprophet

This comment has been minimized.

Member

bitprophet commented Aug 26, 2013

Feels possibly related to (now pretty old/outdated =/) #508. If that is the same issue I suspect the change in this one here is more solid.

@akitada

This comment has been minimized.

akitada commented Aug 26, 2013

How to reproduce "mixed output".

First, add host entries to /etc/hosts for parallel test.

mv /etc/hosts /etc/hosts.orig
for i in {0..4}; do
    echo 127.0.0.1       `hostname`$i
done >> /etc/hosts

And run this fabfile by fab ssh_test | grep '\[.*\['

import socket
from fabric.api import *

env.timeout = 60
env.connection_attempts = 100


@parallel
def _ssh_test():
    run('ls /')


@task
def ssh_test():
    env.hosts = ['%s%d' % (socket.gethostname(), i) for i in xrange(5)]
    for _ in xrange(100):
        with hide('running'):
            execute(_ssh_test)
@akitada

This comment has been minimized.

akitada commented Aug 26, 2013

This patch doesn't always work.
The write size must be PIPE_BUF or less and it must not be a write to a file.
http://pubs.opengroup.org/onlinepubs/9699919799/functions/write.html

@akitada

This comment has been minimized.

akitada commented Nov 21, 2013

I've been using this patch for 3 months without any issues.

@bitprophet

This comment has been minimized.

Member

bitprophet commented Dec 15, 2013

Confirmed that I can replicate the "double line prefix" issue by using the above snippet, from/to my localhost (MBA 11" mid-2013 model.)

Re: PIPE_BUF I assume that maps to http://docs.python.org/2/library/select.html#select.select.PIPE_BUF ?

I presume the suggested fix is basically "don't call flush() when in linewise mode" (and commutatively, in parallel mode, since parallel forces linewise.) Will test that out in a moment.

@bitprophet

This comment has been minimized.

Member

bitprophet commented Dec 15, 2013

Confirmed, with change applied, I don't see any doubled lines (using grep.)

I'm worried this may cause things to not flush as often as they should in some situations (e.g. fast loops) but willing to merge this anyway in hopes that's not the case :) thanks!

bitprophet added a commit that referenced this pull request Dec 15, 2013

@bitprophet

This comment has been minimized.

Member

bitprophet commented Dec 15, 2013

Merging this to master, feels like the kind of bugfix that should come out in a 'feature' release.

@bitprophet bitprophet closed this Dec 15, 2013

@akitada akitada deleted the akitada:fix-mixed-output branch Dec 15, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment