-P should imply --linewise #482

Closed
bitprophet opened this Issue Nov 18, 2011 · 12 comments

Projects

None yet

3 participants

@bitprophet
Member

Rather, ditto for the env vars -- when linewise is being tested for, have the test be "linewise or parallel".

Realistically, linewise shouldn't even be an exposed option as it's almost never going to be useful except in parallel mode; but now that it's out, it can't hurt to leave it as-is so folks can use it if they want.

The inverse, forcing bytewise parallelism, strikes me as an unusual case that can be safely ignored until somebody specifically asks for it.

@bitprophet bitprophet was assigned Nov 18, 2011
@shaftoe
shaftoe commented Sep 11, 2012

Seems that there's been a regression, because env.linewise is False even when @parallel decorator is used. Crf. https://gist.github.com/3697165
We noticed doing a parallel deploy and getting bytewise output. Fabric v.1.4.3 on MacOSX 10.7.4

@bitprophet
Member

I can't replicate this locally -- can you paste your output & the code+invocation used to generate it?

@shaftoe
shaftoe commented Sep 12, 2012

There should be everything in the gist I linked:
https://gist.github.com/3697165

@bitprophet
Member

Sorry, I meant I was hoping for an example of the literally garbled output (and the command generating it), because even though linewise is printing out False (which is certainly odd) I can't actually get bytewise output to occur, everything still ends up line by line when I do e.g. a flight of parallel ls or similar.

@bitprophet
Member

Guess I wasn't trying hard enough, adding even more hosts + changing the ls to a directory with even more files gets it to show up.

What's odd is I already traced the environment updating stuff that sets the 'parallel' and 'linewise' flags, and they are definitely firing, so something else must be going on. I'll figure out wtf. Thanks for the report :)

@shaftoe
shaftoe commented Sep 12, 2012

No problem! Was going to answer that for an output report I'll need to wait until tomorrow cause it's the very task we use to do deploys and I'll have to wait for a dev to do it ;)

@shaftoe
shaftoe commented Sep 14, 2012

This is an example of output we still get mixed up even forcing linewise via decorator. Actually seems more there's a missing line terminator than byte mixing, but maybe it could be of interest too:

[ec2-xxx.eu-west-1.compute.amazonaws.com.] out: [ec2-yyy.eu-west-1.compute.amazonaws.com.] out: INFO: www-redirect SVN update
INFO: www-profiler SVN update

@bitprophet
Member

I am pretty sure I've seen "missing" newlines before and am not 100% sure that's the same issue. Also, I don't believe the decorator will necessarily help -- like I mentioned, the higher level code that tries to ensure the subprocess runs with env.linewise = True is firing correctly, so it must be something lower down that may be ignoring the setting altogether.

@dmcquay
dmcquay commented Oct 2, 2012

I'm having this same problem and it's definitely not just a newline issue. My output is very garbled. Here's an example:

y[prod1.athlete.com] out: Cso'p
/[prod3.athlete.com] out: Cngo p'y/iunsgr /'l/ohcoamle//liubb/upnyttuh/oant2h.l7e/tdei_stt-mppa/cpkuabgleisc//djjsa-nsgro/cc/lonitbr/ipbr/oatdomtiynp/es.tjatsi'c
u[prod3.athlete.com] out: Cdompiyni/njgs /'c/alheonmdea/ru.bjusn't
/[prod1.athlete.com] out: Caotphylinegt e'_/tumspr//pluobclailc/l/ijbs/-psyrtch/olni2b./7g/adrismti-np/auctkialg/eUst/idlj-anBgroo/acdocnatsrtiebr/.ajdsm'in

It seems to only happen when the output is really cooking, so that makes it a little harder to reproduce in a small example script. What is really easy to reproduce though is that if I define a task with the parallel decorator and immediately print out env.parallel and env.linewise, they are both False.

Snippet of my fabfile.py:

@task
@parallel
@roles('webapp')
def deploy(tarball=None):
print("parallel: " + repr(env.parallel))
print("linewise: " + repr(env.linewise))
return

And the resulting output:

$ fab production deploy
[prod1.athlete.com] Executing task 'deploy'
[prod2.athlete.com] Executing task 'deploy'
[prod3.athlete.com] Executing task 'deploy'
parallel: False
linewise: False
parallel: False
linewise: False
parallel: False
linewise: False

Done.

$ fab --version
Fabric 1.4.3
ssh (library) 1.7.14

@shaftoe
shaftoe commented Oct 9, 2012

@dmcquay how are you working around the problem? Forcing env.linewise directly into the def?

@bitprophet
Member

Hey guys -- please take any further discussion over to #767. I will be trying some fixes ASAP and would love feedback once something gets pushed.

@bitprophet
Member

Think I do have a fix. Again -- please follow #767 and give feedback for the fix there. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment