New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

-P should imply --linewise #482

Closed
bitprophet opened this Issue Nov 18, 2011 · 12 comments

Comments

Projects
None yet
3 participants
@bitprophet
Member

bitprophet commented Nov 18, 2011

Rather, ditto for the env vars -- when linewise is being tested for, have the test be "linewise or parallel".

Realistically, linewise shouldn't even be an exposed option as it's almost never going to be useful except in parallel mode; but now that it's out, it can't hurt to leave it as-is so folks can use it if they want.

The inverse, forcing bytewise parallelism, strikes me as an unusual case that can be safely ignored until somebody specifically asks for it.

@ghost ghost assigned bitprophet Nov 18, 2011

@shaftoe

This comment has been minimized.

Show comment
Hide comment
@shaftoe

shaftoe Sep 11, 2012

Seems that there's been a regression, because env.linewise is False even when @parallel decorator is used. Crf. https://gist.github.com/3697165
We noticed doing a parallel deploy and getting bytewise output. Fabric v.1.4.3 on MacOSX 10.7.4

shaftoe commented Sep 11, 2012

Seems that there's been a regression, because env.linewise is False even when @parallel decorator is used. Crf. https://gist.github.com/3697165
We noticed doing a parallel deploy and getting bytewise output. Fabric v.1.4.3 on MacOSX 10.7.4

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Sep 11, 2012

Member

I can't replicate this locally -- can you paste your output & the code+invocation used to generate it?

Member

bitprophet commented Sep 11, 2012

I can't replicate this locally -- can you paste your output & the code+invocation used to generate it?

@shaftoe

This comment has been minimized.

Show comment
Hide comment
@shaftoe

shaftoe Sep 12, 2012

There should be everything in the gist I linked:
https://gist.github.com/3697165

shaftoe commented Sep 12, 2012

There should be everything in the gist I linked:
https://gist.github.com/3697165

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Sep 12, 2012

Member

Sorry, I meant I was hoping for an example of the literally garbled output (and the command generating it), because even though linewise is printing out False (which is certainly odd) I can't actually get bytewise output to occur, everything still ends up line by line when I do e.g. a flight of parallel ls or similar.

Member

bitprophet commented Sep 12, 2012

Sorry, I meant I was hoping for an example of the literally garbled output (and the command generating it), because even though linewise is printing out False (which is certainly odd) I can't actually get bytewise output to occur, everything still ends up line by line when I do e.g. a flight of parallel ls or similar.

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Sep 12, 2012

Member

Guess I wasn't trying hard enough, adding even more hosts + changing the ls to a directory with even more files gets it to show up.

What's odd is I already traced the environment updating stuff that sets the 'parallel' and 'linewise' flags, and they are definitely firing, so something else must be going on. I'll figure out wtf. Thanks for the report :)

Member

bitprophet commented Sep 12, 2012

Guess I wasn't trying hard enough, adding even more hosts + changing the ls to a directory with even more files gets it to show up.

What's odd is I already traced the environment updating stuff that sets the 'parallel' and 'linewise' flags, and they are definitely firing, so something else must be going on. I'll figure out wtf. Thanks for the report :)

@shaftoe

This comment has been minimized.

Show comment
Hide comment
@shaftoe

shaftoe Sep 12, 2012

No problem! Was going to answer that for an output report I'll need to wait until tomorrow cause it's the very task we use to do deploys and I'll have to wait for a dev to do it ;)

shaftoe commented Sep 12, 2012

No problem! Was going to answer that for an output report I'll need to wait until tomorrow cause it's the very task we use to do deploys and I'll have to wait for a dev to do it ;)

@shaftoe

This comment has been minimized.

Show comment
Hide comment
@shaftoe

shaftoe Sep 14, 2012

This is an example of output we still get mixed up even forcing linewise via decorator. Actually seems more there's a missing line terminator than byte mixing, but maybe it could be of interest too:

[ec2-xxx.eu-west-1.compute.amazonaws.com.] out: [ec2-yyy.eu-west-1.compute.amazonaws.com.] out: INFO: www-redirect SVN update
INFO: www-profiler SVN update

shaftoe commented Sep 14, 2012

This is an example of output we still get mixed up even forcing linewise via decorator. Actually seems more there's a missing line terminator than byte mixing, but maybe it could be of interest too:

[ec2-xxx.eu-west-1.compute.amazonaws.com.] out: [ec2-yyy.eu-west-1.compute.amazonaws.com.] out: INFO: www-redirect SVN update
INFO: www-profiler SVN update

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Sep 14, 2012

Member

I am pretty sure I've seen "missing" newlines before and am not 100% sure that's the same issue. Also, I don't believe the decorator will necessarily help -- like I mentioned, the higher level code that tries to ensure the subprocess runs with env.linewise = True is firing correctly, so it must be something lower down that may be ignoring the setting altogether.

Member

bitprophet commented Sep 14, 2012

I am pretty sure I've seen "missing" newlines before and am not 100% sure that's the same issue. Also, I don't believe the decorator will necessarily help -- like I mentioned, the higher level code that tries to ensure the subprocess runs with env.linewise = True is firing correctly, so it must be something lower down that may be ignoring the setting altogether.

@dmcquay

This comment has been minimized.

Show comment
Hide comment
@dmcquay

dmcquay Oct 2, 2012

I'm having this same problem and it's definitely not just a newline issue. My output is very garbled. Here's an example:

y[prod1.athlete.com] out: Cso'p
/[prod3.athlete.com] out: Cngo p'y/iunsgr /'l/ohcoamle//liubb/upnyttuh/oant2h.l7e/tdei_stt-mppa/cpkuabgleisc//djjsa-nsgro/cc/lonitbr/ipbr/oatdomtiynp/es.tjatsi'c
u[prod3.athlete.com] out: Cdompiyni/njgs /'c/alheonmdea/ru.bjusn't
/[prod1.athlete.com] out: Caotphylinegt e'_/tumspr//pluobclailc/l/ijbs/-psyrtch/olni2b./7g/adrismti-np/auctkialg/eUst/idlj-anBgroo/acdocnatsrtiebr/.ajdsm'in

It seems to only happen when the output is really cooking, so that makes it a little harder to reproduce in a small example script. What is really easy to reproduce though is that if I define a task with the parallel decorator and immediately print out env.parallel and env.linewise, they are both False.

Snippet of my fabfile.py:

@task
@parallel
@roles('webapp')
def deploy(tarball=None):
print("parallel: " + repr(env.parallel))
print("linewise: " + repr(env.linewise))
return

And the resulting output:

$ fab production deploy
[prod1.athlete.com] Executing task 'deploy'
[prod2.athlete.com] Executing task 'deploy'
[prod3.athlete.com] Executing task 'deploy'
parallel: False
linewise: False
parallel: False
linewise: False
parallel: False
linewise: False

Done.

$ fab --version
Fabric 1.4.3
ssh (library) 1.7.14

dmcquay commented Oct 2, 2012

I'm having this same problem and it's definitely not just a newline issue. My output is very garbled. Here's an example:

y[prod1.athlete.com] out: Cso'p
/[prod3.athlete.com] out: Cngo p'y/iunsgr /'l/ohcoamle//liubb/upnyttuh/oant2h.l7e/tdei_stt-mppa/cpkuabgleisc//djjsa-nsgro/cc/lonitbr/ipbr/oatdomtiynp/es.tjatsi'c
u[prod3.athlete.com] out: Cdompiyni/njgs /'c/alheonmdea/ru.bjusn't
/[prod1.athlete.com] out: Caotphylinegt e'_/tumspr//pluobclailc/l/ijbs/-psyrtch/olni2b./7g/adrismti-np/auctkialg/eUst/idlj-anBgroo/acdocnatsrtiebr/.ajdsm'in

It seems to only happen when the output is really cooking, so that makes it a little harder to reproduce in a small example script. What is really easy to reproduce though is that if I define a task with the parallel decorator and immediately print out env.parallel and env.linewise, they are both False.

Snippet of my fabfile.py:

@task
@parallel
@roles('webapp')
def deploy(tarball=None):
print("parallel: " + repr(env.parallel))
print("linewise: " + repr(env.linewise))
return

And the resulting output:

$ fab production deploy
[prod1.athlete.com] Executing task 'deploy'
[prod2.athlete.com] Executing task 'deploy'
[prod3.athlete.com] Executing task 'deploy'
parallel: False
linewise: False
parallel: False
linewise: False
parallel: False
linewise: False

Done.

$ fab --version
Fabric 1.4.3
ssh (library) 1.7.14

@shaftoe

This comment has been minimized.

Show comment
Hide comment
@shaftoe

shaftoe Oct 9, 2012

@dmcquay how are you working around the problem? Forcing env.linewise directly into the def?

shaftoe commented Oct 9, 2012

@dmcquay how are you working around the problem? Forcing env.linewise directly into the def?

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Oct 28, 2012

Member

Hey guys -- please take any further discussion over to #767. I will be trying some fixes ASAP and would love feedback once something gets pushed.

Member

bitprophet commented Oct 28, 2012

Hey guys -- please take any further discussion over to #767. I will be trying some fixes ASAP and would love feedback once something gets pushed.

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Oct 29, 2012

Member

Think I do have a fix. Again -- please follow #767 and give feedback for the fix there. Thanks!

Member

bitprophet commented Oct 29, 2012

Think I do have a fix. Again -- please follow #767 and give feedback for the fix there. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment