piping stdout from one process into tar fails

[retrogradeorbit]

~/envoy$ tar cfz test.tgz envoy/
~/envoy$ cat test.tgz | tar -tz
envoy/
envoy/init.pyc
envoy/init.py
envoy/core.py
envoy/core.pyc
~/envoy$ python
Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:48)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import envoy
r = envoy.run("cat test.tgz | tar -tz")
r.status_code
2
r.std_err
'\ngzip: stdin: invalid compressed data--format violated\ntar: Child
returned status 1\ntar: Error is not recoverable: exiting now\n'

[volpino]

I think it is due to https://github.com/kennethreitz/envoy/blob/master/envoy/core.py#L205
I'm having the same issue

[ptwobrussell]

I think I'm getting burned by this same issue as well. A workaround is to have envoy call a bash script that does the piping for you. It's kind of janky, but it works.

[volpino]

@ptwobrussell try my patch #57

[ptwobrussell]

@volpino - Thanks for the suggestion. I will try it out. Right now, my use case is basically piping hundreds of megabytes of output from one process directly into gzip, so this will be a good test for your patch. It's essentially something like this: "processThatGeneratesALotOfOutput | gzip -c > foo.gz"

Do you know if the stdout redirect operator is supported by envoy, though? Seems that I tried to do something like "processThatGeneratesALotOfOutput > foo" followed by "gzip foo" and I ran into issues redirecting the stdout into "foo", but I might be mistaken.

[volpino]

Yes, my patch should work with that.
I don't think that the > is supported. You could just use tee

[kennethreitz]

This project is in a bit of a crisis state — it's really useful, and I use it on a daily basis. However, I wrote it in a few afternoons several years ago and haven't touched it since. In order to get the project into a stable state I'm closing all issues and pull requests

Don't take this as aggressive — it's just necessary for the project to make any progress any time soon (it's pretty clear the project is effectively unmaintained at the moment). Great things to come! Please watch the GitHub logs and feel free to re-open this discussion soon. I just need to really it into a good state first.

✨ ❤️ ✨