Difference between this and ubelt.cmd, others? What makes this problem so difficult?

[tony]

Greetings! 👋

I am trying to understand the lay of the land of cross-platform teeing of subprocess (py). I've also found ubelt.cmd (tests) by @Erotemic.

I originally got here through StackOverflow in a comment, but I can't find the post.

In either event, the problem comes up a lot:

• How to replicate tee behavior in Python when using subprocess?
• Subprocess.Popen: cloning stdout and stderr both to terminal and variables
• How to write to stdout AND to log file simultaneously with Popen?

I am racking my brain when I read the code / solutions for this issue and wondering:

• What makes this issue (subprocess + tee) so tricky to solve? e.g. Is it the program's IO? The environment / system? asyncio?
• What are these libraries, such as subprocess-tee, doing with the command?
• Does ubelt.cmd do the same thing? Is it different n some way?
• Is this the kind of thing a standard library PEP would be able to tackle in a cross-platform way?

Sorry for any interruption, as this is a non-code post and a bit meta 😄

[ssbarnea]

Until I read this, I did not know about ubelt, even if apparently it is popular on github. Still, what worries me about it is "too broad scope" and the fact that it has only one contributor. Even if library which is far less popular has 5.

[tony]

Hi there. My hope is to understand the underlying technical problem better.

"too broad scope"

It this referring to ubelt as a whole package, or ubelt's cmd module? I'm just referring to the ubelt.cmd, which supports tee'ing the output.

has only one contributor

It's @Erotemic, who also authored https://github.com/Erotemic/xdoctest, and made a nice video about it here: https://www.youtube.com/watch?v=CUjCqOw_oFk. I was impressed by this and would trust them 100%. Also I own quite a few projects I'm the sole maintainer of 😆

I also read ubelt's code and it looks fine - though I still don't get why subprocess + tee is so difficult, which is why I wrote this :)

Back to the subprocess issue itself, where do you stand with the underlying subprocess+tee issue? What makes it so difficult? What does this library do to make it easier? I'm still stumped with what's happening underneath the hood.

[Erotemic]

@ssbarnea I'm not sure if extensions to stdilb is too broad of a scope. It's similar to boltons in that way. But I do agree that it might be better to refactor ubelt into several subpackages that ubelt itself unifies. But in it's defense, the package is relatively small and lightweight (and will get even lighter weight once I deprecate 3.6 support and use lazy imports). The use case for it is to provide easy access for tasks that I believe should be supported by the stdlib with a single function call: e.g. downloading a file, running a subprocess, adding a progress bar to a loop, set operations on dictionaries, hashing a file or nested data structure, disk-caching the result of a computation, simplified concurrent.futures, etc... These are fairly generic operations I always want quick access to in my toolbelt.

Also, I was not aware of subprocess-tee (the ubelt.cmd was written a few years before this seems to have come out). I've also been thinking about separating ubelt.cmd into a separate standalone module because the problem of dealing with subprocesses and teeing them could use a bit more of a focused approach.

@tony I'm glad you trust me 100%, that's way more than I trust myself, I'd recommend subtracting at least 0.0001% from your confidence, if only to allow for overwhelming evidence to override your trust 😆

I took a look at your code for subprocess+tee, and it seems to make heavy use of asyncio, whereas in ubelt I have 2 backends: threading and select. The former is much faster and works cross-platform, whereas the latter was easier to implement and only works on unix. One complication in the ubelt.cmd approach is that pre-ubelt 1.0, Python 2.7 support was required. That is no longer a constraint, so maybe the asyncio makes the solution easier?

I'll also note that the implementation in ubelt isn't perfect (although it's very good and works in the majority of situations). It can produce a different ordering of stdout and stderr than an equivalent invocation in a bash shell would produce. There are also situations where interactive processes don't work exactly right (although they often do), but I recently had an issue where gpg didn't prompt me correctly when using ubelt.cmd. I haven't tracked down the exact cause of that.

As far as why is this so hard? In the ubelt code, there are also a few references to stack overflow discussions that I pulled from when researching how to correctly implement this, perhaps they can be helpful here as well. I do remember getting it to work simultaneously on win32, linux, 2.7 and 3.x to be an absolute nightmare. I also think it's tricky to think about asynchronous programs, which perhaps caused unnecessary complexity?

An exercise that might be interesting it taking the ubelt tests and checking to see if they work correctly with this library as well.

Lastly, one thing that might benefit this library is the ubelt feature where it will transform a string-formatted command or a list-formatted command into the appropriate format depending on if shell=True or False. Feel free to blatantly steal any implementation from ubelt and port it over here. If this library fully implements everything ubelt.cmd depends on, I'd be inclined to the implementation and simply point to this repo instead to implement the API under the condition this is well maintained (which the pycontribs org seems to do).