Missing stdout line when process is waiting for input

[mgzenitech]

Minimal example to reproduce (running Terraform as a subprocess):

from subprocessor import Sub

process = Sub("terraform apply")
for is_stdout, line in process:
    print(line, end = "")

Output:

Changes to Outputs:
  + test = "a"

You can apply this plan to save these new output values to the Terraform
state, without changing any real infrastructure.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

However we are missing one more line with input:

Changes to Outputs:
  + test = "a"

You can apply this plan to save these new output values to the Terraform
state, without changing any real infrastructure.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value:

It appears however if I press enter afterwards

[rec]

Hello, and sorry about the delay, I unfortunately have a Monday deadline and am scribbling on that.

And thanks for the good, clear bug report - I didn't even have to look at my code to know what was wrong, though I did validate it.

It's my belief that it is actually "working as intended" except that "intended" in your use case has a less useful result for you. :-/

sproc's code doesn't decide when a line ending happens - that is left up to Python's I/O system, right here.

Unfortunately, input() doesn't send a line ending typically, and so Python doesn't send me the line until after the input has taken in the value, just as you report.

---

I can't change the default behavior, but what I could do is this - to add an optional boolean flag that switches the behavior from "yielding complete lines" to "yielding what comes out as it comes out" by using partial reads instead of readlines in that code above.

My feeling is unfortunately that it would be 15 minutes to write, and an hour to effectively test. My plan is to carefully make the change and make a new release with this "untested undocumented boolean flag" which you can then test and see if it fixes your problem or not.

[rec]

Bad news: I don't see a way to do it effectively.

https://docs.python.org/3/library/asyncio-stream.html#streamreader claims to have a readuntil method, but I don't see it a way to use it in practice, because of the buffering that hides this.

I'm going to have to put this on hold for a bit, let me know if you have any ideas.

[mgzenitech]

Hm... I'm not that good on low level stuff :/ maybe it is possible to yield either complete line or everything buffered (even without newline character) when the subprocess is waiting for some input somehow? Or disable buffering or something...

[rec]

One way to do this would be to read single characters at a time with a timeout and then yield them. It would be somewhat slow, but that might be literally nothing compared to the time you are spending in the subprocess calling terraform! I could write that fairly fast. No matter what you do, this is going to be fragile. For example, if the program you were calling removes that extra carriage return or add a new one, everything will be thrown off.

…

On Mon, Dec 12, 2022 at 10:37 AM mgzenitech ***@***.***> wrote: Hm... I'm not that good on low level stuff :/ maybe it is possible to yield either complete line *or* when the subprocess is waiting for some input somehow? Or disable buffering or something... — Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB53MTHXYAPMV55EABJ6XTWM3W3XANCNFSM6AAAAAASZNPXXY> . You are receiving this because you were assigned.Message ID: ***@***.***>

-- /t PGP Key: ***@***.*** *https://tom.ritchford.com <https://tom.ritchford.com>* *https://tom.swirly.com <https://tom.swirly.com>*

[rec]

Well, I wasn't right. It really doesn't seem to be easy to do at all without going into lower level stuff, which isn't cross-platform, and I don't have the time and facilities to even start on that.

There must be some other way to directly script terraform!

[mgzenitech]

I think Terraform is just an example here, but any app that requires input and does not print newline before that will be affected, no? Maybe it would be possible to create something up for at least unix based systems?

[rec]

So I added something that might actually do what you want.

There's a new parameter by_lines to sproc.Sub that's set to True by default. If you set it to False you get everything in one call to the underlying stream: https://docs.python.org/3/library/io.html#io.BufferedReader.read1

I made a new release. Update, and then set sproc.Sub(..., by_lines=False).

Lemme know!!

[mgzenitech]

superb! Works as expected now! Thanks :)

[rec]

Aces!!!!!!!