Follow STDIN input

[vtronko]

I expect sd to follow the STDIN, as it would do in sed.
For example, I expect
journalctl -u dnscrypt-proxy.service -f | sd machinename home
to give consistent live output, as
journalctl -u dnscrypt-proxy.service -f | sed '/s/machinename/home/
would. Instead, it just waits for EOF, due to
handle.read_to_end(&mut buffer)?;

I'd suggest to use terminal buffering mode (flush buffer at newline), but that'd break multiline replacement.

This is an important usecase for log inspection routines with custom formatting.

[jnnkB]

This would be really cool and could perhaps be archived by adding an line by line mode.

[sebastianblunt]

Easy to replicate with (echo a && sleep 5 && echo a) | sd a b. Probably related to #96

[varenc]

I'm also struggling with this issue. Usually the stdbuf or the unbuffer tools are able to force programs to be line buffered, but no luck using those for sd. (stdbuf relies on tweaking the version of libc that's loaded, but since sd is written in rust, I assume that's why it doesn't work).

When I want to view the live output of a series of pipes, I sadly have to switch back to using sed even though I vastly prefer sd. GNU Sed has an --unbuffered flag which instantly accomplishes this.

[intelfx]

This is not related to buffering. sd seems to work by reading whole stdin until EOF into memory prior to performing any work on it. Thus, sd it is not really a sed alternative as it is not "streaming".

[brunetton]

Same here (super useful comment, I know)

[vtronko]

This is not related to buffering. sd seems to work by reading whole stdin until EOF into memory prior to performing any work on it. Thus, sd it is not really a sed alternative as it is not "streaming".

Exactly why this is related to line-based buffering, the lack thereof…

[vtronko]

For what it's worth, I created a fix which works, but I need to create more tests for it. I will publish the code later.

[borestad]

Any progress on this? 🙂

[corneliusroemer]

It looks like @vtronko's commits on his fork work - I tested them locally. They need to be rebased onto the current main but otherwise looks good. @chmln can you have a look at: vtronko@fcd59ac?

[CosmicHorrorDev]

I know @dev-ardi has interest in making this change. The limiting factor has been maintainer time at the moment (aka I've been very very busy)