Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to read bytes from a pipe, up to a certain character #831

Closed
href opened this issue Jun 2, 2019 · 8 comments
Closed

How to read bytes from a pipe, up to a certain character #831

href opened this issue Jun 2, 2019 · 8 comments

Comments

@href
Copy link
Contributor

@href href commented Jun 2, 2019

I've wrote a package to integrate gitstatus into Elvish, because I wanted real quick git status indicators and this seemed like a great opportunity to get familiar with the language.

I'm quite happy how it turned out and I find the resulting code to be surprisingly short and readable.

There's one thing I could not do without resorting to sh though: I can't seem to find a way to read from a pipe without doing this:

response = (sh -c 'read -rd $''\x1e'' && echo $REPLY' < $state[stdout])

See https://github.com/href/elvish-gitstatus/blob/e73aab15a5104ba357049ed8ba4a0f81fc06d229/gitstatus.elv#L148-L160

I know there's an open issue about implementing read: #741

But I was wondering if there's maybe something that I can do in the meantime to avoid creating a subprocess just to read from a pipe until I find a certain byte. Anyone knows a way?


As an aside, Elvish is such a pleasure to script. Kudos to all the work that was put into it!

@zzamboni
Copy link
Contributor

@zzamboni zzamboni commented Jun 2, 2019

@href First of all, welcome to Elvish!

For reference, there is already a git.elv module written by @muesli, which is used already by several prompt themes, including my own chain theme. This module uses git instead of gitstatus, but I think it would be nice to make the APIs compatible if possible so we can use them exchangeably :)

To answer your question, I don't think character-based reading can be done at the moment in Elvish, so your bash-based workaround seems reasonable for now. Maybe @xiaq can provide some other ideas.

(sidenote: I find the 0x30/31-based separation used by gitstatusd a bit awkward in general. I think a line-based interface would be much easier to use).

@href
Copy link
Contributor Author

@href href commented Jun 2, 2019

Cheers!

I think it's reasonable to at least provide a translation layer compatible with the existing git.elv module. Though I'll probably only go as far as implementing git:status by translating the results of gitstatus:query: href/elvish-gitstatus#2

I'm a bit hesitant to implement functions which throw away results of the status, as the whole point of gitstatus is performance and I think it makes sense to nudge people towards getting the data as a variable. If I include things like git:branch_name, then people might be encouraged to use these functions and lose out on the performance.

I also need to keep a separate set of return values in any case, as gitstatus is still in development and new values get added at times which might not be part of git:status. So I certainly want to stay true to gitstatus there.

And I do agree that 0x30/31 is a bit awkward, but that's not my call to make and from what I can tell this API is pretty much going to be in-flux.

@zzamboni
Copy link
Contributor

@zzamboni zzamboni commented Jun 2, 2019

Makes sense. I didn't know about gitstatus before, looks interesting for performance-sensitive applications such as shell prompts. Even with the existing git package, which is quite optimized, and with Elvish background handling of prompt commands, I still sometimes notices a lag in large repositories. I might give a try to using your gitstatus module for my chain prompt, see how it behaves 😄

@xiaq
Copy link
Member

@xiaq xiaq commented Jul 20, 2019

Try splitting the string and getting the first element?

splits , a,b,c | take 1

@xiaq xiaq added the t:question label Jul 20, 2019
@href
Copy link
Contributor Author

@href href commented Jul 22, 2019

The problem is not the splitting of the string, but getting it in the first place. The string has to be read from a long-running pipe. The record has a variable length, delimited with a special character (not a newline).

So I need a way to read characters from a pipe until I find a certain character.

You can see the problem if you run the following go program:

package main

import "os"

func main() {
	for {
		os.Stdout.WriteString("foo,bar,baz")
	}
}

I'd like to get 'foo' out of this program with elvish. How do I do it? When I run go run main.go | head, the command runs forever, because head waits for a newline. Same goes for go run main.go | splits , (all).

@xiaq
Copy link
Member

@xiaq xiaq commented Dec 30, 2019

I am considering introducing a read-upto command:

read-upto $last

The command reads from the bytes input, one byte at a time. The argument $last may be one of:

  • A string with a single byte, in which case it specifies the byte to stop reading after;
  • A function, in which case read-upto passes each byte as a string to the predicate, and stops after the predicate outputs a false value.

There is something subtle about read-upto: its output will include the first byte that is equal to $last (if $last is a string) or fails the $last predicate (if $last is a function). This is a consequence of how pipelines work.

@xiaq
Copy link
Member

@xiaq xiaq commented Dec 30, 2019

The first variant is now implemented and satisfies the need of this issue. Enjoy!

I have filed #885 to track the implementation of the predicate variant.

@zzamboni
Copy link
Contributor

@zzamboni zzamboni commented Jan 2, 2020

@xiaq thanks!

@href I have submitted href/elvish-gitstatus#4 to use read-upto in your elvish-gitstatusd package.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.