special-casing PATH/MANPATH/CDPATH is weird; we need a more general solution like zsh "tied" variables

[xiaq]

In expand.h:

/** Character for separating two array elements. We use 30, i.e. the ascii record separator since that seems logical. */
#define ARRAY_SEP 0x1e

/** String containing the character for separating two array elements */
#define ARRAY_SEP_STR L"\x1e"

This results in:

xiaq@blackie ~> set a (printf 'a\x1eb')
xiaq@blackie ~> count $a
2
xiaq@blackie ~> set a (printf 'a\x1fb')
xiaq@blackie ~> count $a
1

It's clear that the char \x1e is treated specially as element delimiter.

[ridiculousfish]

Is the concern that there's no way to represent \x1e? Or are you thinking about architectural improvements?

I think the array separator is mainly used in persisting arrays in places that only take strings, like universal variables or environment variables.

[xiaq]

在 2012-12-12 上午2:09，"ridiculousfish" notifications@github.com写道：

Is the concern that there's no way to represent \x1e? Or are you thinking
about architectural improvements?

I'd say I have both conerns in mind. For the former, just think about a
filename with embeded \x1e. POSIX says anything but \0 is allowed in
filenames, so it's perfectly possible. Using \0 as array delimiters may be
a slightly better choice, but that leads to the second concern - it's
fragile and plainly wrong.

I think the array separator is mainly used in persisting arrays in places
that only take strings, like universal variables or environment variables.

If persisting means "serialized", no. Arrays are always stored in
\x1e-delimited form. Exported environment arrays are joined by ":". They
are used in universal variables though - which IMHO should be implemented
with proper escaping instead.

—
Reply to this email directly or view it on GitHub.

[JanKanis]

How about using a private use area character as separator? Fish already uses some of those in some cases.

[xiaq]

@JanKanis That's no better; it's perfectly possible in filenames (considering filesystems using non-utf8, native encodings) and other strings.

I would say that among others, "\0" is a slightly better choice among others, since that's the only character UNIX forbids in filenames, but it still has the bad smell to me. Also, we are already doing a lot of splitting and assembling of arrays, I would expect implementing true arrays to result in both better architecture and less code.

[JanKanis]

Fish already handles private use characters and invalid bytes when it encodes external strings to wchars. These special values are encoded byte by byte into a specific set of private use chars that fish also decodes again on output, so in principle using another private use char could work. However I agree using true arrays is much better. There is one complication in that communication between fish and fishd happens over a socket using utf8 strings, and there fish uses (I think) the escape sequence "\x1e" (rather than an 0x1e byte) to separate array items. But that could probably be solved by using e.g. a private unused escape sequence.

[anordal]

I share xiaq's concerns, but (for the practical ones) I find the implicit splitting on \n way more offensive:

a@raspeball ~> count (printf 'a\x1eb')
1
a@raspeball ~> count (printf 'a\nb')
2

The (newline delimited) array interpretation of subprocess output should be explicit & optional!

But that has probably nothing to do with the underlying storage of arrays…

[szhu]

@xiaq: What's wrong with using \0? xargs seems to use it as its "reliable" option.

[zanchey]

You can't use \0 in environment variables, so it would make it hard to export arrays to child shells.

[krader1961]

I'm migrating from zsh where I use this sequence to define the $LESS env var in a sane fashion by leveraging its "tied" variables feature:

typeset -xT LESS less ' '
less=(
    --HILITE-UNREAD
    --LONG-PROMPT
)

I've omitted the complete list of options for brevity. In zsh that results in the $LESS env var being a space separated list of the options in the $less array. The equivalent in fish results in the elements being separated by the record separator (\x1e) character. Despite the documentation saying that the elements of the array will be separated by spaces (modulo the special arrays such as PATH). I have to explicitly do an assignment that interpolates the values into a single string to get the expected result:

set -x LESS --HILITE-UNREAD \
    --LONG-PROMPT
set LESS "$LESS"

At the moment I don't really care if \x1e is used internally for serializing arrays rather than \x00. I do care that exported arrays have their elements separated by \x1e. That's just broken, wrong, fubar. Pick your adjective. It's also inconsistent with the aforementioned workaround and documented behavior. This issue should be tagged as a bug IMHO.

P.S., Nowhere in the documentation is the use of the record separator character (\x1e) mentioned. Which is another problem.

[ridiculousfish]

@krader1961 Thanks for sharing this. There's no standard Unix convention for list-like environment variables - some are colon delimited, others are space delimited. fish uses \x1e so it can distinguish its own arrays.

Can you please point us at the erroneous documentation?

How do you think arrays be exported - colons, spaces, newlines, something else? Should fish tokenize environment variables on this character as well?

It looks like less expects space-delimited arguments. Probably the simplest workaround is set -x LESS '--HILITE-UNREAD --LONG-PROMPT', etc.

[krader1961]

There's no standard for list like environment variables because by definition they're an arbitrary sequence of bytes composed of a key and value separated by an equal sign and terminated by a null byte. They don't even have to be printable characters. The only widely accepted convention for a higher level of abstraction is the one established by the execlp() function for the PATH env var.

The documentation is erroneous in as far as it makes no mention of using \x1E, \036, 30, or the "record separator" character to separate elements of an array when exporting a var with more than one element. The documentation does state that

..., and array variables will be concatenated using the space character.

That's from the section "Variable expansion" in http://fishshell.com/docs/current/index.html. It's reasonable to infer that statement also applies to exported vars that are not special-cased as documented in the "Arrays" and "Special variables" sections of that same document.

It's my feeling that fish should not automatically tokenize env vars into a list outside of the colon-delimited special case vars such as PATH. There should, however, be a robust means by which a user can tokenize a var into an array on an arbitrary character.

Absent a mechanism for configuring the character to be used on a var by var basis (ala the zsh "typeset -T" command) a space should be used when concatenating the elements of the array (again, excluding the colon separated special-case vars). Obviously this does not apply to private data stores such as the storage of universal variables.

Lastly, I couldn't find any uses in the standard fish functions where an env var is used to pass an array containing more than one element to another function or script. Such use cases may exist but it should require the scripts to explicitly cooperate in the serialization/deserialization of the data rather than rely on fish to implicitly reconstruct arrays from vars whose strings contain the record separator character.

[ridiculousfish]

Thanks for your thoughtful response. The section you quoted about concatenation using space is specifically for double-quoted strings. We should add some discussion of what happens with exported arrays.

Users can tokenize strings with e.g. set SOMEPATH (string split : $SOMEPATH).

The downside of space-concatenating exported variables is that they get changed when fish is run recursively. Today this works:

> set -x list 1 2 3
> count $list
3
> fish -c 'count $list'
3

But if we exported with spaces, this would show 1 for the recursive call. As you say we don't rely on this, but it's nice from a consistency standpoint.

[faho]

Thanks for your thoughtful response.

I'll have to second that! Always nice to have a fresh perspective on things.

For those new to this discussion, I think I'll have to bring in some of the things that are related to this.

What comes to mind immediately is the listify whitelist, which shows up in issues like #2090.

This means that for $PATH, $CDPATH and $MANPATH, they'll appear as lists/arrays to fish, but when exported, will be joined with ":" again. Then a fish-inside-a-fish will split them again. This operates on colons, not \x1e. From my understanding of the code it appears to do it on every colon, with no chance for escaping, so it might break on $PATH entries with a colon inside them - which UNIX allows inside filepaths, though it seems broken for $PATH at least. This scheme is also used for e.g. PYTHONPATH and GOPATH.

I'd love to have something slightly more explicit for splitting environment variables than the implicit always-split-on-\x1e-except-for-these-three-split-them-on-colon, because this is actually two different schemes in one and exporting a list currently will always confuse everything but fish.

My preferred solution would be a function like splitenv var1 var2 var3:

function splitenv --no-scope-shadowing
    set -e IFS # string split doesn't have superpowers, so unset IFS lest we split on both : and \n
    for arg in $argv
        set -q $arg; and set $arg (string split ":" $$arg)
    end
end

(If string split had superpowers, this would be a bit simpler)

All lists would then be joined-with-colons when exported, so a user can explicitly unjoin them with splitenv (though I'm not dead-set on a helper function, I do believe making this trivial is a good thing to do). For backwards-compatibility, splitenv PATH CDPATH MANPATH would be executed on startup. If a user wishes to export it differently, string join is available.

All of this means that we no longer need \x1e, we have a scheme that at least has a fighting chance of being understood by other programs, but the (rather exotic IMHO) fish-inside-fish now becomes fish -c 'splitenv list; count $list'.

The problem is of course that, as mentioned, the usual colon-separated-list scheme has no way of escaping a colon, and if we wanted to add one, string split doesn't have a "--unescaped" option to only split on unescaped separators.

Am I making any sense?