POSIX shell compatibility layer

[xiaq]

It would be really nice if we can source ~/.bashrc directly. That will make migration much easier.

[scateu]

Yet another Bourne Again Shell Again? YABASHA!

…

It would be really nice if we can source `~/.bashrc` directly. That will make migration much easier.

[eikenb]

+1 for POSIX compatibility.

Regarding the sourcing .bashrc... what would happen when you hit a non-POSIX bash extension?

[alexherbo2]

@eikenb We could refuse to source non-POSIX scripts or add a Bash layer (one POSIX and one Bash).

[krader1961]

I've been working on rejuvenating the ksh shell. The POSIX shell standard is a nightmare. Requests for POSIX support occurs regularly in the fish shell project. Once in a while that project implements a POSIX feature such as a && b || c as an alternative to the fish equivalent a; and b; or c. Which then leads people to say "that's great, now I can copy/paste random statements meant for a POSIX shell into fish." Umm, no, you can't other than trivial statements.

[alexherbo2]

@krader1961 It’s not to add POSIX features to the Elvish language, but to add a layer to compile sources from POSIX scripts.

[krader1961]

It’s not to add POSIX features to the Elvish language, but to add a layer to compile sources from POSIX scripts.

Thanks, but I understand that. The distinction is also irrelevant. Have you read the POSIX shell standard? Have you looked at the source for ksh, bash, or zsh? I've done both. In my experience people who ask for this feature greatly underestimate the complexity because they're thinking solely in terms of straightforward statements like export VAR=val. Handling those in a compatibility layer has value but is a tiny subset of the POSIX shell standard.

[xiaq]

Implementing a POSIX shell is not easy, but I also feel that its difficulty has been somewhat overestimated, probably due to the fact that the codebases of the traditional POSIX-ish shells are not the cleanest and often burdened with legacy stuff.

Implementing a bash clone is obviously even harder, but I am not convinced that it is totally unachievable. For instance, the Oil project has come quite close to a full bash clone recently.

Whether doing either of those is worth the effort is another question, hence the "maybe" label.

[ZHOUYue67]

Something similar to bass will be sufficient?

Bass is created to make it possible to use bash uilities in fish shell without any modification. It works by capturing what environment variables are modified by the utility of interest, and replay the changes in fish.

[progandy]

mrsh looks like an interesting POSIX shell that can be used as a library. That might be better than bass for pure POSIX compatibility and synchronizing environment variables has to be done either way.

There is also mvdan/sh.

[krader1961]

@xiaq I still feel this issue should be closed. If Elvish did have a mode that would allow it to source 99.9% of user's ~/.bashrc scripts (given that such scripts tend to use a small subset of bash features) should Elvish support running arbitrary bash scripts? With 100% fidelity to how bash would execute the script? What would it mean to run elvish path/to/script if Elvish supported both its syntax and POSIX syntax?

What does it mean to be POSIX compliant? Does it mean being compatible with bash, ksh, zsh, some other random shell that is more or less POSIX compliant? That is typically what people who ask for this feature expect. They are usually not asking for compatibility with the POSIX shell standard. But even if they are asking for compatibility with the lowest common denominator (i.e., the literal POSIX shell standard) how should that be implemented? Do we really want to read ~/.bashrc and ~/.kshrc and ~/.zshrc when an interactive elvish shell runs? Which of those, and many other widely used, POSIX "compatible" config scripts should elvish source at startup? What should elvish do when those scripts use features not in the POSIX standard?

[eikenb]

What does it mean to be POSIX compliant? [...] They are usually not asking for compatibility with the POSIX shell standard.

This is exactly what I was asking for. Most Linux systems come with dash as the default shell and people script with that in mind. Dash is more or less a strictly POSIX compliant shell, with no extensions. Due to this these scripts can be run using any of the big interactive shells (bash, zsh, etc) as they all support the POSIX feature subset. This makes working with the POSIX scripts easier and I prefer my interactive shell to be compatible with them.

I post to clarify what I was thinking when I chimed in support of this earlier.

[krader1961]

@eikenb, Thanks for the clarification but I'm curious how you envision this feature working. POSIX and elvish are fundamentally incompatible -- both syntax and semantics. We could add a POSIX emulation mode, invoked by elvish -posix or if the program name was sh, but it couldn't use any elvish features in that mode. Conversely, if normal elvish mode it couldn't use any POSIX features not already part of elvish (e.g., I/O redirection). It's not clear what the point would be since you can just use dash or a similar shell if you need POSIX support. Surely the limited development resources of the Elvish community are better invested making elvish better rather than reinventing the wheel.

P.S., I've noticed that people who use non traditional shells who ask for POSIX support almost always go on to ask for some bash/zsh/ksh feature they like, that isn't part of POSIX, to also be supported.

P.P.S., The whole reason I've been using fish, and now elvish, is because the POSIX standard is awful. It's only redeeming feature is that it is a well established standard that provides a lingua franca for writing scripts. Note that POSIX was based on the Bourne and Korn88 shell implementations. That is, it was meant to codify the behavior of those shells; warts and all. No sane software engineer today would create a shell with the same behavior (in particular how $IFS affects parsing).

[eikenb]

@krader1961 Thanks for responding.

In general I don't think POSIX compatibility is required and having it will probably never really matter. As is if I adopted it today (I've played with it, but still use Zsh as my daily driver) I wouldn't notice that much of a difference as I'd still be using dash as my scripting target (/bin/sh). So it not having POSIX wouldn't matter.

Really, having reflected on this more over the past couple years, I'm not sure I'll ever fully move to a non-traditional shell like this as it would hurt my work. Shell scripts are mostly useful due to 2 factors;

1. Convenience. I can easily switch between interactive use and scripting, which make shell scripts very convenient to test/write. Having a non-traditional shell would break that (as I'd still be writing POSIX shells due to point 2).
2. /bin/sh is a POSIX compatible shell everywhere.

I think POSIX compatibility could matter if any non-traditional shell ever became more than niche. Where it would be the default shell installed on a system. You'd want backwards compatibility to run scripts written in the only standard shell language. You could get this by installing dash or like, but then you'd have to have 2 shells by default.

I don't mean to be down on the project. Elvish is one of a few really interesting new shell environments and I'd really love to see one grow a significant community and be a real candidate for replacing current shells.

[krader1961]

I think POSIX compatibility could matter if any non-traditional shell ever became more than niche. Where it would be the default shell installed on a system. You'd want backwards compatibility to run scripts written in the only standard shell language.

I don't see the need to reinvent the wheel to support that situation. The system would still make /bin/sh refer to an extant POSIX compatible shell such as dash or bash. Interactive users would use /bin/elvish as would non-interactive scripts who want a sane shell rather than the mess that is the POSIX standard. The key here is "backward compatibility". That does not require that Elvish have a POSIX compatible mode. It only requires that the platform provide a POSIX compatible shell in addition to Elvish.

P.S., Note that most shells, including bash, are not fully POSIX compliant by default. In the case of bash you have to invoke it with the --posix flag. See https://www.gnu.org/software/bash/manual/html_node/Bash-POSIX-Mode.html. Most of those deviations from the POSIX standard are unlikely to be noticed by most users. But I think it important to note because far too many people think that bash is a POSIX shell when invoked without its --posix flag. Heck, even ksh93 isn't entirely POSIX compliant by default.

[xiaq]

Let's not overthink this. To start with, let's define some non-goals:

• Replace existing shells' role as a standalone script interpreter. It's perfectly fine to run sh some.sh or bash some.sh from Elvish.

• Re-implement Bash completely [1].

• Make the Elvish language backward compatible with POSIX syntax.

The benefit of a POSIX compatibility layer is solely for the interactive user, and it needs to be activated explicitly, e.g. by a special editor mode ("press Ctrl-P" to switch to POSIX compatibility mode) or a command (say posix 'posix shell code goes here')

There are 3 typical interactive use cases, in increasing order of difficulty to satisfy:

1. Run simple installation or compilation instructions that are just a series of simple commands.

   Elvish is already able to run some of such instructions. Such instructions sometimes use environment variables or interpolation within double quotes, which are not supported by Elvish.

2. Source scripts that set up environments, such as virtualenv.

   These scripts can use more advanced shell features, like function definition and pipelines. These scripts are usually written only using well understood shell features. The developers may not be explicitly programming against POSIX, but most of such features happen to part of the POSIX shell standard (not by accident; POSIX shell standard is mostly a description of well understood shell features).

3. Source .bashrc from Elvish.

   Being able to do this 100% certainly requires a complete re-implementation of Bash. However, the use of bash features in .bashrc is not even: I am sure we can find a lot of .bashrc files that consist of (1) a lot of alias (2) a lot of environment variable assignments and (3) a PS1 config.

   Instead of executing what's in .bashrc - since we have a parser - we can also instead convert it to a rc.elv automatically.

   Less used features can be left out, maybe showing a message linking to the Elvish documentation.

[1] Not that it's impossible. Andy Chu's oil shell nearly contains a perfect reimplementation of Bash (if it doesn't already have that). Re-implementing Bash in Go does come with its unique challenges; certain process semantics are not possible to implement in Go, and hard to emulate.

[krader1961]

Andy Chu's oil shell nearly contains a perfect reimplementation of Bash (if it doesn't already have that).

Is that true? I am familiar with that project. I've seen nothing to suggest it is 100% (or anything close to it) compatible with bash. Their AMA (ask me anything) just nine months ago contains comments from Andy such as

Another is the fact that bash hides a lot of stuff under shopt -- there are at least 20 options there. That tends to come up when sourcing .bashrc.

These aren't necessarily "limiting", but they're what I would call "time sinks". So far I've found "solutions" to most issues -- it's just a lot of work!

One thing I haven't addressed yet is a corner in backtick parsing -- that is surprisingly complicated. Tip: use $(echo hi) instead of echo hi! The former behaves exactly the same and is parsed more clearly with respect to quotes. The bash project also recommends this.

Note that the Korn shell, ksh, supposedly has a bash compatibility mode when invoked as bash. Yet even trivial things like the behavior of set and shopt are not fully bash compatible when the emulation is enabled.

I am sure we can find a lot of .bashrc files that consist of...

Yes, we can find such interactive configuration scripts. The question is what to do when loading such scripts which don't adhere to that very narrow set of features. What happens when a feature, such as an export command, that can be emulated is encountered yet the value assigned depends on a preceding statement that could not be emulated?

P.S., Why is ~.bashrc special? What about ~/.zshrc or ~/.kshrc?

[xiaq]

I've seen nothing to suggest it is 100% (or anything close to it) compatible with bash.

Hmm, my impression is wrong then - so fully emulating Bash is quite hard :)

The question is what to do when loading such scripts which don't adhere to that very narrow set of features.

If a command uses a feature that is not implemented by the compatibility layer, then that command is not executed - that's the only sensible thing.

What happens when a feature, such as an export command, that can be emulated is encountered yet the value assigned depends on a preceding statement that could not be emulated?

I don't have a good answer to that question.

I'd really like to shift the mental framework of discussion here: we can keep asking "can we do this 100% perfectly" and since the answer is "no", and conclude that there is 0 value in doing this.

But that is not really a good approach: instead of thinking in "all or nothing", take a step back, and think about how much value can be delivered by relatively little work. The two easier use cases in #205 (comment) can be a good way to start, for example.

Ideally, I'd quantify the return over investment as a number - for example, if we emulate environment variable, string interpolation and alias, how much of installation instructions found on GitHub can be supported, in terms of (# supported lines) / (# total lines)? How much of .bashrc found on GitHub? I don't know for sure, but I feel it's pretty high.

Why is ~.bashrc special?

Bash is the most used, so after the two easier use cases, thinking about bash emulation will yield higher return on investment.

[krader1961]

I'd really like to shift the mental framework of discussion here: we can keep asking "can we do this 100% perfectly" and since the answer is "no", and conclude that there is 0 value in doing this.

I agree. But that's why I've been trying to get people to realize that emulating bash (or even just POSIX), with even just 90% fidelity so it can handle most bash/POSIX scripts, isn't a realistic goal. It seems to me the goal should be to make it as painless as possible for a bash/zsh/ksh/fish user to migrate to elvish. And that is probably best served by a couple of things. One is to improve the "out of the box" experience. That is, when someone uses elvish for the first time a reasonable rc.elv should be created if one doesn't exist and the contents should have reasonable defaults such as automatically including statements such as

use epm
use readline-binding
use github.com/zzamboni/elvish-modules/alias

Etcetera. We should also provide a tool that migrates the things we can, such as export and alias statements from the user's .bashrc to the user's rc.elv file. What we shouldn't do is automatically source ~/.bashrc when elvish starts. Note that bash aliases are most closely map to elvish abbreviations. And some of these things might best be accomplished by a script the user runs after starting bash interactively. For example, rather than trying to parse alias commands it might be simpler and more robust to convert the content of $BASH_ALIASES from a user's interactive bash session.

P.S., Note that aliases in bash are by default only valid in interactive context. However, if the user does shopt expand_aliases then they are valid in non-interactive context. Which is a prime example of why trying to be bash compatible is a fools errand.

[krader1961]

What we shouldn't do is automatically source ~/.bashrc when elvish starts.

In addition to the points made in previous comments there is another reason not to do this. We want to make it easy for users of alternative shells, be it ksh, zsh, or fish, to switch to elvish. We certainly do not want to source ~/.bashrc, ~/.zshrc, ~/.kshrc, and ~/.config/fish/config.fish when elvish starts. What happens when a user uses two or more of those shells or simply has more than one of those startup scripts even if they only ever use one of those shells on a regular basis?

[xiaq]

@krader1961 Oh yeah, I definitely didn't mean that Elvish should source .bashrc at startup, but rather, it should be able to make a best-effort attempt of sourcing .bashrc when explicitly asked to.

And I agree with you that the focus should be helping new users migrate, rather than supporting using .bashrc as a substitute for rc.elv.

In any case, let me emphasize that the ability to source .bashrc (with best effort) is really the hardest, final goal in the 3 use cases I outlined in #205 (comment). Cases 1 and 2 are not only easier, but also improve the user experience of existing Elvish users.

[krader1961]

... rather, it should be able to make a best-effort attempt of sourcing .bashrc when explicitly asked to.

I've read this issue multiple times since it was opened. What does it mean to explicitly ask that ~/.bashrc be sourced? Does that mean simply starting an interactive elvish shell without any options related to this issue? Or is the idea that ~/.bashrc is only read if elvish -bashrc is executed? And why not also support reading the interactive startup files ~/.kshrc or ~/.zshrc? All three shells are more or less POSIX compatible and thus it should be possible to execute their content with roughly the same success rate as ~/.bashrc.

[LdBeth]

At least being about to parse output of /usr/libexec/path_helper on Mac

PATH="/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/opt/X11/bin:/opt/pkg/sbin:/opt/pkg/bin"; export PATH;

would be nice.

Otherwise, a mechanism to exec into a POSIX shell to handle the login profile and exec back at startup needs to be considered.
In that way nothing from POSIX shell needs to be recreated, since all Unix like system has /bin/sh (whether it is bash or dash or ksh) and can be reused.

[hanche]

At least being about to parse output of /usr/libexec/path_helper on Mac

Instead of running the program, I just duplicate its functionality myself:

set @paths = ~/.local/bin (
  put /etc/paths /etc/paths.d/*[nomatch-ok] |
  each {|p|
    cat $p |
    each {|d|
      if (!=s '' $d) { put $d }
    }
  }
)

– from my elvish setup, slightly edited for readability.

It could be useful to have a collection of minor snippets like this one for the most popular operating systems. To be useful, such a collection should be as minimal as possible, just addressing the issues that almost every user of each OS is likely to face. I tend to not use others' scripts myself, as I then find myself saddled with their idea of the kitchen sink and what it should do. But I steal ideas from other unabashedly. To each his own, of course.

[krader1961]

I just read http://www.oilshell.org/blog/2022/05/release-0.10.0.html. It's pretty clear that the Oil shell project also recognizes that the POSIX standard is a mess and is attempting to fix the POSIX warts by implementing non-POSIX features while, more or less, trying to remain POSIX compatible. Which, of course, still results in a non-POSIX compatible shell.

[tesujimath]

bash-env-elvish may help. I initially wrote a bash-env plugin for Nushell, then ported it for Elvish.

This is a Bash script bash-env-elvish and Elvish modules bash-env and virtualenv for:

• importing Bash environment into Elvish
• extracting Bash style shell variables from source files like /etc/os-release
• activating/deactivating Python virtualenv

Source files may be arbitrarily complex Bash, including conditionals, etc.

I'm pretty new to Elvish, so there may be a nicer way to integrate this. Suggestions welcome.