-
Notifications
You must be signed in to change notification settings - Fork 152
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RFC: are non-forking subshells still worth having? #480
Comments
👍 👍 👍 It would be interesting to see how hard it would be to disable this "feature". I'm especially curious how entwined it is with support for |
Since |
One thing to keep in mind is the behaviour of the |
In my opinion, non-forking subshells are another killer feature of korn shell and I for one would not like to see it go. Removing it may cause significant script breakage. Consider this code, which works for
Code like that has always worked for korn shell, as far as I can remember; even for ksh88. It has never worked for |
@dannyweldon: Actually, that is completely unrelated to non-forking subshells. The killer feature in question is that ksh88 and ksh93 do not execute the last element of a pipe in a subshell at all (forked or otherwise) and there is no reason why that would change if subshells are forked. As you've observed, zsh does the same thing (and always has), and it forks its subshells. And ksh88 does the same thing too, even though non-forked subshells were not introduced until ksh93. These days, bash does the same thing too if job control is disabled and the edit: I did a little experimenting and it looks like ksh93 already uses a forked subshell for the first element of the pipe in a construct like |
Thanks @McDutchie, that's good to know.
Yes, it still has to fork to run subprocesses, even if they are shell functions, I believe. |
All the advanced ("experimental") features of ksh93 (types, object oriented stuff, disciplines) more or less rely on that (or would be far less appealing without). The main advantage is for things like (it also has the |
Note that it's the other way round, forks used to be cheap but nowadays are very expensive. For most executed simple commands (think things like |
I'll admit to not understanding that. How do types, object oriented stuff, and disciplines depend on not forking subshells? How would forking subshells make them less appealing, or even affect them at all? Isn't a discipline function simply a special kind of shell function whose execution is triggered when a corresponding variable is read or changed? Shell functions are normally executed in the current environment.
That's interesting, I didn't know that. Could you elaborate on "a lot more"? Like what? |
The builtins in command substitutions simulate their output. In
I was mostly thinking of command substitution as mentioned above, which you'd heavily use if you were to use ksh93 object orienting facilities ( In another words, if you're going to do some object oriented programming with ksh93, since the way to return data is with command substitution, once command substitution forks (which would make it thousands of times slower), that feature becomes less appealing. |
That's a bit of an exageration.
More like 50 times as slow (here on Linux amd64). Note that the subshell simulation has quite a significant overhead:
So where possible, you'd still want to avoid subshells even if they don't fork. For completeness and to show that forking is indeed expensive compared to loading and initialising an external executable:
|
I was wrong there. No it doesn't, not in that case. |
Yes, the
On FreeBSD, I find:
So it must do some form of subshell emulation (here at least make the variables local to the subshell). I find that it still forks for |
Thus limiting or even obliterating their performance benefit vs. forked subshells in real-world usage. Yes, I'd noticed that. (I emailed you an example of a performance comparison with dash.)
But you're doing a fork + exec instead of just a fork, so you're not proving anything regarding the relative cost of fork() vs. exec'ing and initialising an external command. Also, bash is the slowest shell around by a large margin, so using it to prove your performance point is a bit unfair. :-) A better way of measuring that would be to using a builtin, and then an external command, on a fast shell that always forks its command substitutions -- say, dash. What do you get for these? On my laptop with macOS, I get:
So on MacOS, with a fork() that is as slow as an arthritic dog, launching an external command takes 4 times as long as a simple command substitution fork(). On Linux (Intel Xeon 4 core 3.3Ghz), I get:
That's a factor ten difference. So yeah, I maintain my position that forks are cheap. :-) edit: At the risk of adding noise, I just can't resist adding this. This is FreeBSD, with bash and dash from ports, on a VirtualBox VM, on the same Mac laptop that got the dog-slow results above...
A factor three difference, so still relatively cheap forks. But, holy blazing speed, Batman! What is the secret of the FreeBSD people? |
I did, that was:
Showing that forks make cmdsubsts with builtins prohibitive and that the forking is significative compared to fork+exec. You have a point about bash being the slowest shells of all.
10s/50000 = 0.2ms to store "x" into a variable is not "cheap". As discussed privately, ksh93 advertised goal was to be a programming language playing in the same field as perl and python. The equivalent of:
in ksh93 is
(yes, I know ksh93 can also pass variables by reference which is another option to return value). |
Here's a benchmark where ksh is much faster than other shells. It uses the slow recursive algorithm for the Fibonacci sequence; I adapted it from Ruby's sample/fib.rb. The syntax
|
BTW, It's helpful to put this at the end of all the above
eg.
|
Optimizing shell code by not calling fork(2) is no longer needed in the modern world. |
I'll have to admit that
|
I would also point out that this can be done much faster still with nothing but pure POSIX shell and utilities, and with only one fork:
That takes 0.014s on my system. The ksh variant with I think this demonstrates mainly the need to use the right tool for the job. |
This is the kind of issue where I would have asked for advice from ksh's original creators. Unfortunately I have not managed to get response from any of them. Non-forking subshells have caused inconsistent behavior on different occasions, but removing them is going to be a major change. At this time I can only speculate how it is going to affect existing scripts. This is something that I would not target for the next release, but definitely we should consider afterwards. |
Stéphane and others have been somewhat convincing in pointing out the advantages of fast command substitution subshells, altough I still think those arguments are a bit far-fetched and have limited relevance to real-world usage. I'll concede, though, that non-forking command substitutions have a real advantage for some use cases. However, the question is, is that worth the bugs? I'm particularly thinking of #73. Unsetting or redefining a function within a subshell is silently ignored. Silently executing the wrong code, in what universe is that acceptable? Frankly, I am gobsmacked that this bug has been present for the last 25 years and was never considered a release blocker. There are all sorts of possible reasons to unset or redefine functions within subshells. For example, it is a quite conceivable to wrap some command in a function by the same name, e.g. (Yes, I know Another possible use case is temporarily changing the behaviour of a program by replacing one of its functions with another by the same name within a subshell. That fails as well. So the question the developers need to answer is not: is there a legitimate use case for non-forking subshells? but: is it worth the bugs? The answer depends on whether those bugs (especially #73) can be solved. If you can't make it work properly for the next release, please at least make the shell print an error and exit when attempting to redefine or unset a function within a subshell. That would still be much preferable to executing the wrong code. |
I need to take some time to read the earlier comments thoughtfully but my knee-jerk reaction is a resounding "no". Even if we fixed the problems with the existing non-fork subshell behavior it is not at all obvious the added code complexity, and potential source of new bugs, is worth the seemingly insignificant performance benefits. Correct behavior trumps other considerations such as performance. And while that is not always true it is true for a program like ksh. |
Optimizations never worth bugs. There is nothing more non-efficient than a software that does not work. Unfortunately there are many places where the original developers have chosen optimizations over consistent behavior (for e.g. #468). If the code related to non-forking subshells can be removed without creating any side effects, I am fine with removing it. |
It should be kept in mind that any such side effect is another bug, either in ksh or in the script using ksh. Because, by definition (see POSIX definition quoted at the top of the thread), a subshell is a completely separate execution environment, initialised as a verbatim copy of the current shell, including non-exported variables, functions, etc., with the sole exception of traps which are reset in subshells. Thus, any script that relies on the broken or incomplete separation of a subshell environment is relying on a bug, and is thereby itself broken. |
2018-04-25 13:03:13 +0000, Martijn Dekker:
[...]
Thus, any script that relies on the broken or incomplete
separation of a subshell environment is relying on a bug, and
is thereby itself broken.
[...]
Some other things that are impacted by a non-forking subshell:
$ ksh -c '(echo foo); echo done >&2' | :
$ zsh -c '(echo foo); echo done >&2' | :
done
For ksh, it's the main process that got the SIGPIPE because the
"echo" (which is a builtin, yet another optimisation that
affects the behaviour here, though that particular optimisation
is done by most shells) was not done in a child process.
$ mksh -c 'printf foo; echo "$? done" >&2' | :
141 done
$ dash -c 'printf foo; echo "$? done" >&2' | :
$ zsh -c 'printf foo; echo "$? done" >&2' | :
$ bash -c 'printf foo; echo "$? done" >&2' | :
(in the case of printf, mksh is the one that behaves as expected
as it doesn't do the optimisation (doesn't have printf built
in)).
Also on limits:
$ ksh -c 'ulimit -t 1; (while :; do :; done); echo done'
zsh: killed ksh -c 'ulimit -t 1; (while :; do :; done); echo done'
$ zsh -c 'ulimit -t 1; (while :; do :; done); echo done'
done
Though, if I move the ulimit inside the subshell, that's better:
$ ksh -c 'echo $$; (ulimit -t 1; while :; do :; done); echo done'
3581
ksh: 3582: Killed
done
There was a fork there. So it looks like it's a cheap way to
force a fork (ulimit -t unlimited also forces a fork AFAICT).
…--
Stephane
|
Thank you for discovering a way to force a fork! So far I only discovered that including a redirection in a command substitution forces that command substitution to be forked in certain cases. edit: Hmm... bltins/ulimit.c contains this interesting line:
Could |
So now that it's known that there is a simple way to force a fork at any point during the execution of a subshell, my opinion on this issue has changed. I now think you should probably keep the non-forking subshells, but be much more conservative about forcing a fork using Any time things like functions, traps, or aliases are set, unset or changed in a subshell, or the state of that subshell is drastically altered in some other way, you should fork. This way, things can hopefully be made robust while keeping the vast majority of the performance advantage. |
BTW. K.R.E. had ideas about vfork(2) with a kick-the-parent syscall option. It might be like an intermediate solution. |
@krytarowski, Can you provide more detail? I can't grok your comment. We also, recently, dropped support for @McDutchie, I would prefer to simplify the code rather than complicate it with more ad-hoc special cases. I haven't yet found the time to carefully consider the performance arguments for a non-forking subshell. But I'll bet that the performance benefits for the 98'th percentile of ksh scripts are so small that the added complexity, and potential bugs, can't be justified. Yes, there are undoubtedly ksh scripts that cache so much data they benefit from the optimization. But I don't see how catering to those atypical scripts at the expense of all the other use cases (in particular the interactive use case) can be justified. Too many people seem to believe that just because an optimization is theoretically possible it is worth implementing regardless of the cost. Costs that are both immediate and long term. |
I thought I would try it as a math function to see if it was faster, but came across two issues. As a single liner, I had to add a newline after the math function opening brace:
Otherwise I get a syntax error:
Also, if I go any higher, say fib(9), I get a recursion error, such as:
which is odd considering that the previous non-math versions work just fine. |
FWIW, I have carefully re-read this issue and am still unconvinced that the current behavior that tries to avoid forking subshells is worth the complexity and bugs. As I said earlier even if the current bugs with respect to this feature were fixed the complexity all but guarantees new bugs would occur in the future as the code evolves. The POSIX 1003 standard for shells like Yes, given unlimited resources the optimization can be made to work. AT&T didn't make it work. And I don't see anyone else stepping up to commit those resources. Too, if you're using |
There is a bug in |
@cococlyde There are some comments in favor of and against non-forking subshells in this thread. I would like to hear your thoughts on it. Key questions are :
|
I hate to add another comment to an issue that is already way too long but a bug I documented in issue #1074 needs to be duplicated here rather than just as a link. Precisely because it is a crystal clear example why this sort of optimization is problematic. This statement ast/src/cmd/ksh93/sh/subshell.c Line 582 in e756d5b
is broken as the condition can never be true since There are so many long standing (i.e., more than a decade old) bugs involving this optimization that I find it difficult to justify the performance improvement it provides. Notwithstanding the fact I have spent a lot of my professional life dealing with performance problems above and beyond what any competent software engineer would be concerned about. As I and others have stated in previous comments correct behavior is far more important than better performance. Especially since something on the order of 90% of ksh scripts are not sensitive to the performance of Yes, I pulled the 90% figure in the previous paragraph out of my ass. I think the value is far higher. I will eat a printed version of this issue if someone can show that at least 10% of a random sampling of ksh scripts would be slowed down by removing the no-fork subshell optimization to an extent that anyone would complain. This is ultimately about whether ksh can be made a general purpose language despite its origins as a POSIX shell (e.g., Bourne shell) language. I think the answer is "no" given the idiosyncratic behavior of shells that adhere to that standard. |
(Please note: below I use the POSIX definition of "subshell", and not any other confused and misleading notion of "subshell" used in various shell manuals and tutorials. To quote: "A subshell environment shall be created as a duplicate of the shell environment, except that signal traps that are not being ignored shall be set to the default action. Changes made to the subshell environment shall not affect the shell environment. Command substitution, commands that are grouped with parentheses, and asynchronous lists shall be executed in a subshell environment. Additionally, each command of a multi-command pipeline is in a subshell environment; as an extension, however, any or all commands in a pipeline may be executed in the current environment.")
All shells except ksh93 fork off a new process to create a subshell environment. ksh93's non-forking implementation of subshell environments has always been rife with problems. Even David Korn has never managed to create a cleanly separate execution environment without forking a new process.
One egregious issue is #73, which can silently cause the execution of the wrong code. Every ksh93 version has that bug. Other issues include #108 and #416.
#478 is a recent regression related to non-forking subshells. In that issue, @siteshwar inserted a telling comment:
I cannot but agree. I, for one, am unable to understand the ksh93 source code. It is very convoluted and there are hardly any comments. I have to say I have great respect for @krader1961 and @siteshwar for having taken this on.
Now, as far as I know, there are two original reasons why ksh93 implemented subshells without forking a process:
I don't think either of these still apply. Forks have been pretty cheap for a long time now. Even on Windows it's much better than it used to be. There are shells such as dash and FreeBSD sh that are faster than ksh93 while still forking their subshells. And good luck finding a system without fork() younger than 20 years. I assume you're not trying to be compatible with FreeDOS. :)
And of course ksh93 is already perfectly capable of forking off a subshell in a new process: by necessity it does so for a background job. In certain cases it also forks off a command substitution containing a redirection (see my comment on #478). And of course, forked subshells are always cleanly separate because the separation is enforced by the kernel.
Non-forking subshells were an interesting experiment conceived at AT&T that may have served a purpose for a time, but never worked quite right. Now that operating systems have matured, it seems to me that the need for them has gone away.
So here's a radical idea. Why not simply disable and then remove the whole non-forking subshell experiment and fork all subshells like other shells do? I think you'd see a lot of bugs simply disappear, and you'd be removing a maintenance nightmare as well.
What do you think?
The text was updated successfully, but these errors were encountered: