New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strict mode options #805

Closed
timbertson opened this Issue May 22, 2013 · 45 comments

Comments

Projects
None yet
@timbertson
Contributor

timbertson commented May 22, 2013

When writing scripts in bash, I always use some combination of:

  • set -e (fail if any command fails)
  • set -u (fail if any undefined variable is referenced)
  • set -x (print each command before it's executed)
  • set -o pipefail (a failed command in a pipeline will cause the parent command to fail)

Additionally, it would be nice to have an option like pipefail that applies to subshells (i.e causes echo (false) to return nonzero status).

It doesn't seem like any of these are available in fish, which doesn't usually matter for interactive use but kills it as a scripting language for me (which is a shame, because fish's sensible variable-expansion would make it much harder to accidentally get quoting wrong).

Any chance of adding similar features?

@dag

This comment has been minimized.

Show comment
Hide comment
@dag

dag May 22, 2013

Contributor

We loathe options and configuration in fish. 😉

I'm inclined to suggest that strict mode should be the default and the only option, and we should instead provide escape hatches for more lenient execution. One problem with this though is that if ; is ; and then ; or is ; and or which doesn't really work, so we'd have to special case that or redesign and/or.

set -x could perhaps be an argument for the fish executable.

Contributor

dag commented May 22, 2013

We loathe options and configuration in fish. 😉

I'm inclined to suggest that strict mode should be the default and the only option, and we should instead provide escape hatches for more lenient execution. One problem with this though is that if ; is ; and then ; or is ; and or which doesn't really work, so we'd have to special case that or redesign and/or.

set -x could perhaps be an argument for the fish executable.

@dag

This comment has been minimized.

Show comment
Hide comment
@dag

dag May 22, 2013

Contributor

Oh and it would mean making ; and commandline -f execute behave differently, which may or may not be a problem.

Contributor

dag commented May 22, 2013

Oh and it would mean making ; and commandline -f execute behave differently, which may or may not be a problem.

@timbertson

This comment has been minimized.

Show comment
Hide comment
@timbertson

timbertson May 22, 2013

Contributor

I'd love strict mode to be the default.

-x could also be an option to begin, if it's possible for begin to take arguments. I often have individual sections of script with -x disabled if they produce a lot of useless logging, so it'd be nice to control this at a finer level than at process startup.

Yes, and/or is a problem (well, actually only or is). Without special casing, I'm not sure there's much we can do. It does provide one argument for making them part of the syntax though as opposed to just builtins.

Also, if -u is a default then there should be convenience method for defaulting variables, i.e some more readable and terse version of if not set -q y; set y "default"; end.

Contributor

timbertson commented May 22, 2013

I'd love strict mode to be the default.

-x could also be an option to begin, if it's possible for begin to take arguments. I often have individual sections of script with -x disabled if they produce a lot of useless logging, so it'd be nice to control this at a finer level than at process startup.

Yes, and/or is a problem (well, actually only or is). Without special casing, I'm not sure there's much we can do. It does provide one argument for making them part of the syntax though as opposed to just builtins.

Also, if -u is a default then there should be convenience method for defaulting variables, i.e some more readable and terse version of if not set -q y; set y "default"; end.

@dag

This comment has been minimized.

Show comment
Hide comment
@dag

dag May 22, 2013

Contributor
set -q y; or set y default

But maybe a set -d is warranted.

I do think it's a bit odd that wildcards error when not matched but undefined variables don't. I guess another option would be the inverse: make non-matching wildcards expand to empty lists. That would probably be even less convenient and even more confusing than the current behavior, though, since now if you do e.g. wget some/url/*.txt you end up running wget with no arguments and it prints usage hints, instead of the more helpful fish error for unmatched wildcard.

Contributor

dag commented May 22, 2013

set -q y; or set y default

But maybe a set -d is warranted.

I do think it's a bit odd that wildcards error when not matched but undefined variables don't. I guess another option would be the inverse: make non-matching wildcards expand to empty lists. That would probably be even less convenient and even more confusing than the current behavior, though, since now if you do e.g. wget some/url/*.txt you end up running wget with no arguments and it prints usage hints, instead of the more helpful fish error for unmatched wildcard.

@timbertson

This comment has been minimized.

Show comment
Hide comment
@timbertson

timbertson May 22, 2013

Contributor

Yep, I'd definitely rather more things be strict than less (wildcards are a perfect example, being lenient about them only causes errors later, which can be dangerous).

set -d sounds like a good idea to me.

Contributor

timbertson commented May 22, 2013

Yep, I'd definitely rather more things be strict than less (wildcards are a perfect example, being lenient about them only causes errors later, which can be dangerous).

set -d sounds like a good idea to me.

@ridiculousfish

This comment has been minimized.

Show comment
Hide comment
@ridiculousfish

ridiculousfish Jun 9, 2013

Member

If we have a strict mode, then we will need to ensure that all of fish's provided functions work correctly in strict mode.

fail if any undefined variable is referenced

This seems like a weird thing to give the user a choice about. We should just figure out what the right behavior is and implement that.

fail if any command fails

fish relies more heavily on functions than most shells, and most functions cannot easily be made compatible with this. So this option ought to be lexically scoped, like local variables, so it applies to only a block of code, and not function calls.

A more idiomatic way to do this in fish would be an event handler, something like this:

function die --on-event command-failed
    exit
end

but I'm not sure how this could be lexically scoped.

print each command before it's executed

This seems like it would be better suited as an option to the fish executable, like the existing --no-execute.

a failed command in a pipeline will cause the parent command to fail

I think it would be error-prone and not fishy to make fundamental pipeline behavior depend on distant configuration options. Ideally you would want to specify this the pipeline itself.

Three ideas that I think are pretty good:

  1. Make and work in pipelines (currently it is disallowed). Then we could write for example:

    grep root /etc/passwd | and cut -f5 -d:
    

    which is a beautiful and intuitive syntax.

    We would need to decide on the behavior of and in a pipeline. Would it be true to itself, and not execute the cut command at all? Or would it still execute the command, and just affect the exit status?

  2. Add a new variant of the pipe which accumulates $status, say, |&

    grep root /etc/passwd |& cut -f5 -d:
    

    This seems like a nice syntax as well.

  3. Just make pipefail the default any only behavior of pipes. Any reason why it should not be?

A few other ideas which I think are bad:

  1. Add a separate variable $pipestatus which tracks $status, except it's nonzero if any command in the pipeline failed. It's unclear how this would be reflected into $status for the purpose of if statements, etc.
  2. Add a separate variable $failcount which tracks how many commands have failed. You would use it like so: set failcount 0; foo | bar | baz; if test $failcount -ge 0 ...
  3. Add a builtin pipe which can head a pipeline and configure it in various ways. pipe --overall-status | foo | bar.
Member

ridiculousfish commented Jun 9, 2013

If we have a strict mode, then we will need to ensure that all of fish's provided functions work correctly in strict mode.

fail if any undefined variable is referenced

This seems like a weird thing to give the user a choice about. We should just figure out what the right behavior is and implement that.

fail if any command fails

fish relies more heavily on functions than most shells, and most functions cannot easily be made compatible with this. So this option ought to be lexically scoped, like local variables, so it applies to only a block of code, and not function calls.

A more idiomatic way to do this in fish would be an event handler, something like this:

function die --on-event command-failed
    exit
end

but I'm not sure how this could be lexically scoped.

print each command before it's executed

This seems like it would be better suited as an option to the fish executable, like the existing --no-execute.

a failed command in a pipeline will cause the parent command to fail

I think it would be error-prone and not fishy to make fundamental pipeline behavior depend on distant configuration options. Ideally you would want to specify this the pipeline itself.

Three ideas that I think are pretty good:

  1. Make and work in pipelines (currently it is disallowed). Then we could write for example:

    grep root /etc/passwd | and cut -f5 -d:
    

    which is a beautiful and intuitive syntax.

    We would need to decide on the behavior of and in a pipeline. Would it be true to itself, and not execute the cut command at all? Or would it still execute the command, and just affect the exit status?

  2. Add a new variant of the pipe which accumulates $status, say, |&

    grep root /etc/passwd |& cut -f5 -d:
    

    This seems like a nice syntax as well.

  3. Just make pipefail the default any only behavior of pipes. Any reason why it should not be?

A few other ideas which I think are bad:

  1. Add a separate variable $pipestatus which tracks $status, except it's nonzero if any command in the pipeline failed. It's unclear how this would be reflected into $status for the purpose of if statements, etc.
  2. Add a separate variable $failcount which tracks how many commands have failed. You would use it like so: set failcount 0; foo | bar | baz; if test $failcount -ge 0 ...
  3. Add a builtin pipe which can head a pipeline and configure it in various ways. pipe --overall-status | foo | bar.
@timbertson

This comment has been minimized.

Show comment
Hide comment
@timbertson

timbertson Jun 10, 2013

Contributor

fail if any undefined variable is referenced

With something like set -d, I think this makes sense to be the default (and only) behaviour.

fail if any command fails

fish relies more heavily on functions than most shells, and most functions cannot easily be made compatible with this

I don't understand why - I would have though that to implement this, a function failure would be treated like a command failure. And a failed command call made in a function would cause the function to also fail.

Regarding:

grep root /etc/passwd | and cut -f5 -d:

I don't think you could prevent cut from running if grep failed - something like tail -f would run indefinitely, so you'd need to start cut sometime. So it'd be more of a propagation of status codes rather than changing what gets run.

As an aside, I think and should be the default behvaiour (especially if "fail if any command fails" is the default - otherwise it's inconsistent).

Is there a way to preemtively allow a command to fail? i.e instead of modifying the syntax to allow a failed command if it's followed by an or, perhaps just introduce a builtin ("try"?) that runs the given command and returns its exit code but prevents the fail-on-error behaviour, Some examples:

try grep root /etc/passwd | cut -f5 -d: ; or echo "no root!"
# (cut would be run in this case, but do nothing)

try rm /whatever
# allowed to fail, you can still inspect $status to check whether it did
Contributor

timbertson commented Jun 10, 2013

fail if any undefined variable is referenced

With something like set -d, I think this makes sense to be the default (and only) behaviour.

fail if any command fails

fish relies more heavily on functions than most shells, and most functions cannot easily be made compatible with this

I don't understand why - I would have though that to implement this, a function failure would be treated like a command failure. And a failed command call made in a function would cause the function to also fail.

Regarding:

grep root /etc/passwd | and cut -f5 -d:

I don't think you could prevent cut from running if grep failed - something like tail -f would run indefinitely, so you'd need to start cut sometime. So it'd be more of a propagation of status codes rather than changing what gets run.

As an aside, I think and should be the default behvaiour (especially if "fail if any command fails" is the default - otherwise it's inconsistent).

Is there a way to preemtively allow a command to fail? i.e instead of modifying the syntax to allow a failed command if it's followed by an or, perhaps just introduce a builtin ("try"?) that runs the given command and returns its exit code but prevents the fail-on-error behaviour, Some examples:

try grep root /etc/passwd | cut -f5 -d: ; or echo "no root!"
# (cut would be run in this case, but do nothing)

try rm /whatever
# allowed to fail, you can still inspect $status to check whether it did
@andrusha

This comment has been minimized.

Show comment
Hide comment
@andrusha

andrusha Mar 29, 2014

It would be great to have strict mode to be enforceable for certain functions, not necessary for the whole fish-script. Currently I have a set of functions, which looks like this:

function update_dump -d "Downloads new version of dump, then imports it to DB"
        rsync -av --progress ...:~/dump.tar.gz dump.tar.gz;\
        and tar xzf ./dump.tar.gz;\
        and rake db:drop db:create;\
        and psql development < ./dev.sql;\
        and rm ./dev.sql;\
        and rake db:migrate db:test:prepare
end

Ideally I would like to have something like this:

function update_dump --strict -d "Downloads new version of dump, then imports it to DB"
        rsync -av --progress ...:~/dump.tar.gz dump.tar.gz
        tar xzf ./dump.tar.gz
        rake db:drop db:create
        psql development < ./dev.sql
        rm ./dev.sql
        rake db:migrate db:test:prepare
end

andrusha commented Mar 29, 2014

It would be great to have strict mode to be enforceable for certain functions, not necessary for the whole fish-script. Currently I have a set of functions, which looks like this:

function update_dump -d "Downloads new version of dump, then imports it to DB"
        rsync -av --progress ...:~/dump.tar.gz dump.tar.gz;\
        and tar xzf ./dump.tar.gz;\
        and rake db:drop db:create;\
        and psql development < ./dev.sql;\
        and rm ./dev.sql;\
        and rake db:migrate db:test:prepare
end

Ideally I would like to have something like this:

function update_dump --strict -d "Downloads new version of dump, then imports it to DB"
        rsync -av --progress ...:~/dump.tar.gz dump.tar.gz
        tar xzf ./dump.tar.gz
        rake db:drop db:create
        psql development < ./dev.sql
        rm ./dev.sql
        rake db:migrate db:test:prepare
end
@najamelan

This comment has been minimized.

Show comment
Hide comment
@najamelan

najamelan Apr 18, 2014

I would like to vouch for this issue. Creating a less cryptic syntax for shell scripting is a noble cause and since fish is on the right track, it would be good not to focus only on the interactive part.

najamelan commented Apr 18, 2014

I would like to vouch for this issue. Creating a less cryptic syntax for shell scripting is a noble cause and since fish is on the right track, it would be good not to focus only on the interactive part.

@ridiculousfish

This comment has been minimized.

Show comment
Hide comment
@ridiculousfish

ridiculousfish Apr 18, 2014

Member

Along the same lines, we could support a --strict option for begin as well.

Member

ridiculousfish commented Apr 18, 2014

Along the same lines, we could support a --strict option for begin as well.

@oconnor663

This comment has been minimized.

Show comment
Hide comment
@oconnor663

oconnor663 Jul 26, 2015

A more idiomatic way to do this in fish would be an event handler, something like this:

function die --on-event command-failed
    exit
end

but I'm not sure how this could be lexically scoped.

Solving this problem in general would be very useful. I've run into wanting to cd just for the duration of a function. (So did these guys.) It sounds like the same solution could apply there.

The brute force answer might be to make blocks aware of the cwd and of event listeners and other things like that, so that then we could have some special command for block-scoped-cd or block-scoped-on-signal. But that seems like a lot of special casing and work. One idea that seems to work well is what Python does with context managers. If there was some mechanism for defining a function to "do work now -- and then do this other thing when the current block ends", then it would be super easy to define block-scoped cd and listeners and everything else, and fish could consider defining some of those functions by default too.

oconnor663 commented Jul 26, 2015

A more idiomatic way to do this in fish would be an event handler, something like this:

function die --on-event command-failed
    exit
end

but I'm not sure how this could be lexically scoped.

Solving this problem in general would be very useful. I've run into wanting to cd just for the duration of a function. (So did these guys.) It sounds like the same solution could apply there.

The brute force answer might be to make blocks aware of the cwd and of event listeners and other things like that, so that then we could have some special command for block-scoped-cd or block-scoped-on-signal. But that seems like a lot of special casing and work. One idea that seems to work well is what Python does with context managers. If there was some mechanism for defining a function to "do work now -- and then do this other thing when the current block ends", then it would be super easy to define block-scoped cd and listeners and everything else, and fish could consider defining some of those functions by default too.

@Ericson2314

This comment has been minimized.

Show comment
Hide comment
@Ericson2314

Ericson2314 Jul 31, 2015

For the or problem. we could preserve the meaning of ;, ; and, and ; or, but make it so that newlines have the meaning of ; and.

Might sound weird to bash users, but Rust has got me use to final ; or lack thereof making a difference.

Ericson2314 commented Jul 31, 2015

For the or problem. we could preserve the meaning of ;, ; and, and ; or, but make it so that newlines have the meaning of ; and.

Might sound weird to bash users, but Rust has got me use to final ; or lack thereof making a difference.

@oconnor663

This comment has been minimized.

Show comment
Hide comment
@oconnor663

oconnor663 Jul 31, 2015

Special casing or for strict mode makes sense to me, unless that would be a huge change in the way or is implemented now. (Is it actually just a regular command?)

  1. Just make pipefail the default any only behavior of pipes. Any reason why it should not be?

That sounds great. For commands in a pipe whose exit status you want to suppress, you can use do something like this (assuming or does get special-cased):

... | begin command_with_weird_error_codes; or true; end | ...

This is the same thing you'd do in bash with pipefail on, just with fish's grouping syntax. Alternatively, you could define a function that does the same thing, and put that in the pipe.

oconnor663 commented Jul 31, 2015

Special casing or for strict mode makes sense to me, unless that would be a huge change in the way or is implemented now. (Is it actually just a regular command?)

  1. Just make pipefail the default any only behavior of pipes. Any reason why it should not be?

That sounds great. For commands in a pipe whose exit status you want to suppress, you can use do something like this (assuming or does get special-cased):

... | begin command_with_weird_error_codes; or true; end | ...

This is the same thing you'd do in bash with pipefail on, just with fish's grouping syntax. Alternatively, you could define a function that does the same thing, and put that in the pipe.

@faho

This comment has been minimized.

Show comment
Hide comment
@faho

faho Sep 15, 2015

Member

See also #510, which asks for a set -e equivalent.

Member

faho commented Sep 15, 2015

See also #510, which asks for a set -e equivalent.

@nateleavitt

This comment has been minimized.

Show comment
Hide comment
@nateleavitt

nateleavitt Oct 15, 2015

I also agree with this. I've been trying to write a fish script that does a variety of things with command line arguments, and it's been difficult. Maybe fish scripts don't fit well here...?

nateleavitt commented Oct 15, 2015

I also agree with this. I've been trying to write a fish script that does a variety of things with command line arguments, and it's been difficult. Maybe fish scripts don't fit well here...?

@hexsel

This comment has been minimized.

Show comment
Hide comment
@hexsel

hexsel Nov 30, 2015

I'd be ok with some functions disallowing failures or having handlers. Alternatively, maybe a try-catch style construct?

try
    cp abc.txt /someinvalidfolder
catch
    echo $_fish_failure $_fish_failed_cmd $fish_failure_line
    exit $_fish_failure_code
end

hexsel commented Nov 30, 2015

I'd be ok with some functions disallowing failures or having handlers. Alternatively, maybe a try-catch style construct?

try
    cp abc.txt /someinvalidfolder
catch
    echo $_fish_failure $_fish_failed_cmd $fish_failure_line
    exit $_fish_failure_code
end
@Ericson2314

This comment has been minimized.

Show comment
Hide comment
@Ericson2314

Ericson2314 Nov 30, 2015

For process failure, begin ...; end; or ... is try-catch. Conversely, I don't think using undefined variables should be recoverable.

Ericson2314 commented Nov 30, 2015

For process failure, begin ...; end; or ... is try-catch. Conversely, I don't think using undefined variables should be recoverable.

@hexsel

This comment has been minimized.

Show comment
Hide comment
@hexsel

hexsel Nov 30, 2015

@Ericson2314 maybe I'm missing something, the documentation for "begin" (http://fishshell.com/docs/current/commands.html#begin) seems to mention it "The block is unconditionally executed" which is the opposite that this is suggesting.

The point here IMO was to avoid writing a 100 line script with every single line preceded by "and".

hexsel commented Nov 30, 2015

@Ericson2314 maybe I'm missing something, the documentation for "begin" (http://fishshell.com/docs/current/commands.html#begin) seems to mention it "The block is unconditionally executed" which is the opposite that this is suggesting.

The point here IMO was to avoid writing a 100 line script with every single line preceded by "and".

@Ericson2314

This comment has been minimized.

Show comment
Hide comment
@Ericson2314

Ericson2314 Nov 30, 2015

Sorry, I meant that with respect to one of the proposals above, where we change either the meaning of ; or make newlines desugar to ; and instead of ;. Then the first command to fail causes the entire begin block to fail.

Ericson2314 commented Nov 30, 2015

Sorry, I meant that with respect to one of the proposals above, where we change either the meaning of ; or make newlines desugar to ; and instead of ;. Then the first command to fail causes the entire begin block to fail.

@cben

This comment has been minimized.

Show comment
Hide comment
@cben

cben Dec 21, 2015

Contributor

I scripts/blocks fail on first failing command (whether opt-in or by default),
I think a non-silent failure would be more useful.
Don't just stop, print the offending command and exit code to stderr. Like make does.
Not sure that'll always be desired, but I think silently stopping, with no hint where it stopped or that it stopped at all [1] is — in most cases — a bug in itself.

Verbose failure would also mean foo; and bar at top level also needs some special casing.

About newline differing from ; — it's a clever hack but will backfire on this quite sensible style:

long_command_1
or long_command_2
or long_command_3

IMHO too subtle non-orthogonality. No matter how well it's documented, people who reformat code between ; and separate lines will face some very surprising debugging.

It sounds like special-casing the or/and builtins solves everything?
(Custom or-like functions that inspect output from previous commands would still be mostly useless.)

Contributor

cben commented Dec 21, 2015

I scripts/blocks fail on first failing command (whether opt-in or by default),
I think a non-silent failure would be more useful.
Don't just stop, print the offending command and exit code to stderr. Like make does.
Not sure that'll always be desired, but I think silently stopping, with no hint where it stopped or that it stopped at all [1] is — in most cases — a bug in itself.

Verbose failure would also mean foo; and bar at top level also needs some special casing.

About newline differing from ; — it's a clever hack but will backfire on this quite sensible style:

long_command_1
or long_command_2
or long_command_3

IMHO too subtle non-orthogonality. No matter how well it's documented, people who reformat code between ; and separate lines will face some very surprising debugging.

It sounds like special-casing the or/and builtins solves everything?
(Custom or-like functions that inspect output from previous commands would still be mostly useless.)

@Ericson2314

This comment has been minimized.

Show comment
Hide comment
@Ericson2314

Ericson2314 Dec 22, 2015

Yeah it's a dirty trick, I won't deny it. But note that there is no need to use or foo on its own line when plain foo means the same thing (in "strict mode").

I must say, I am not a fan of the ; or/; and syntax. These are binary operators, so syntactically making them look like unary ones is completely misleading.

Ericson2314 commented Dec 22, 2015

Yeah it's a dirty trick, I won't deny it. But note that there is no need to use or foo on its own line when plain foo means the same thing (in "strict mode").

I must say, I am not a fan of the ; or/; and syntax. These are binary operators, so syntactically making them look like unary ones is completely misleading.

@cben

This comment has been minimized.

Show comment
Hide comment
@cben

cben Dec 31, 2015

Contributor

note that there is no need to use or foo on its own line when plain foo means the same thing (in "strict mode").

The reason to say or foo on its own line is when commands are long and putting each on its own line improves readability. I don't want strict mode to break the ;/newline equivalence, because I think nobody will remember the subtle difference.


I'm not entirely a fan of the unary or/and either. It's just that until recently it had no deal-breaking drawbacks. The other complaints I've seen were:

  1. It's not like in bash.
  2. if and while take one command (unlike bash which has then/do) so require awkward if begin COND1; and COND2; end; BODY; end.
  3. Nesting conditions requires awkward begin COND1; or COND2; end; and COND3.

I'd say (2) and (3) are actually complaints about begin..end.
(2) has been practically solved in #1428 by making and/or significant syntax — they magically group after if/while.

If we didn't care about backward compatibility, I'd say the right syntax is not unary nor binary — it's Lispish surrounding. Using "and"/"or" in prefix position wouldn't read like english (and is too incompatible) but perhaps:

all
    COND1
    either
        COND2
        COND3
    end
end

would be good?

However given compatibility it seems we're gonna keep unary or/and, yet make them more and more significant syntax.
Which is slightly messy, as they can no longer be explained simply as "check $status and maybe run command"...

Contributor

cben commented Dec 31, 2015

note that there is no need to use or foo on its own line when plain foo means the same thing (in "strict mode").

The reason to say or foo on its own line is when commands are long and putting each on its own line improves readability. I don't want strict mode to break the ;/newline equivalence, because I think nobody will remember the subtle difference.


I'm not entirely a fan of the unary or/and either. It's just that until recently it had no deal-breaking drawbacks. The other complaints I've seen were:

  1. It's not like in bash.
  2. if and while take one command (unlike bash which has then/do) so require awkward if begin COND1; and COND2; end; BODY; end.
  3. Nesting conditions requires awkward begin COND1; or COND2; end; and COND3.

I'd say (2) and (3) are actually complaints about begin..end.
(2) has been practically solved in #1428 by making and/or significant syntax — they magically group after if/while.

If we didn't care about backward compatibility, I'd say the right syntax is not unary nor binary — it's Lispish surrounding. Using "and"/"or" in prefix position wouldn't read like english (and is too incompatible) but perhaps:

all
    COND1
    either
        COND2
        COND3
    end
end

would be good?

However given compatibility it seems we're gonna keep unary or/and, yet make them more and more significant syntax.
Which is slightly messy, as they can no longer be explained simply as "check $status and maybe run command"...

@Ericson2314

This comment has been minimized.

Show comment
Hide comment
@Ericson2314

Ericson2314 Dec 31, 2015

Hmm I like it! begin, all, and any ("either" implies 2, and "any" is used for this elsewhere) are 3 basic kinds of blocks, each of which interpret newline/; in different ways. Safe mode is about making other blocks act like all rather than begin (Including the implicit top-level block).

Ericson2314 commented Dec 31, 2015

Hmm I like it! begin, all, and any ("either" implies 2, and "any" is used for this elsewhere) are 3 basic kinds of blocks, each of which interpret newline/; in different ways. Safe mode is about making other blocks act like all rather than begin (Including the implicit top-level block).

@timbertson

This comment has been minimized.

Show comment
Hide comment
@timbertson

timbertson Jan 5, 2016

Contributor

I'd argue that either / any could be misleading - I might expect something called any to bail out as soon as one command succeeds (like a chain of || in bash). I think try may be a better name. There's no catch, it's just that failed statements in a try block do not automatically cause termination. So it'd be:

all
    COND1
    try
        COND2; or echo "That failed, but we don't care..."
        COND3
    end
end

..and by default (to maintain backwards compat), the topmost block is implicitly a try. I don't know how begin would interact with this - maybe it could turn into a deprecated alias for try?

Contributor

timbertson commented Jan 5, 2016

I'd argue that either / any could be misleading - I might expect something called any to bail out as soon as one command succeeds (like a chain of || in bash). I think try may be a better name. There's no catch, it's just that failed statements in a try block do not automatically cause termination. So it'd be:

all
    COND1
    try
        COND2; or echo "That failed, but we don't care..."
        COND3
    end
end

..and by default (to maintain backwards compat), the topmost block is implicitly a try. I don't know how begin would interact with this - maybe it could turn into a deprecated alias for try?

@Ericson2314

This comment has been minimized.

Show comment
Hide comment
@Ericson2314

Ericson2314 Jan 6, 2016

@timbertson haha, I would expect that too, because that is what I intended! begin...end is ; chain, all...end is && chain, and any...end is || chain. Because there is no catch as you noted, I prefer all over try.

Now all and any should mean the same thing in "safe mode" and "scary legacy mode". Maybe begin would be called unconditional in "safe mode" to be a bit clearer? We can bikeshed a shorter name, or just keep begin.

Let's just disallow or and and in "safe mode", since nobody has a good plan for implementing them or making them not terribly confusing.

Ericson2314 commented Jan 6, 2016

@timbertson haha, I would expect that too, because that is what I intended! begin...end is ; chain, all...end is && chain, and any...end is || chain. Because there is no catch as you noted, I prefer all over try.

Now all and any should mean the same thing in "safe mode" and "scary legacy mode". Maybe begin would be called unconditional in "safe mode" to be a bit clearer? We can bikeshed a shorter name, or just keep begin.

Let's just disallow or and and in "safe mode", since nobody has a good plan for implementing them or making them not terribly confusing.

@timbertson

This comment has been minimized.

Show comment
Hide comment
@timbertson

timbertson Jan 6, 2016

Contributor

I think that may be a bit too drastic - making inside-out lisp-ish logic for

What about a very targeted change - by default, everything is strict. But we introduce a try keyword which can prefix any command. It won't prevent that command from returning a nonzero $status, but it will (internally, somehow) prevent the terminate-on-error logic. So you could then do:

do_something
try something_which_might_work; or echo "that failed!"

and and or still do what they're supposed to (although and becomes less necessary), while we get an obvious prefix (try) for commands that we're allowing to fail.

Contributor

timbertson commented Jan 6, 2016

I think that may be a bit too drastic - making inside-out lisp-ish logic for

What about a very targeted change - by default, everything is strict. But we introduce a try keyword which can prefix any command. It won't prevent that command from returning a nonzero $status, but it will (internally, somehow) prevent the terminate-on-error logic. So you could then do:

do_something
try something_which_might_work; or echo "that failed!"

and and or still do what they're supposed to (although and becomes less necessary), while we get an obvious prefix (try) for commands that we're allowing to fail.

@Ericson2314

This comment has been minimized.

Show comment
Hide comment
@Ericson2314

Ericson2314 Jan 6, 2016

what do these do?

begin; begin; try false; end; end;
begin; not try true; end;

IMO, the correct way to think about strict mode is not that things terminate on error, but that ignored error codes are bubbled up until something inspects them. I'd argue that it is due to this confusion that set -e is implement so inconsistently in traditional shells (http://www.in-ulm.de/~mascheck/various/set-e/).

Also is it so drastic if strict mode is optional? Like the old syntax, it re-uses an existing syntactic form foobar...end, rather than introducing something new such as infix syntax.

Ericson2314 commented Jan 6, 2016

what do these do?

begin; begin; try false; end; end;
begin; not try true; end;

IMO, the correct way to think about strict mode is not that things terminate on error, but that ignored error codes are bubbled up until something inspects them. I'd argue that it is due to this confusion that set -e is implement so inconsistently in traditional shells (http://www.in-ulm.de/~mascheck/various/set-e/).

Also is it so drastic if strict mode is optional? Like the old syntax, it re-uses an existing syntactic form foobar...end, rather than introducing something new such as infix syntax.

@anordal

This comment has been minimized.

Show comment
Hide comment
@anordal

anordal Mar 22, 2016

Contributor

One reason to want strictness in fish, is that bash is beyond hope hard to write safely. Just look at this:

fish

$a

bash (in set set -u mode)

${a[@]+"${a[@]}"}

Yep, set -u in bash treats empty arrays as an error, unless you're willing to accept that workaround. That's unforgivable!

Or how about this bash function:

timesTen() { echo "$10"; }

…prints the first argument and then a zero!

I thought I knew how to write bash safely, you know, by quoting variables like a nazi and using all the "strict mode" tricks in the book. But the more I learn, the more betrayed and heartbroken I feel, which makes me turn to fish in higher hopes.

I guess set -u might be the least problematic of the "strict mode" options to enable in fish — would anyone even want to turn it off?

Contributor

anordal commented Mar 22, 2016

One reason to want strictness in fish, is that bash is beyond hope hard to write safely. Just look at this:

fish

$a

bash (in set set -u mode)

${a[@]+"${a[@]}"}

Yep, set -u in bash treats empty arrays as an error, unless you're willing to accept that workaround. That's unforgivable!

Or how about this bash function:

timesTen() { echo "$10"; }

…prints the first argument and then a zero!

I thought I knew how to write bash safely, you know, by quoting variables like a nazi and using all the "strict mode" tricks in the book. But the more I learn, the more betrayed and heartbroken I feel, which makes me turn to fish in higher hopes.

I guess set -u might be the least problematic of the "strict mode" options to enable in fish — would anyone even want to turn it off?

@actionless

This comment has been minimized.

Show comment
Hide comment
@actionless

actionless Mar 28, 2016

Contributor

is anyone working on it or it's still on the discussion stage?
i am feeling like i could try to implement it if it's cleared up how the syntax for that could look like

Contributor

actionless commented Mar 28, 2016

is anyone working on it or it's still on the discussion stage?
i am feeling like i could try to implement it if it's cleared up how the syntax for that could look like

@krader1961 krader1961 modified the milestones: fish-tank, fish-future Mar 28, 2016

@krader1961

This comment has been minimized.

Show comment
Hide comment
@krader1961

krader1961 Mar 28, 2016

Contributor

I just quickly skimmed this issue. It is most definitely still in the discussion stage. And some of the proposals, like make the default behavior be "strict", could only be done in a major release because it breaks backward compatibility.

Contributor

krader1961 commented Mar 28, 2016

I just quickly skimmed this issue. It is most definitely still in the discussion stage. And some of the proposals, like make the default behavior be "strict", could only be done in a major release because it breaks backward compatibility.

@actionless

This comment has been minimized.

Show comment
Hide comment
@actionless

actionless Mar 28, 2016

Contributor

i personally prefer the way with setting that behavior via variable

Contributor

actionless commented Mar 28, 2016

i personally prefer the way with setting that behavior via variable

@OlivierDupre

This comment has been minimized.

Show comment
Hide comment
@OlivierDupre

OlivierDupre Jun 22, 2016

For information, I have a workaround for set -x equivalent.

set cmd "My command here with my "'$param" here and "$anotherParam
echo $cmd
eval $cmd

Of course, this is less elegant, but it works... ;)

OlivierDupre commented Jun 22, 2016

For information, I have a workaround for set -x equivalent.

set cmd "My command here with my "'$param" here and "$anotherParam
echo $cmd
eval $cmd

Of course, this is less elegant, but it works... ;)

@anordal

This comment has been minimized.

Show comment
Hide comment
@anordal

anordal Jul 8, 2016

Contributor

Yes, set -x is for debugging, and there are other ways to debug that are nearly as good. Plus, people tend to clean up their debug printouts when done debugging, as opposed to leaving set -x enabled, which I'm no fan of – I agree with Dag that it could be an option to fish. Anyway, I think set -x is a separate discussion, as it doesn't make the language any stricter. set -euo pipefail is bash's strict mode.

But while I'm at it:

echo $cmd

My favourite is echo «$cmd». Exploits cartesian product to print each argument enclosed in those funny guillemet quotes (choose something funny on your keyboard) so you can tell the arguments apart in case they contain whitespace.

Contributor

anordal commented Jul 8, 2016

Yes, set -x is for debugging, and there are other ways to debug that are nearly as good. Plus, people tend to clean up their debug printouts when done debugging, as opposed to leaving set -x enabled, which I'm no fan of – I agree with Dag that it could be an option to fish. Anyway, I think set -x is a separate discussion, as it doesn't make the language any stricter. set -euo pipefail is bash's strict mode.

But while I'm at it:

echo $cmd

My favourite is echo «$cmd». Exploits cartesian product to print each argument enclosed in those funny guillemet quotes (choose something funny on your keyboard) so you can tell the arguments apart in case they contain whitespace.

@anordal

This comment has been minimized.

Show comment
Hide comment
@anordal

anordal Jul 8, 2016

Contributor

Why do we need a separate mode?

  • in bash, set -e is useless in interactive mode because any failing command logs you out. But I think it should be possible to have the same strict language (wrt. failure propagation) interactively, and just not get logged out.
  • backward compatibility (with people's scripts). Or we could do it for fish 3.0 and forget about backward compatibility (à la python3).

fail if any command fails

fish relies more heavily on functions than most shells, and most functions cannot easily be made compatible with this

I don't understand why - I would have thought that to implement this, a function failure would be treated like a command failure. And a failed command call made in a function would cause the function to also fail.

I don't understand either. Are return values hard to control? Do we use $status to communicate other things than failure? I do know a builtin that does that: count with no arguments returns 1 on success. We have at least 15 functions that depend on that (grep for "if count"). In my mind, that just means we must fix those as well. As a first step, the documentation should avoid advertising any entropy in $status on success.

Contributor

anordal commented Jul 8, 2016

Why do we need a separate mode?

  • in bash, set -e is useless in interactive mode because any failing command logs you out. But I think it should be possible to have the same strict language (wrt. failure propagation) interactively, and just not get logged out.
  • backward compatibility (with people's scripts). Or we could do it for fish 3.0 and forget about backward compatibility (à la python3).

fail if any command fails

fish relies more heavily on functions than most shells, and most functions cannot easily be made compatible with this

I don't understand why - I would have thought that to implement this, a function failure would be treated like a command failure. And a failed command call made in a function would cause the function to also fail.

I don't understand either. Are return values hard to control? Do we use $status to communicate other things than failure? I do know a builtin that does that: count with no arguments returns 1 on success. We have at least 15 functions that depend on that (grep for "if count"). In my mind, that just means we must fix those as well. As a first step, the documentation should avoid advertising any entropy in $status on success.

@Ericson2314

This comment has been minimized.

Show comment
Hide comment
@Ericson2314

Ericson2314 Jul 9, 2016

@anordal I agree this should work fine in interactive mode. Every real language with fatal errors doesn't exit the REPL when one is encountered.

Comparability is still a concern though---consensus to a make a breaking release doesn't mean the problem goes a way but just that people decided to move forwards despite it.

Ericson2314 commented Jul 9, 2016

@anordal I agree this should work fine in interactive mode. Every real language with fatal errors doesn't exit the REPL when one is encountered.

Comparability is still a concern though---consensus to a make a breaking release doesn't mean the problem goes a way but just that people decided to move forwards despite it.

@kopischke

This comment has been minimized.

Show comment
Hide comment
@kopischke

kopischke Jul 10, 2016

Having had to contend with side effects of rbenv’s decision to go “strict” I can only deem idiotic [1, 2], I must say I am less than sold on it. I tend to agree with GreyCat’s assessment that bash’s set -e is a non-feature, bordering on an antipattern, very much inferior to explicit error checking.

kopischke commented Jul 10, 2016

Having had to contend with side effects of rbenv’s decision to go “strict” I can only deem idiotic [1, 2], I must say I am less than sold on it. I tend to agree with GreyCat’s assessment that bash’s set -e is a non-feature, bordering on an antipattern, very much inferior to explicit error checking.

@Ericson2314

This comment has been minimized.

Show comment
Hide comment
@Ericson2314

Ericson2314 Jul 10, 2016

@kopischke I agree that bash's set -e is broken, and introduces as many subtle gotchas as it alleviates. But... I think that's just sh being sh and not an idea with the problem itself?

See my comment in #805 (comment) I think sh's crucial mistake was to approach the mode from a "failure means exit" perspective. Then things like x=${foo} and if false ... are important exceptions to the rule. If one thinks instead in terms of making sure errors are not implicitly ignored, as I say in comment, then it's clear that the assignment to x "remember" the exit status, and if handles the exit status, so aborting should not be done.

Not only is this a good mental model for the fish user, but it also lends itself to an implementation strategy (make sure whatever holds the exit status isn't destroyed but moved), so I'm confident that we can avoid gotchas.

Ericson2314 commented Jul 10, 2016

@kopischke I agree that bash's set -e is broken, and introduces as many subtle gotchas as it alleviates. But... I think that's just sh being sh and not an idea with the problem itself?

See my comment in #805 (comment) I think sh's crucial mistake was to approach the mode from a "failure means exit" perspective. Then things like x=${foo} and if false ... are important exceptions to the rule. If one thinks instead in terms of making sure errors are not implicitly ignored, as I say in comment, then it's clear that the assignment to x "remember" the exit status, and if handles the exit status, so aborting should not be done.

Not only is this a good mental model for the fish user, but it also lends itself to an implementation strategy (make sure whatever holds the exit status isn't destroyed but moved), so I'm confident that we can avoid gotchas.

@kopischke

This comment has been minimized.

Show comment
Hide comment
@kopischke

kopischke Jul 10, 2016

IMO, the correct way to think about strict mode is not that things terminate on error, but that ignored error codes are bubbled up until something inspects them.

Well, that is certainly a much more sane concept than the errexit mess, but I am still dubious: what bit the rbenv-plugin developers in the, ahem, lower back, and what I tried to fix was the assumption that return codes above 0 signal an error, when, in the case of grep, the code is a Boolean return value.

With exit values the only reliable way to communicate results without worrying about stdout or stderr (which can be suppressed and often have another dedicated use, as is the case for grep), that kind of behaviour makes sense and is orthogonal to the assumption that non-zero is an error. bash’s errexit adds insult to injury, but at the core is an issue every “error handling” mechanism will have to cope with, barring us all starting to use something like PowerShell.

Also is it so drastic if strict mode is optional?

That depends entirely on how painless it is to break out of its scope, be it in plugins / extensions / subscripts or globally without affecting the main script, I’d say.

kopischke commented Jul 10, 2016

IMO, the correct way to think about strict mode is not that things terminate on error, but that ignored error codes are bubbled up until something inspects them.

Well, that is certainly a much more sane concept than the errexit mess, but I am still dubious: what bit the rbenv-plugin developers in the, ahem, lower back, and what I tried to fix was the assumption that return codes above 0 signal an error, when, in the case of grep, the code is a Boolean return value.

With exit values the only reliable way to communicate results without worrying about stdout or stderr (which can be suppressed and often have another dedicated use, as is the case for grep), that kind of behaviour makes sense and is orthogonal to the assumption that non-zero is an error. bash’s errexit adds insult to injury, but at the core is an issue every “error handling” mechanism will have to cope with, barring us all starting to use something like PowerShell.

Also is it so drastic if strict mode is optional?

That depends entirely on how painless it is to break out of its scope, be it in plugins / extensions / subscripts or globally without affecting the main script, I’d say.

@ridiculousfish

This comment has been minimized.

Show comment
Hide comment
@ridiculousfish

ridiculousfish Jul 10, 2016

Member

Thanks for your links to the ruby issues - very instructive. One idea I had is that strict mode is a property of a scope. This would avoid opting code like plugins in when they don't expect it. Do you think that would help?

Member

ridiculousfish commented Jul 10, 2016

Thanks for your links to the ruby issues - very instructive. One idea I had is that strict mode is a property of a scope. This would avoid opting code like plugins in when they don't expect it. Do you think that would help?

@kopischke

This comment has been minimized.

Show comment
Hide comment
@kopischke

kopischke Jul 10, 2016

@ridiculousfish that would certainly help: as far as I’m concerned, the more limited and explicit the scoping of this kind of feature, the better.

kopischke commented Jul 10, 2016

@ridiculousfish that would certainly help: as far as I’m concerned, the more limited and explicit the scoping of this kind of feature, the better.

@Ericson2314

This comment has been minimized.

Show comment
Hide comment
@Ericson2314

Ericson2314 Jul 11, 2016

Yeah strict mode can and should be lexical. I'd like it to eventually become mandatory (say with breaking, major number bump, fish release), but even then we have three kinds of blocks: do all commands, break at first failure, break at first success. (Look earlier in the thread to read more about this.)

@kopischke In your case I see two principles. If the non-standard error codes are explicitly handled, then no implicit aborts will happen. If the non-standard error codes are not implicitly handled, then whatever information they additionally encode will be discarded anyways. I'd make a function, taking a command and args, which calls the command than converts the error code to the standard. Then the error code of the function will be handled implicit by strict mode in the original script, which will be the correct behavior.

my-wrapper-fun the-ruby-cmd args...

Ericson2314 commented Jul 11, 2016

Yeah strict mode can and should be lexical. I'd like it to eventually become mandatory (say with breaking, major number bump, fish release), but even then we have three kinds of blocks: do all commands, break at first failure, break at first success. (Look earlier in the thread to read more about this.)

@kopischke In your case I see two principles. If the non-standard error codes are explicitly handled, then no implicit aborts will happen. If the non-standard error codes are not implicitly handled, then whatever information they additionally encode will be discarded anyways. I'd make a function, taking a command and args, which calls the command than converts the error code to the standard. Then the error code of the function will be handled implicit by strict mode in the original script, which will be the correct behavior.

my-wrapper-fun the-ruby-cmd args...
@anordal

This comment has been minimized.

Show comment
Hide comment
@anordal

anordal Jul 17, 2016

Contributor

Agree that when it comes to modes, lexical scoping is preferrable over global state. But what about inner scopes (such as if, while, for and so on) of a strict mode scope, are they also strict? I think they ought to be, all the way in to a scope that is explicitly not, but not cross function boundaries:

begin --break-on-failure
    if true
        # still in strict mode
        begin --no-break-on-failure
            # strict mode disabled
        end
    end
end

Alternatively, it could be purely a function-scope thing.

I think the requirements are:

  1. it must not cross software module boundaries. I suppose not crossing function boundaries would satisfy this.
  2. it makes sense to restrict the number of places one has to look for it
  3. as a safety feature, it's not good if it's easy to accidentally disable by adding an inner scope

Reason for (2): You cannot proofread a piece of code without knowing if strict mode is on or off: In normal mode, you must handle the errors; in strict mode you must handle the non-errors (usually by ; or true). Expected errors should be handled anyway, for human readable error messages, but that's a different concern.

the assumption that return codes above 0 signal an error, when, in the case of grep, the code is a Boolean return value.

aka. the non-errors. But depends on context, as you would consider it an error if grep doesn't find the match in a situation where you expect it to.

Reason for (3): The way I look at all kinds of strict mode, they are safety features: They make sloppy code fail, so the paranoid programmer can know that the code is free of certain classes of bugs. Specifically with --break-on-failure, that if the program works, it means that no exit status handling is missing.

Contributor

anordal commented Jul 17, 2016

Agree that when it comes to modes, lexical scoping is preferrable over global state. But what about inner scopes (such as if, while, for and so on) of a strict mode scope, are they also strict? I think they ought to be, all the way in to a scope that is explicitly not, but not cross function boundaries:

begin --break-on-failure
    if true
        # still in strict mode
        begin --no-break-on-failure
            # strict mode disabled
        end
    end
end

Alternatively, it could be purely a function-scope thing.

I think the requirements are:

  1. it must not cross software module boundaries. I suppose not crossing function boundaries would satisfy this.
  2. it makes sense to restrict the number of places one has to look for it
  3. as a safety feature, it's not good if it's easy to accidentally disable by adding an inner scope

Reason for (2): You cannot proofread a piece of code without knowing if strict mode is on or off: In normal mode, you must handle the errors; in strict mode you must handle the non-errors (usually by ; or true). Expected errors should be handled anyway, for human readable error messages, but that's a different concern.

the assumption that return codes above 0 signal an error, when, in the case of grep, the code is a Boolean return value.

aka. the non-errors. But depends on context, as you would consider it an error if grep doesn't find the match in a situation where you expect it to.

Reason for (3): The way I look at all kinds of strict mode, they are safety features: They make sloppy code fail, so the paranoid programmer can know that the code is free of certain classes of bugs. Specifically with --break-on-failure, that if the program works, it means that no exit status handling is missing.

@krader1961

This comment has been minimized.

Show comment
Hide comment
@krader1961

krader1961 Jul 17, 2016

Contributor
  1. it must not cross software module boundaries. I suppose not crossing function boundaries would satisfy this.

Presumably this rule also includes source since it represents a module boundary akin to python's import.

FWIW, I used to use set -e (or set --errexit) a lot. Then I learned better habits and stopped using it for the crutch that it is. So I'm not a fan of introducing it to fish but do recognize for certain classes of scripts it has utility. If we do add this feature it absolutely has to have lexical scoping and function and module boundaries represent different lexical scopes. It also must not be the default behavior.

Contributor

krader1961 commented Jul 17, 2016

  1. it must not cross software module boundaries. I suppose not crossing function boundaries would satisfy this.

Presumably this rule also includes source since it represents a module boundary akin to python's import.

FWIW, I used to use set -e (or set --errexit) a lot. Then I learned better habits and stopped using it for the crutch that it is. So I'm not a fan of introducing it to fish but do recognize for certain classes of scripts it has utility. If we do add this feature it absolutely has to have lexical scoping and function and module boundaries represent different lexical scopes. It also must not be the default behavior.

@actionless

This comment has been minimized.

Show comment
Hide comment
@actionless

actionless Sep 2, 2016

Contributor

Then I learned better habits

do you mean handling each command's return code/output or smth else?

Contributor

actionless commented Sep 2, 2016

Then I learned better habits

do you mean handling each command's return code/output or smth else?

@krader1961

This comment has been minimized.

Show comment
Hide comment
@krader1961

krader1961 Oct 6, 2016

Contributor

Closing this as a duplicate of issue #510. While it is true this issue was originally broader than just dealing with unhandled command failures the fact is the discussion has been focused almost solely on the latter. So I'm going to close this as a duplicate of #510. Other topics, such as a bash set -x equivalent for fish should be discussed in issues dealing with each specific feature (e.g., #3427).

Contributor

krader1961 commented Oct 6, 2016

Closing this as a duplicate of issue #510. While it is true this issue was originally broader than just dealing with unhandled command failures the fact is the discussion has been focused almost solely on the latter. So I'm going to close this as a duplicate of #510. Other topics, such as a bash set -x equivalent for fish should be discussed in issues dealing with each specific feature (e.g., #3427).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment