Leverage knowledge of subshell status

[jmooring]

Yes, the title is a bit cryptic. In short, it's possible to know whether or not a function is executing within a subshell, as would be the case when invoked via command substitution.

If we know that we're not in a subshell, then we know that it's OK to update global variables that existed in the calling function or script, and we know that global variables that we create will be visible to the calling function or script.

This allows us to code defensively, and potentially provide additional functionality.

Example 1: Defensive Coding

The bfl::declare_ansi_escape_sequences library function creates a number of global variables. It is invoked directly (not via command substitution) immediately after it is defined. If someone were to invoke it via command substitution (i.e., in a subshell), the global variables it creates would be global to the subshell, and not visible in the parent shell.

We could test how the function was called, and throw an error if it is being called in a subshell.

Example 2: Additional Functionality

The bfl::trim library function prints a trimmed version of string to stdout, and is normally called via command substitution to assign the output to a variable:

trimmed=$(bfl::trim "${foo}") || exit 1
echo "${trimmed}"

But, if we really wanted to, if we were sure we weren't in a subshell, we could (additionally) have the function create a global variable with the same name as the function, and assign the output to that.

In pseudo-code the bfl::trim function might look like this:

if [[ in_subshell ]]; then
  declare -g bfl_trim
  bfl_trim=${something}
else
  printf "%s" ${something}
fi

Then you could do either of these:

bfl::trim "${foo}" || exit 1
echo "bfl_trim"

trimmed=$(bfl::trim "${foo}") || exit 1
echo "${trimmed}"

Background: https://unix.stackexchange.com/questions/594806/can-i-determine-if-the-current-function-has-been-invoked-via-command-substitutio#594809

Working example:

foo() {
  if [[ "${BASH_SUBSHELL}" -ne "0" ]]; then
    echo "foo() is running in a subshell."
  else
    echo "foo() is NOT running in a subshell."
  fi
}

# Call foo directly:
foo

# Call foo via command substitution:
a=$(foo) || exit 1
echo "${a}"

[jmooring]

Maybe a bfl::in_subshell function to make it easy... as long as we don't call it via command substitution 😏