macOS: retdec-* breaks due to explicit use of #!/bin/bash

[AshKetch-um]

On macOS the built-in bash and the associated tools are incompatible with many different things being done by retdec-* scripts. The solution is for this is to use homebrew or Mac Ports to provide a GNU compatible environment (see: Run on MacOS).

There are a number of complicating factors between homebrew/ports and GNU-targeted tools however, one of the key issues is the collisions between built-in tools (e.g. bash, sed, redline) and those provided by these package managers. A common solution is to place the path to the GNU tools symlink tree earlier in the PATH than the built-in tools (e.g. export PATH=/opt/local/bin:/opt/local/sbin:/opt/local/libexec/gnubin) thus allowing GNU targeted tools to work without having to be specifically aware of the quirks of macOS/brew/ports.

Most of these issues have been identified and resolved as part of the README.md however, the retdec-* scripts explicitly utilize #!/bin/bash which will cause them to execute using the built-in bash rather than that of the current shell. This results in retdec-* only working when run using a dedicated brew/ports bash instance (not the current shell) and when run from the installation directory of retdec. This is because the explicit #!/bin/bash is causing the built-in bash to be used in place of the currently running shell, causing cascading issues in a number of different places (see: Observed Behavior below).

The fix that worked for me is replacing the explicit usage of #!/bin/bash in all to the shell scripts with the more environmentally friendly version #!/usr/bin/env bash. This allows the running shell (e.g. /opt/local/bin/bash to be used and thus, allows retdec's installation path to included into the PATH and thus run from anywhere. ( Pull Request #259 )

Environment
macOS 10.13.3
Bash: 4.4.19(1)-release (x86_64-apple-darwin17.4.0)
Mac Ports
changeset: 31cf35f

Steps to Reproduce
Download, Build and Install on MacOS

sudo port selfupdate
# See #257
sudo port install cmake git perl5 python27 bison flex autoconf automake libtool coreutils wget gsed doxygen upx graphviz

# See #98
export PATH=/opt/local/bin:/opt/local/sbin:/opt/local/libexec/gnubin:$PATH

git clone https://github.com/avast-tl/retdec
cd retdec
mkdir build && cd build
cmake .. -DCMAKE_INSTALLPREFIX=/opt/local/retdec
make -j6
sudo make install

Run against a simple test 32-bit Linux ELF binary

export PATH=$PATH:/opt/local/retdec/bin
retdec-decompiler.sh /tmp/test

Expected Behavior
Clean run

user@host: ~$ retdec-decompiler.sh /tmp/test
...
Running phase: emission of the target code [c] ( 0.03s )
Running phase: finalization ( 0.03s )
Running phase: cleanup ( 0.04s )

##### Done!

Observed Behavior
Run from path directly on current shell:

user@host:~$ retdec-decompiler.sh /tmp/test
##### Checking if file is a Mach-O Universal static library...
RUN: /opt/local/retdec/bin/retdec-macho-extractor --list /private/tmp/test

##### Checking if file is an archive...
RUN: /opt/local/retdec/bin/retdec-ar-extractor --arch-magic /private/tmp/test
Not an archive, going to the next step.
/opt/local/retdec/bin/retdec-decompiler.sh: line 967: unexpected EOF while looking for matching `"'
/opt/local/retdec/bin/retdec-decompiler.sh: line 968: syntax error: unexpected end of file

Run from path using dedicate bash instance:

user@host:~$ bash retdec-decompiler.sh /tmp/test
/opt/local/retdec/bin/retdec-decompiler.sh: line 12: ./retdec-utils.sh: No such file or directory
/opt/local/retdec/bin/retdec-decompiler.sh: line 156: get_realpath: command not found
/opt/local/retdec/bin/retdec-decompiler.sh: line 157: get_realpath: command not found
##### Checking if file is a Mach-O Universal static library...
RUN:  --list 
/opt/local/retdec/bin/retdec-decompiler.sh: line 501: is_macho_archive: command not found

##### Checking if file is an archive...
RUN:  --arch-magic 
/opt/local/retdec/bin/retdec-decompiler.sh: line 527: has_archive_signature: command not found
Not an archive, going to the next step.

##### Gathering file information...
RUN:  -c .json --similarity  --no-hashes=all
/opt/local/retdec/bin/retdec-decompiler.sh: line 648: : command not found
/opt/local/retdec/bin/retdec-decompiler.sh: line 654: print_error_and_die: command not found
/opt/local/retdec/bin/retdec-decompiler.sh: line 662: : command not found
/opt/local/retdec/bin/retdec-decompiler.sh: line 709: : command not found
/opt/local/retdec/bin/retdec-decompiler.sh: line 715: : command not found
/opt/local/retdec/bin/retdec-decompiler.sh: line 736: print_error_and_die: command not found
/opt/local/retdec/bin/retdec-decompiler.sh: line 741: : command not found
/opt/local/retdec/bin/retdec-decompiler.sh: line 744: print_error_and_die: command not found
/opt/local/retdec/bin/retdec-decompiler.sh: line 754: : command not found
/opt/local/retdec/bin/retdec-decompiler.sh: line 184: : command not found
/opt/local/retdec/bin/retdec-decompiler.sh: line 792: : command not found
/opt/local/retdec/bin/retdec-decompiler.sh: line 814: : command not found
/opt/local/retdec/bin/retdec-decompiler.sh: line 815: : command not found
/opt/local/retdec/bin/retdec-decompiler.sh: line 816: : command not found
/opt/local/retdec/bin/retdec-decompiler.sh: line 822: : command not found

##### Decompiling  into .backend.bc...
RUN:  -provider-init -config-path .json -decoder -o .backend.bc
/opt/local/retdec/bin/retdec-decompiler.sh: line 868: : command not found
/opt/local/retdec/bin/retdec-decompiler.sh: line 873: print_error_and_die: command not found

##### Decompiling .backend.bc into ...
RUN:  -target-hll=c -var-renamer=readable -var-name-gen=fruit -var-name-gen-prefix= -call-info-obtainer=optim -arithm-expr-evaluator=c -validate-module -llvmir2bir-converter=orig -o  .backend.bc -enable-debug -emit-debug-comments -config-path=.json
/opt/local/retdec/bin/retdec-decompiler.sh: line 922: : command not found
/opt/local/retdec/bin/retdec-decompiler.sh: line 927: print_error_and_die: command not found
sed: can't read : No such file or directory
sed: can't read : No such file or directory

##### Done!

Run from same directory as install directly from current shell:

user@host:~$ ./retdec-decompiler.sh /tmp/test
##### Checking if file is a Mach-O Universal static library...
RUN: /opt/local/retdec/bin/retdec-macho-extractor --list /private/tmp/test

##### Checking if file is an archive...
RUN: /opt/local/retdec/bin/retdec-ar-extractor --arch-magic /private/tmp/test
Not an archive, going to the next step.
./retdec-decompiler.sh: line 967: unexpected EOF while looking for matching `"'
./retdec-decompiler.sh: line 968: syntax error: unexpected end of file

Run from install directory of retdec with a dedicate bash instance:

user@host:~$ cd /opt/local/retdec/bin
user@host:/opt/local/retdec/bin/$ bash ./retdec-decompiler.sh /tmp/test
...
Running phase: emission of the target code [c] ( 0.03s )
Running phase: finalization ( 0.03s )
Running phase: cleanup ( 0.04s )

##### Done!

[s3rvac]

Thank you for the very well-written report and your PR #259, which fixes the issue!

[AshKetch-um]

Happy to help - thanks for the quick response!