Merge branch 'master' into wget-s3-fix

CHANGELOG.md

@@ -0,0 +1,12 @@
+# Changelog
+
+# dev
+
+* Fix: Remove duplicate `lib` prefix when installing [libdwarf](https://github.com/avast-tl/libdwarf) libraries (#31).
+* Fix: When installing the decompiler, do not remove the entire `share` directory (#12).
+* Fix: Improve OS type detection when installing the decompiler.
+* Fix: Remove useless OS type detection when running decompilations (#10).
+
+# v3.0 (2017-12-13)
+
+Initial public release.

README.md

@@ -20,24 +20,28 @@ Features:
 * Output in two high-level languages: C and a Python-like language.
 * Generation of call graphs, control-flow graphs, and various statistics.
 
+For more information, you can check out [our Botconf 2017 talk](https://retdec.com/web/files/publications/retdec-slides-botconf-2017.pdf) or [our publications](https://retdec.com/publications/).
+
 ## Installation and Use
 
-Currently, we support only Windows and Linux.
+Currently, we support only Windows (7 or later) and Linux.
+
+**Warning: Decompilations of larger binaries (1 MB or more) may require a lot of RAM. When running decompilations, we advise you to limit the maximal virtual memory for processes before decompiling to prevent potential swapping and unresponsiveness. On Linux, you can run e.g. `ulimit -Sv 9863168` in your shell to limit the maximal virtual memory to 8 GB.**
 
 ### Windows
 
-1. Either download and unpack a pre-built package, or build and install the decompiler by yourself (the process is described below):
+1. Either download and unpack a pre-built package from the following list, or build and install the decompiler by yourself (the process is described below):
 
-    * 32b Windows: TBA
-    * 64b Windows: TBA
+    * [32b Windows](https://github.com/avast-tl/retdec/releases/download/v3.0/retdec-v3.0-windows-32b.zip) (v3.0)
+    * [64b Windows](https://github.com/avast-tl/retdec/releases/download/v3.0/retdec-v3.0-windows-64b.zip) (v3.0)
 
 2. Install [Microsoft Visual C++ Redistributable for Visual Studio 2015](https://www.microsoft.com/en-us/download/details.aspx?id=48145).
 
 3. Install [MSYS2](http://www.msys2.org/) and other needed applications by following RetDec's [Windows environment setup guide](https://github.com/avast-tl/retdec/wiki/Windows-Environment).
 
 3. Now, you are all set to run the decompiler. To decompile a binary file named `test.exe`, go into `$RETDEC_INSTALLED_DIR/bin` and run:
 
-    ```
+    ```sh
     bash decompile.sh test.exe
     ```
 
@@ -49,14 +53,14 @@ Currently, we support only Windows and Linux.
 
 2. After you have built the decompiler, you will need to install the following packages via your distribution's package manager:
 
-    * [GNU Bash](https://www.gnu.org/software/bash/)
+    * [Bash](https://www.gnu.org/software/bash/)
     * [UPX](https://upx.github.io/)
     * [bc](https://www.gnu.org/software/bc/)
     * [Graphviz](http://www.graphviz.org/)
 
 3. Now, you are all set to run the decompiler. To decompile a binary file named `test.exe`, go into `$RETDEC_INSTALLED_DIR/bin` and run:
 
-    ```
+    ```sh
     ./decompile.sh test.exe
     ```
 
@@ -68,18 +72,45 @@ This section describes a manual build and installation of RetDec.
 
 ### Requirements
 
-* Linux:
-  * A compiler supporting C++14
-  * [CMake](https://cmake.org/) (version >= 3.6)
-  * [Perl](https://www.perl.org/)
-  * standard tools (e.g. `sh`, `wget`, `sha256sum`)
-* Windows:
-  * Microsoft Visual C++ (version >= Visual Studio 2015 Update 2)
-  * [MSYS2](http://www.msys2.org/) and some other applications. Follow RetDec's [Windows environment setup guide](https://github.com/avast-tl/retdec/wiki/Windows-Environment) to get everything you need on Windows.
-  * [Active Perl](https://www.activestate.com/activeperl). It needs to be the first Perl in `PATH`, or it has to be provided to CMake using `CMAKE_PROGRAM_PATH` variable, e.g. `-DCMAKE_PROGRAM_PATH=/c/perl/bin`.
+#### Linux
+
+* A C++ compiler and standard C++ library supporting C++14 (e.g. GCC >= 4.9)
+* [CMake](https://cmake.org/) (version >= 3.6)
+* [Git](https://git-scm.com/)
+* [Perl](https://www.perl.org/)
+* [Python](https://www.python.org/) (version >= 3.4)
+* [Bash](https://www.gnu.org/software/bash/)
+* [coreutils](https://www.gnu.org/software/coreutils)
+* [wget](https://www.gnu.org/software/wget/)
+* [libtool](https://www.gnu.org/software/libtool/)
+* [ncurses](http://invisible-island.net/ncurses/) (for `libtinfo`)
+* [zlib](http://zlib.net/)
+* Optional: [Doxygen](http://www.stack.nl/~dimitri/doxygen/) and [Graphviz](http://www.graphviz.org/) for generating API documentation
+
+On Debian-based distributions (e.g. Ubuntu), the required packages can be installed with `apt-get`:
+
+```sh
+sudo apt-get install build-essential cmake git perl python3 bash coreutils wget bc doxygen graphviz upx flex bison zlib1g-dev libtinfo-dev autoconf pkg-config m4 libtool
+```
+
+On RPM-based distributions (e.g. Fedora), the required packages can be installed with `dnf`:
+
+```sh
+sudo dnf install git cmake make gcc gcc-c++ perl python3 bash zlib-devel flex bison m4 coreutils autoconf automake libtool ncurses-devel wget bc doxygen graphviz upx pkg-config
+```
+
+#### Windows
+
+* Microsoft Visual C++ (version >= Visual Studio 2015 Update 2)
+* [Git](https://git-scm.com/)
+* [MSYS2](http://www.msys2.org/) and some other applications. Follow RetDec's [Windows environment setup guide](https://github.com/avast-tl/retdec/wiki/Windows-Environment) to get everything you need on Windows.
+* [Active Perl](https://www.activestate.com/activeperl). It needs to be the first Perl in `PATH`, or it has to be provided to CMake using `CMAKE_PROGRAM_PATH` variable, e.g. `-DCMAKE_PROGRAM_PATH=/c/perl/bin`.
+* [Python](https://www.python.org/) (version >= 3.4)
 
 ### Process
 
+**Warning: Currently, RetDec has to be installed into a clean, dedicated directory. Do NOT install it into `/usr`, `/usr/local`, etc. because our build system is not yet ready for system-wide installations. So, when running `cmake`, always set `-DCMAKE_INSTALL_PREFIX=<path>` to a directory that will be used just by RetDec. For more details, see #12.**
+
 * Recursively clone the repository (it contains submodules):
   * `git clone --recursive https://github.com/avast-tl/retdec`
 * Linux:
@@ -133,7 +164,7 @@ This repository contains the following scripts:
   * `run-unit-test.sh` -- run all tests in the unit test directory.
   * `utils.sh` -- a collection of bash utilities.
 
-## Related repositories
+## Related Repositories
 
 * [RetDec IDA plugin](https://github.com/avast-tl/retdec-idaplugin) -- embeds RetDec into IDA (Interactive Disassembler) and makes its use much easier.
 * [RetDec Regression Tests](https://github.com/avast-tl/retdec-regression-tests-framework) -- provides means to run and create regression tests for RetDec and related tools. This is a must if you plan to contribute to the RetDec project.

cmake/deps-config.cmake

@@ -18,6 +18,9 @@ set(DEMANGLER_TESTS ${DEPS_TESTS} CACHE BOOL "enable demangler tests" FORCE)
 # fileformat
 set(FILEFORMAT_TESTS ${DEPS_TESTS} CACHE BOOL "enable fileformat tests" FORCE)
 
+# libdwarf
+set(LIBDWARF_INSTALL_TO_UNITTESTS ${RETDEC_TESTS} CACHE BOOL "enable libdwarf installation to unit tests directory" FORCE)
+
 # retdec-config
 set(RETDEC_CONFIG_TOOLS ON CACHE BOOL "enable retdec-config tools" FORCE)
 set(RETDEC_CONFIG_TESTS ${DEPS_TESTS} CACHE BOOL "enable retdec-config tests" FORCE)

cmake/install-share.sh

@@ -3,101 +3,106 @@
 # Get RetDec share directory.
 #
 
+# Check arguments.
+if [ "$#" -ne 1 ]; then
+	echo "ERROR: Unexpected number of arguments."
+	exit 1
+fi
+
 ###############################################################################
 
 VERSION_FILE_NAME="version.txt"
 ARCH_SUFFIX="tar.xz"
 
-SHA256SUM_REF="6376af57a77147f1363896963d8c1b3745ddb9a6bcec83d63a5846c3f78aeef9"
-VERSION="2017-12-12"
+SHA256SUM_REF="4ecbe72d38d6e95f057c5d18c79e25b5340265416687368ce7e5fd1ebb744b32"
+VERSION="2017-12-15"
 
 ###############################################################################
 
 ARCH_NAME="retdec-support"_"$VERSION.$ARCH_SUFFIX"
 
-cleanup()
-{
-    rm -f "$INSTALL_PATH/$ARCH_NAME"
-    rm -rf "$SHARE_DIR"
-}
-
-# Check arguments.
-if [ "$#" -ne 1 ]; then
-    echo "ERROR: Unexpected number of arguments."
-    exit 1
-fi
-
 # Get install path from script options.
 INSTALL_PATH="$1"
 # Convert from Windows to Unix path on Windows.
-if [[ $(uname -s) == *MINGW* ]] || [[ $(uname -s) == *MSYS* ]]; then
-	INSTALL_PATH="$(sed -e 's/\\/\//g' -e 's/://' <<< "/$INSTALL_PATH")"
-fi
+case "$(uname -s)" in
+	*Windows*|*CYGWIN*|*MINGW*|*MSYS*)
+		INSTALL_PATH="$(sed -e 's/\\/\//g' -e 's/://' <<< "/$INSTALL_PATH")"
+		;;
+esac
 
 SHARE_DIR="$INSTALL_PATH/share"
+SHARE_RETDEC_DIR="$SHARE_DIR/retdec"
+SUPPORT_DIR="$SHARE_RETDEC_DIR/support"
+
+###############################################################################
+
+cleanup()
+{
+	rm -rf "$SUPPORT_DIR"
+}
 
 # Share directory exists.
-if [ -d "$SHARE_DIR" ]; then
-    # Version file exists.
-    if [ -f "$SHARE_DIR/$VERSION_FILE_NAME" ]; then
-        VERSION_FROM_FILE=$(cat "$SHARE_DIR/$VERSION_FILE_NAME")
-        # Version is ok.
-        if [ "$VERSION" = "$VERSION_FROM_FILE" ]; then
-            echo "$SHARE_DIR already exists, version is ok"
-            exit
-        else
-            echo "versions is not as expected -> replace with expected version"
-        fi
-    fi
-
-    rm -rf "$SHARE_DIR"
+if [ -d "$SUPPORT_DIR" ]; then
+	# Version file exists.
+	if [ -f "$SUPPORT_DIR/$VERSION_FILE_NAME" ]; then
+		VERSION_FROM_FILE=$(cat "$SUPPORT_DIR/$VERSION_FILE_NAME")
+		# Version is ok.
+		if [ "$VERSION" = "$VERSION_FROM_FILE" ]; then
+			echo "$SUPPORT_DIR already exists, version is ok"
+			exit
+		else
+			echo "versions is not as expected -> replace with expected version"
+		fi
+	fi
+
+	cleanup
 fi
 
 # Make sure destination directory exists.
-mkdir -p "$INSTALL_PATH"
+mkdir -p "$SUPPORT_DIR"
 
 # Get archive using wget.
-WGET_PARAMS=("https://github.com/avast-tl/retdec-support/releases/download/$VERSION/$ARCH_NAME" --read-timeout=10 -O "$INSTALL_PATH/$ARCH_NAME")
+WGET_PARAMS=(--no-verbose "https://github.com/avast-tl/retdec-support/releases/download/$VERSION/$ARCH_NAME" --read-timeout=10 -O "$SUPPORT_DIR/$ARCH_NAME")
 echo "RUN: wget ${WGET_PARAMS[@]}"
 wget "${WGET_PARAMS[@]}"
 WGET_RC=$?
 if [ "$WGET_RC" -ne 0 ]; then
-    echo "ERROR: wget failed"
-    cleanup
-    exit 1
+	echo "ERROR: wget failed"
+	cleanup
+	exit 1
 fi
 
 # Compute hash of the downloaded archive.
-SHA256SUM_PARAMS=("$INSTALL_PATH/$ARCH_NAME")
+SHA256SUM_PARAMS=("$SUPPORT_DIR/$ARCH_NAME")
 echo "RUN: sha256sum ${SHA256SUM_PARAMS[@]}"
 SHA256SUM=$(sha256sum "${SHA256SUM_PARAMS[@]}" | cut -d' ' -f1)
 SHA256SUM_RC=$?
 if [ "$SHA256SUM_RC" -ne 0 ]; then
-    echo "ERROR: sha256sum failed"
-    cleanup
-    exit 1
+	echo "ERROR: sha256sum failed"
+	cleanup
+	exit 1
 fi
 
 # Check that hash is ok.
 if [ "$SHA256SUM" != "$SHA256SUM_REF" ]; then
-    echo "ERROR: hash check failed"
-    cleanup
-    exit 1
+	echo "ERROR: hash check failed"
+	cleanup
+	exit 1
 fi
 
 # Unpack archive.
-UNPACK_PARAMS=("$INSTALL_PATH/$ARCH_NAME" "--directory=$INSTALL_PATH")
+UNPACK_PARAMS=("$SUPPORT_DIR/$ARCH_NAME" "--directory=$SUPPORT_DIR")
 echo "RUN: tar xf ${UNPACK_PARAMS[@]}"
 tar xf "${UNPACK_PARAMS[@]}" &> /dev/null
 UNPACK_RC=$?
 if [ "$UNPACK_RC" -ne 0 ]; then
-    echo "ERROR: unpacking failed"
-    cleanup
-    exit 1
+	echo "ERROR: unpacking failed"
+	cleanup
+	exit 1
 fi
 
 # Remove archive.
-rm -f "$INSTALL_PATH/$ARCH_NAME"
+rm -f "$SUPPORT_DIR/$ARCH_NAME"
 
-echo "RetDec share directory downloaded OK"
+echo "RetDec support directory downloaded OK"
 exit

doc/CMakeLists.txt

@@ -31,4 +31,4 @@ add_custom_target(doc ALL
 set_directory_properties(PROPERTIES ADDITIONAL_MAKE_CLEAN_FILES "${DOXYGEN_OUTPUT_DIR}")
 
 # Install.
-install(DIRECTORY "${DOXYGEN_OUTPUT_DIR}" DESTINATION doc)
+install(DIRECTORY "${DOXYGEN_OUTPUT_DIR}" DESTINATION share/retdec/doc)

scripts/config.sh

@@ -9,29 +9,6 @@ if [ -z "$DECOMPILER_CONFIG" ]; then
 	DECOMPILER_CONFIG="./config.sh"
 fi
 
-#
-# Detect OS type (win / linux / linux64)
-#
-get_SYS()
-{
-	if [[ $(uname -s) == *Linux* ]]; then
-		echo "linux"
-	elif [[ $(uname -s) == *MINGW* ]] || [[ $(uname -s) == *MSYS* ]]; then
-		echo "win"
-	else
-		echo "unknown"
-	fi
-}
-
-##
-## Settings.
-##
-SYS="$(get_SYS)"
-if [ "$SYS" != "win" -a "$SYS" != "linux" ]; then
-	echo "Error: Unknown OS" >&2
-	exit 1
-fi
-
 ##
 ## Paths (everything has to be without the ending slash '/').
 ##
@@ -40,18 +17,19 @@ fi
 INSTALL_BIN_DIR="$SCRIPTPATH"
 UNIT_TESTS_DIR="$INSTALL_BIN_DIR/unit-tests"
 UNPACKER_PLUGINS_DIR="$INSTALL_BIN_DIR/unpacker-plugins"
-INSTALL_SHARE_DIR="$INSTALL_BIN_DIR/../share"
-INSTALL_SHARE_YARA_DIR="$INSTALL_SHARE_DIR/generic/yara_patterns"
+INSTALL_SHARE_DIR="$INSTALL_BIN_DIR/../share/retdec"
+INSTALL_SUPPORT_DIR="$INSTALL_SHARE_DIR/support"
+INSTALL_SHARE_YARA_DIR="$INSTALL_SUPPORT_DIR/generic/yara_patterns"
 
 ## generic configuration
-GENERIC_TYPES_DIR="$INSTALL_SHARE_DIR/generic/types"
-GENERIC_SIGNATURES_DIR="$INSTALL_SHARE_DIR/generic/yara_patterns/static-code"
+GENERIC_TYPES_DIR="$INSTALL_SUPPORT_DIR/generic/types"
+GENERIC_SIGNATURES_DIR="$INSTALL_SHARE_YARA_DIR/static-code"
 
 ## ARM-specific configuration
-ARM_ORDS_DIR="$INSTALL_SHARE_DIR/arm/ords"
+ARM_ORDS_DIR="$INSTALL_SUPPORT_DIR/arm/ords"
 
 ## X86-specific configuration
-X86_ORDS_DIR="$INSTALL_SHARE_DIR/x86/ords"
+X86_ORDS_DIR="$INSTALL_SUPPORT_DIR/x86/ords"
 
 ## BIN2LLVMIR parameters
 # The following list of passes is -O3