Build is failing on macOS (FLEX_INCLUDE_DIR is set to NOTFOUND)

[ThauEx]

Hey,
when I try to bo build retdec, it fails during the compilation of yaramod

build/external/src/yaramod-project-stamp/yaramod-project-configure-err.log:

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
/Users/thau/tmp/retdec/build/external/src/yaramod-project/src/FLEX_INCLUDE_DIR
   used as include directory in directory /Users/thau/tmp/retdec/build/external/src/yaramod-project/src

build/external/src/yaramod-project-stamp/yaramod-project-configure-out.log

-- The CXX compiler identification is AppleClang 9.0.0.9000039
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found FLEX: /Users/thau/.local/opt/flex/bin/flex (found version "2.6.4")
-- Found BISON: /Users/thau/.local/opt/bison/bin/bison (found version "3.0.4")
-- Configuring incomplete, errors occurred!
See also "/Users/thau/tmp/retdec/build/external/src/yaramod-project-build/CMakeFiles/CMakeOutput.log".

Console output:

Scanning dependencies of target yaramod-project
[  8%] Creating directories for 'yaramod-project'
[  8%] Performing download step (download, verify and extract) for 'yaramod-project'
-- yaramod-project download command succeeded.  See also /Users/thau/tmp/retdec/build/external/src/yaramod-project-stamp/yaramod-project-download-*.log
[  8%] No patch step for 'yaramod-project'
[  8%] No update step for 'yaramod-project'
[  8%] Performing configure step for 'yaramod-project'
CMake Error at /Users/thau/tmp/retdec/build/external/src/yaramod-project-stamp/yaramod-project-configure-Release.cmake:16 (message):
  Command failed: 1

   '/Users/thau/.local/Cellar/cmake/3.10.1/bin/cmake' '-DCMAKE_BUILD_TYPE=Release' '-DCMAKE_C_COMPILER=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc' '-DCMAKE_CXX_COMPILER=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++' '-GUnix Makefiles' '/Users/kmadel/tmp/retdec/build/external/src/yaramod-project'

  See also

    /Users/thau/tmp/retdec/build/external/src/yaramod-project-stamp/yaramod-project-configure-*.log


make[2]: *** [external/src/yaramod-project-stamp/yaramod-project-configure] Error 1
make[1]: *** [deps/yaramod/CMakeFiles/yaramod-project.dir/all] Error 2
make: *** [all] Error 2

[s3rvac]

Hi. Did you set the following variables before running cmake, as suggested in our build instructions for macOS in the README?

# Apple ships old Flex & Bison, so Homebrew versions should be used.
export CMAKE_INCLUDE_PATH="/usr/local/opt/flex/include"
export CMAKE_LIBRARY_PATH="/usr/local/opt/flex/lib;/usr/local/opt/bison/lib"
export PATH="/usr/local/opt/flex/bin:/usr/local/opt/bison/bin:$PATH"

Of course, if you have installed Flex and Bison to a different directory (your output suggests that), you will have to adjust the paths.

[ThauEx]

Well I did, see second log file...

$ echo $CMAKE_INCLUDE_PATH
/Users/thau/.local/opt/flex/include;/Users/thau/.local/opt/ncurses/include

$ echo $CMAKE_LIBRARY_PATH
/Users/thau/.local/opt/flex/lib;/Users/thau/.local/opt/bison/lib;/Users/thau/.local/opt/ncurses/lib

$ echo $PATH
/Users/thau/.local/opt/flex/bin:/Users/thau/.local/opt/bison/bin:/Users/thau/.local/opt/coreutils/libexec/gnubin:/Users/thau/.local/sbin:/Users/thau/.local/bin::/usr/local/sbin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:/opt/X11/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands::/Users/thau/.local/Caskroom/android-sdk/25.2.3/platform-tools/

[s3rvac]

Can you please verify that the following file exists?

/Users/thau/.local/opt/flex/include/FlexLexer.h

This is the file that CMake seems to be unable to find (as per the output from yaramod-project-configure-err.log).

[ThauEx]

it does:

#ifndef __FLEX_LEXER_H
// Never included before - need to define base class.
#define __FLEX_LEXER_H

#include <iostream>

extern "C++" {

struct yy_buffer_state;
typedef int yy_state_type;

class FlexLexer
{
public:
  virtual ~FlexLexer()        { }

  const char* YYText() const  { return yytext; }
  int YYLeng()        const   { return yyleng; }

  virtual void
[...]

[s3rvac]

I stumbled upon this issue, where the user appears to have the same problem. I am not sure about the applicability of the provided solution though as yaramod is built via an external project (not directly), so it will be tricky to pass FLEX_INCLUDE_DIR to it.

[bdemick]

I was getting the same "FLEX_INCLUDE_DIR is set to NOTFOUND" error (but that ended up to be user error), but I'm now getting this error:

parser/yy/parser.y:7.1-5: invalid directive: `%code'
parser/yy/parser.y:7.7-14: syntax error, unexpected identifier
make[5]: *** [src/yy_parser.cpp] Error 1
make[4]: *** [src/CMakeFiles/yaramod.dir/all] Error 2
make[3]: *** [all] Error 2

Seems kind of related (I'm on macOS 10.13.3), but if it needs to be reported as a separate issue, I'll do so.

[s3rvac]

@bdemick Thank you for the report.

I was getting the same "FLEX_INCLUDE_DIR is set to NOTFOUND" error (but that ended up to be user error)

Could you please share with us how you solved it? Maybe it might help other people as well.

but I'm now getting this error:
parser/yy/parser.y:7.1-5: invalid directive: `%code'

I believe this might be caused by an old version of Bison. Have you installed newer versions of Flex and Bison from Homebrew and run the following commands before build (see the instructions in our README)?

# Apple ships old Flex & Bison, so Homebrew versions should be used.
export CMAKE_INCLUDE_PATH="/usr/local/opt/flex/include"
export CMAKE_LIBRARY_PATH="/usr/local/opt/flex/lib;/usr/local/opt/bison/lib"
export PATH="/usr/local/opt/flex/bin:/usr/local/opt/bison/bin:$PATH"

[bdemick]

@s3rvac The FLEX_INCLUDE_DIR error was solved by the Apple-specific environment variables suggested in the readme. I had just restarted my terminal between build attempts and forgotten to re-set them after debugging another issue.

So theoretically, if I got past the original include directory notfound issue with the environment variables configured properly, the one you're suggesting I currently have would be the same root cause. I'm currently getting by with the docker image so I stopped troubleshooting, but I'll give it another go today and see if I can confirm it's using the Homebrew packages.

[s3rvac]

I am closing this issue as we are unfortunately unable to reproduce the problem. If anyone who reads this is able to reproduce the problem, please post a comment and we will open the issue.

[ghost]

I have meet this problem too...
parser/yy/parser.y:7.1-5: invalid directive: `%code'
parser/yy/parser.y:7.7-14: syntax error, unexpected identifier
make[5]: *** [src/yaramod/yy/yy_parser.cpp] Error 1
make[4]: *** [src/CMakeFiles/yaramod.dir/all] Error 2
make[3]: *** [all] Error 2

[ghost]

OK I know what happened....
When I clone the project into /usr/local/opt and changed -DCMAKE_INSTALL_PREFIX into /usr/local/opt/retdec
I get it installed with no error.....

[s3rvac]

So if I understand it correctly, you got the error only if you ran cmake without a custom value of -DCMAKE_INSTALL_PREFIX?

[ghost]

nope
I use -DCMAKE_INSTALL_PREFIX=/usr/local/opt/retdec and i clone it in downloads

[s3rvac]

I see. Could you please give us all steps that you have performed to reproduce the invalid directive: %code issue? For example:

git clone https://github.com/avast-tl/retdec
cd retdec
mkdir build
...

[ghost]

cd /usr/local/opt
git clone https://github.com/avast-tl/retdec
mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/opt/retdec
make -j4
make install

[s3rvac]

@Zzm317 Are you sure that the above steps lead to the invalid directive: %code issue? Because as the last step, you have listed make install, but the error should appear during make -j4, so you should not be able to even get to make install. Can you please check that you have given us steps to reproduce the problem?

[ghost]

After i clone it into /usr/local/opt
I passed cmake and make with no errors
Zzm317-MacBook-Pro:build zzm317$ sudo make install
[ 0%] Built target capstone-project
[ 1%] Built target jsoncpp-project
[ 2%] Built target llvm-project
[ 3%] Built target openssl
[ 4%] Built target rapidjson-project
[ 5%] Built target tinyxml2-project
[ 6%] Built target elfio-project
[ 7%] Built target libdwarf-project
[ 8%] Built target pelib-project
[ 9%] Built target yaracpp-project
[ 10%] Built target yaramod-project
[ 11%] Built target whereami
[ 12%] Built target retdec-utils
[ 12%] Built target retdec-ar-extractor
[ 13%] Built target retdec-ar-extractortool
[ 14%] Built target retdec-capstone2llvmir
[ 16%] Built target retdec-config
[ 16%] Built target retdec-crypto
[ 20%] Built target retdec-ctypes
[ 20%] Built target retdec-ctypesparser
[ 21%] Built target retdec-demangler
[ 28%] Built target retdec-fileformat
[ 30%] Built target retdec-dwarfparser
[ 30%] Built target retdec-pdbparser
[ 30%] Built target retdec-debugformat
[ 30%] Built target retdec-llvm-support
[ 32%] Built target retdec-loader
[ 33%] Built target retdec-rtti-finder
[ 33%] Built target retdec-stacofin
[ 42%] Built target retdec-bin2llvmir
[ 42%] Built target retdec-bin2llvmirtool
[ 43%] Built target retdec-patterngen
[ 44%] Built target retdec-bin2pat
[ 44%] Built target retdec-configtool
[ 47%] Built target retdec-cpdetect
[ 59%] Built target retdec-fileinfo
[ 59%] Built target retdec-idr2pat
[ 95%] Built target retdec-llvmir2hll
[ 95%] Built target retdec-llvmir2hlltool
[ 95%] Built target retdec-macho-extractor
[ 96%] Built target retdec-macho-extractortool
[ 96%] Built target retdec-pat2yara
[ 96%] Built target retdec-stacofintool
[ 97%] Built target retdec-unpacker
[ 97%] Built target retdec-unpacker-mpress
[ 98%] Built target retdec-unpacker-upx
[ 99%] Built target retdec-unpackertool
[100%] Built target retdec-unpacker-example
[100%] Built target retdec-getsig
Install the project...
-- Install configuration: "Release"
/usr/local/retdec/share/retdec/support already exists, version is ok
-- Up-to-date: /usr/local/retdec/lib
-- Up-to-date: /usr/local/retdec/lib/libretdec-libelf.dylib
-- Up-to-date: /usr/local/retdec/lib/libretdec-libdwarf.dylib
compiling yara signatures...
signatures compiled successfully
-- Up-to-date: /usr/local/retdec/bin/retdec-color-c.py
-- Up-to-date: /usr/local/retdec/bin/retdec-config.sh
-- Up-to-date: /usr/local/retdec/bin/retdec-archive-decompiler.sh
-- Up-to-date: /usr/local/retdec/bin/retdec-decompiler.sh
-- Up-to-date: /usr/local/retdec/bin/retdec-fileinfo.sh
-- Up-to-date: /usr/local/retdec/bin/retdec-signature-from-library-creator.sh
-- Up-to-date: /usr/local/retdec/bin/retdec-unpacker.sh
-- Up-to-date: /usr/local/retdec/bin/retdec-utils.sh
-- Installing: /usr/local/retdec/bin/retdec-ar-extractor
-- Installing: /usr/local/retdec/bin/retdec-bin2llvmir
-- Installing: /usr/local/retdec/bin/retdec-bin2pat
-- Installing: /usr/local/retdec/bin/retdec-config
-- Installing: /usr/local/retdec/bin/retdec-fileinfo
-- Installing: /usr/local/retdec/bin/retdec-idr2pat
-- Installing: /usr/local/retdec/bin/retdec-llvmir2hll
-- Installing: /usr/local/retdec/bin/retdec-macho-extractor
-- Installing: /usr/local/retdec/bin/retdec-pat2yara
-- Installing: /usr/local/retdec/bin/retdec-stacofin
-- Installing: /usr/local/retdec/bin/retdec-unpacker
-- Installing: /usr/local/retdec/bin/retdec-getsig

[s3rvac]

Yes, but we are interested in the case when you got the invalid directive: %code error. That is, we need a list of steps that you have originally performed that led to the invalid directive: %code error.

Do not get me wrong - we are happy that you got the build working, but to be able to solve the problem reported in this issue, we need a list of steps to reproduce the problem. Otherwise, we will have to close it as cannot be reproduced.

[ghost]

Zzm317-MacBook-Pro:~ zzm317$ cd downloads
Zzm317-MacBook-Pro:downloads zzm317$ cd retdec
Zzm317-MacBook-Pro:retdec zzm317$ cd build
Zzm317-MacBook-Pro:build zzm317$ ls
CMakeCache.txt cmake_install.cmake scripts
CMakeFiles deps src
Makefile external
Zzm317-MacBook-Pro:build zzm317$ make install
[ 0%] Built target capstone-project
[ 1%] Built target jsoncpp-project
[ 2%] Built target llvm-project
[ 3%] Built target openssl
[ 4%] Built target rapidjson-project
[ 5%] Built target tinyxml2-project
[ 6%] Built target elfio-project
[ 7%] Built target libdwarf-project
[ 8%] Built target pelib-project
[ 9%] Built target yaracpp-project
[ 10%] Performing build step for 'yaramod-project'
CMake Error at /Users/zzm317/downloads/retdec/build/external/src/yaramod-project-stamp/yaramod-project-build-Release.cmake:16 (message):
Command failed: 2

'/Applications/Xcode.app/Contents/Developer/usr/bin/make'

See also

/Users/zzm317/downloads/retdec/build/external/src/yaramod-project-stamp/yaramod-project-build-*.log

make[2]: *** [external/src/yaramod-project-stamp/yaramod-project-build] Error 1
make[1]: *** [deps/yaramod/CMakeFiles/yaramod-project.dir/all] Error 2
make: *** [all] Error 2
Zzm317-MacBook-Pro:build zzm317$ make clean

[danoe]

Had a similar issue here, think i had flex and bison not installed from brew before so it used system versions. So i installed them using brew, wiped the build directory again, made sure those variables were set:

# Apple ships old Flex & Bison, so Homebrew versions should be used.
export CMAKE_INCLUDE_PATH="/usr/local/opt/flex/include"
export CMAKE_LIBRARY_PATH="/usr/local/opt/flex/lib;/usr/local/opt/bison/lib"
export PATH="/usr/local/opt/flex/bin:/usr/local/opt/bison/bin:$PATH"

Then used the steps to build it again:

mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/opt/retdec
make -j8
make install

No more errors, everything looks good.

[slw07g]

Experiencing the same issue on Mojave. The suggested fix does not work for me.

[slw07g]

The issue was resolved for me by removing the yaramod-project-* directories and rebuilding.

[PeterMatula]

Yaramod (adn therefore RetDec) no longer needs Flex and Bison.