Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build is failing on macOS (FLEX_INCLUDE_DIR is set to NOTFOUND) #103

Closed
ThauEx opened this issue Jan 25, 2018 · 23 comments · Fixed by #680
Closed

Build is failing on macOS (FLEX_INCLUDE_DIR is set to NOTFOUND) #103

ThauEx opened this issue Jan 25, 2018 · 23 comments · Fixed by #680

Comments

@ThauEx
Copy link

ThauEx commented Jan 25, 2018

Hey,
when I try to bo build retdec, it fails during the compilation of yaramod

build/external/src/yaramod-project-stamp/yaramod-project-configure-err.log:

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
/Users/thau/tmp/retdec/build/external/src/yaramod-project/src/FLEX_INCLUDE_DIR
   used as include directory in directory /Users/thau/tmp/retdec/build/external/src/yaramod-project/src

build/external/src/yaramod-project-stamp/yaramod-project-configure-out.log

-- The CXX compiler identification is AppleClang 9.0.0.9000039
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found FLEX: /Users/thau/.local/opt/flex/bin/flex (found version "2.6.4")
-- Found BISON: /Users/thau/.local/opt/bison/bin/bison (found version "3.0.4")
-- Configuring incomplete, errors occurred!
See also "/Users/thau/tmp/retdec/build/external/src/yaramod-project-build/CMakeFiles/CMakeOutput.log".

Console output:

Scanning dependencies of target yaramod-project
[  8%] Creating directories for 'yaramod-project'
[  8%] Performing download step (download, verify and extract) for 'yaramod-project'
-- yaramod-project download command succeeded.  See also /Users/thau/tmp/retdec/build/external/src/yaramod-project-stamp/yaramod-project-download-*.log
[  8%] No patch step for 'yaramod-project'
[  8%] No update step for 'yaramod-project'
[  8%] Performing configure step for 'yaramod-project'
CMake Error at /Users/thau/tmp/retdec/build/external/src/yaramod-project-stamp/yaramod-project-configure-Release.cmake:16 (message):
  Command failed: 1

   '/Users/thau/.local/Cellar/cmake/3.10.1/bin/cmake' '-DCMAKE_BUILD_TYPE=Release' '-DCMAKE_C_COMPILER=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc' '-DCMAKE_CXX_COMPILER=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++' '-GUnix Makefiles' '/Users/kmadel/tmp/retdec/build/external/src/yaramod-project'

  See also

    /Users/thau/tmp/retdec/build/external/src/yaramod-project-stamp/yaramod-project-configure-*.log


make[2]: *** [external/src/yaramod-project-stamp/yaramod-project-configure] Error 1
make[1]: *** [deps/yaramod/CMakeFiles/yaramod-project.dir/all] Error 2
make: *** [all] Error 2
@s3rvac
Copy link
Member

s3rvac commented Jan 25, 2018

Hi. Did you set the following variables before running cmake, as suggested in our build instructions for macOS in the README?

# Apple ships old Flex & Bison, so Homebrew versions should be used.
export CMAKE_INCLUDE_PATH="/usr/local/opt/flex/include"
export CMAKE_LIBRARY_PATH="/usr/local/opt/flex/lib;/usr/local/opt/bison/lib"
export PATH="/usr/local/opt/flex/bin:/usr/local/opt/bison/bin:$PATH"

Of course, if you have installed Flex and Bison to a different directory (your output suggests that), you will have to adjust the paths.

@ThauEx
Copy link
Author

ThauEx commented Jan 25, 2018

Well I did, see second log file...

$ echo $CMAKE_INCLUDE_PATH
/Users/thau/.local/opt/flex/include;/Users/thau/.local/opt/ncurses/include
$ echo $CMAKE_LIBRARY_PATH
/Users/thau/.local/opt/flex/lib;/Users/thau/.local/opt/bison/lib;/Users/thau/.local/opt/ncurses/lib
$ echo $PATH
/Users/thau/.local/opt/flex/bin:/Users/thau/.local/opt/bison/bin:/Users/thau/.local/opt/coreutils/libexec/gnubin:/Users/thau/.local/sbin:/Users/thau/.local/bin::/usr/local/sbin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:/opt/X11/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands::/Users/thau/.local/Caskroom/android-sdk/25.2.3/platform-tools/

@s3rvac
Copy link
Member

s3rvac commented Jan 25, 2018

Can you please verify that the following file exists?

/Users/thau/.local/opt/flex/include/FlexLexer.h

This is the file that CMake seems to be unable to find (as per the output from yaramod-project-configure-err.log).

@ThauEx
Copy link
Author

ThauEx commented Jan 25, 2018

it does:

#ifndef __FLEX_LEXER_H
// Never included before - need to define base class.
#define __FLEX_LEXER_H

#include <iostream>

extern "C++" {

struct yy_buffer_state;
typedef int yy_state_type;

class FlexLexer
{
public:
  virtual ~FlexLexer()        { }

  const char* YYText() const  { return yytext; }
  int YYLeng()        const   { return yyleng; }

  virtual void
[...]

@s3rvac s3rvac changed the title Build is failing on macOS 10.13.2 Build is failing on macOS 10.13.2 (FLEX_INCLUDE_DIR is set to NOTFOUND) Jan 25, 2018
@s3rvac
Copy link
Member

s3rvac commented Jan 25, 2018

I stumbled upon this issue, where the user appears to have the same problem. I am not sure about the applicability of the provided solution though as yaramod is built via an external project (not directly), so it will be tricky to pass FLEX_INCLUDE_DIR to it.

@bdemick
Copy link

bdemick commented Feb 1, 2018

I was getting the same "FLEX_INCLUDE_DIR is set to NOTFOUND" error (but that ended up to be user error), but I'm now getting this error:

parser/yy/parser.y:7.1-5: invalid directive: `%code'
parser/yy/parser.y:7.7-14: syntax error, unexpected identifier
make[5]: *** [src/yy_parser.cpp] Error 1
make[4]: *** [src/CMakeFiles/yaramod.dir/all] Error 2
make[3]: *** [all] Error 2

Seems kind of related (I'm on macOS 10.13.3), but if it needs to be reported as a separate issue, I'll do so.

@s3rvac
Copy link
Member

s3rvac commented Feb 2, 2018

@bdemick Thank you for the report.

I was getting the same "FLEX_INCLUDE_DIR is set to NOTFOUND" error (but that ended up to be user error)

Could you please share with us how you solved it? Maybe it might help other people as well.

but I'm now getting this error:
parser/yy/parser.y:7.1-5: invalid directive: `%code'

I believe this might be caused by an old version of Bison. Have you installed newer versions of Flex and Bison from Homebrew and run the following commands before build (see the instructions in our README)?

# Apple ships old Flex & Bison, so Homebrew versions should be used.
export CMAKE_INCLUDE_PATH="/usr/local/opt/flex/include"
export CMAKE_LIBRARY_PATH="/usr/local/opt/flex/lib;/usr/local/opt/bison/lib"
export PATH="/usr/local/opt/flex/bin:/usr/local/opt/bison/bin:$PATH"

@bdemick
Copy link

bdemick commented Feb 2, 2018

@s3rvac The FLEX_INCLUDE_DIR error was solved by the Apple-specific environment variables suggested in the readme. I had just restarted my terminal between build attempts and forgotten to re-set them after debugging another issue.

So theoretically, if I got past the original include directory notfound issue with the environment variables configured properly, the one you're suggesting I currently have would be the same root cause. I'm currently getting by with the docker image so I stopped troubleshooting, but I'll give it another go today and see if I can confirm it's using the Homebrew packages.

@s3rvac
Copy link
Member

s3rvac commented Mar 6, 2018

I am closing this issue as we are unfortunately unable to reproduce the problem. If anyone who reads this is able to reproduce the problem, please post a comment and we will open the issue.

@s3rvac s3rvac closed this as completed Mar 6, 2018
@ghost
Copy link

ghost commented Jun 25, 2018

I have meet this problem too...
parser/yy/parser.y:7.1-5: invalid directive: `%code'
parser/yy/parser.y:7.7-14: syntax error, unexpected identifier
make[5]: *** [src/yaramod/yy/yy_parser.cpp] Error 1
make[4]: *** [src/CMakeFiles/yaramod.dir/all] Error 2
make[3]: *** [all] Error 2

@ghost
Copy link

ghost commented Jun 25, 2018

OK I know what happened....
When I clone the project into /usr/local/opt and changed -DCMAKE_INSTALL_PREFIX into /usr/local/opt/retdec
I get it installed with no error.....

@s3rvac
Copy link
Member

s3rvac commented Jun 25, 2018

So if I understand it correctly, you got the error only if you ran cmake without a custom value of -DCMAKE_INSTALL_PREFIX?

@ghost
Copy link

ghost commented Jun 25, 2018

nope
I use -DCMAKE_INSTALL_PREFIX=/usr/local/opt/retdec and i clone it in downloads

@s3rvac
Copy link
Member

s3rvac commented Jun 25, 2018

I see. Could you please give us all steps that you have performed to reproduce the invalid directive: %code issue? For example:

git clone https://github.com/avast-tl/retdec
cd retdec
mkdir build
...

@ghost
Copy link

ghost commented Jun 26, 2018

cd /usr/local/opt
git clone https://github.com/avast-tl/retdec
mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/opt/retdec
make -j4
make install

@s3rvac
Copy link
Member

s3rvac commented Jun 26, 2018

@Zzm317 Are you sure that the above steps lead to the invalid directive: %code issue? Because as the last step, you have listed make install, but the error should appear during make -j4, so you should not be able to even get to make install. Can you please check that you have given us steps to reproduce the problem?

@ghost
Copy link

ghost commented Jun 26, 2018

After i clone it into /usr/local/opt
I passed cmake and make with no errors
Zzm317-MacBook-Pro:build zzm317$ sudo make install
[ 0%] Built target capstone-project
[ 1%] Built target jsoncpp-project
[ 2%] Built target llvm-project
[ 3%] Built target openssl
[ 4%] Built target rapidjson-project
[ 5%] Built target tinyxml2-project
[ 6%] Built target elfio-project
[ 7%] Built target libdwarf-project
[ 8%] Built target pelib-project
[ 9%] Built target yaracpp-project
[ 10%] Built target yaramod-project
[ 11%] Built target whereami
[ 12%] Built target retdec-utils
[ 12%] Built target retdec-ar-extractor
[ 13%] Built target retdec-ar-extractortool
[ 14%] Built target retdec-capstone2llvmir
[ 16%] Built target retdec-config
[ 16%] Built target retdec-crypto
[ 20%] Built target retdec-ctypes
[ 20%] Built target retdec-ctypesparser
[ 21%] Built target retdec-demangler
[ 28%] Built target retdec-fileformat
[ 30%] Built target retdec-dwarfparser
[ 30%] Built target retdec-pdbparser
[ 30%] Built target retdec-debugformat
[ 30%] Built target retdec-llvm-support
[ 32%] Built target retdec-loader
[ 33%] Built target retdec-rtti-finder
[ 33%] Built target retdec-stacofin
[ 42%] Built target retdec-bin2llvmir
[ 42%] Built target retdec-bin2llvmirtool
[ 43%] Built target retdec-patterngen
[ 44%] Built target retdec-bin2pat
[ 44%] Built target retdec-configtool
[ 47%] Built target retdec-cpdetect
[ 59%] Built target retdec-fileinfo
[ 59%] Built target retdec-idr2pat
[ 95%] Built target retdec-llvmir2hll
[ 95%] Built target retdec-llvmir2hlltool
[ 95%] Built target retdec-macho-extractor
[ 96%] Built target retdec-macho-extractortool
[ 96%] Built target retdec-pat2yara
[ 96%] Built target retdec-stacofintool
[ 97%] Built target retdec-unpacker
[ 97%] Built target retdec-unpacker-mpress
[ 98%] Built target retdec-unpacker-upx
[ 99%] Built target retdec-unpackertool
[100%] Built target retdec-unpacker-example
[100%] Built target retdec-getsig
Install the project...
-- Install configuration: "Release"
/usr/local/retdec/share/retdec/support already exists, version is ok
-- Up-to-date: /usr/local/retdec/lib
-- Up-to-date: /usr/local/retdec/lib/libretdec-libelf.dylib
-- Up-to-date: /usr/local/retdec/lib/libretdec-libdwarf.dylib
compiling yara signatures...
signatures compiled successfully
-- Up-to-date: /usr/local/retdec/bin/retdec-color-c.py
-- Up-to-date: /usr/local/retdec/bin/retdec-config.sh
-- Up-to-date: /usr/local/retdec/bin/retdec-archive-decompiler.sh
-- Up-to-date: /usr/local/retdec/bin/retdec-decompiler.sh
-- Up-to-date: /usr/local/retdec/bin/retdec-fileinfo.sh
-- Up-to-date: /usr/local/retdec/bin/retdec-signature-from-library-creator.sh
-- Up-to-date: /usr/local/retdec/bin/retdec-unpacker.sh
-- Up-to-date: /usr/local/retdec/bin/retdec-utils.sh
-- Installing: /usr/local/retdec/bin/retdec-ar-extractor
-- Installing: /usr/local/retdec/bin/retdec-bin2llvmir
-- Installing: /usr/local/retdec/bin/retdec-bin2pat
-- Installing: /usr/local/retdec/bin/retdec-config
-- Installing: /usr/local/retdec/bin/retdec-fileinfo
-- Installing: /usr/local/retdec/bin/retdec-idr2pat
-- Installing: /usr/local/retdec/bin/retdec-llvmir2hll
-- Installing: /usr/local/retdec/bin/retdec-macho-extractor
-- Installing: /usr/local/retdec/bin/retdec-pat2yara
-- Installing: /usr/local/retdec/bin/retdec-stacofin
-- Installing: /usr/local/retdec/bin/retdec-unpacker
-- Installing: /usr/local/retdec/bin/retdec-getsig

@s3rvac
Copy link
Member

s3rvac commented Jun 26, 2018

Yes, but we are interested in the case when you got the invalid directive: %code error. That is, we need a list of steps that you have originally performed that led to the invalid directive: %code error.

Do not get me wrong - we are happy that you got the build working, but to be able to solve the problem reported in this issue, we need a list of steps to reproduce the problem. Otherwise, we will have to close it as cannot be reproduced.

@ghost
Copy link

ghost commented Jun 26, 2018

Zzm317-MacBook-Pro:~ zzm317$ cd downloads
Zzm317-MacBook-Pro:downloads zzm317$ cd retdec
Zzm317-MacBook-Pro:retdec zzm317$ cd build
Zzm317-MacBook-Pro:build zzm317$ ls
CMakeCache.txt cmake_install.cmake scripts
CMakeFiles deps src
Makefile external
Zzm317-MacBook-Pro:build zzm317$ make install
[ 0%] Built target capstone-project
[ 1%] Built target jsoncpp-project
[ 2%] Built target llvm-project
[ 3%] Built target openssl
[ 4%] Built target rapidjson-project
[ 5%] Built target tinyxml2-project
[ 6%] Built target elfio-project
[ 7%] Built target libdwarf-project
[ 8%] Built target pelib-project
[ 9%] Built target yaracpp-project
[ 10%] Performing build step for 'yaramod-project'
CMake Error at /Users/zzm317/downloads/retdec/build/external/src/yaramod-project-stamp/yaramod-project-build-Release.cmake:16 (message):
Command failed: 2

'/Applications/Xcode.app/Contents/Developer/usr/bin/make'

See also

/Users/zzm317/downloads/retdec/build/external/src/yaramod-project-stamp/yaramod-project-build-*.log

make[2]: *** [external/src/yaramod-project-stamp/yaramod-project-build] Error 1
make[1]: *** [deps/yaramod/CMakeFiles/yaramod-project.dir/all] Error 2
make: *** [all] Error 2
Zzm317-MacBook-Pro:build zzm317$ make clean

@s3rvac s3rvac changed the title Build is failing on macOS 10.13.2 (FLEX_INCLUDE_DIR is set to NOTFOUND) Build is failing on macOS (FLEX_INCLUDE_DIR is set to NOTFOUND) Aug 24, 2018
@danoe
Copy link

danoe commented Aug 30, 2018

Had a similar issue here, think i had flex and bison not installed from brew before so it used system versions. So i installed them using brew, wiped the build directory again, made sure those variables were set:

# Apple ships old Flex & Bison, so Homebrew versions should be used.
export CMAKE_INCLUDE_PATH="/usr/local/opt/flex/include"
export CMAKE_LIBRARY_PATH="/usr/local/opt/flex/lib;/usr/local/opt/bison/lib"
export PATH="/usr/local/opt/flex/bin:/usr/local/opt/bison/bin:$PATH"

Then used the steps to build it again:

mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/opt/retdec
make -j8
make install

No more errors, everything looks good.

@avast avast deleted a comment from PavelKotov1 Nov 2, 2018
@avast avast deleted a comment from PavelKotov1 Nov 2, 2018
@slw07g
Copy link

slw07g commented Feb 16, 2019

Experiencing the same issue on Mojave. The suggested fix does not work for me.

@slw07g
Copy link

slw07g commented Feb 18, 2019

The issue was resolved for me by removing the yaramod-project-* directories and rebuilding.

@PeterMatula
Copy link
Collaborator

Yaramod (adn therefore RetDec) no longer needs Flex and Bison.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants