make fails in latest release if git is not installed

[rofl0r]

"/bin/sh" capstone.sh "https://github.com/aquynh/capstone.git" "next" "3722c74f69ffa5b705d36cd49c19acdf958a9b7a" ""
[capstone] release with no git?
cd capstone ; git reset --hard
/bin/sh: git: not found
make[1]: *** [capstone-sync] Error 127

[rofl0r]

i hope its not trying to DL stuff from the webs behind the users back (supposedly the tarball is meant to be usable offline)

[radare]

this happens if you download the tarball from github, because github is retarded and doesnt knows how to handle git submodules, so the tarball you should use is the one in the homepage.

also, if you dont want to use custom capstone just build with —with-syscapstone and this git step shouldnt happen

On 29 Jan 2016, at 16:50, rofl0r notifications@github.com wrote:

i hope its not trying to DL stuff from the webs behind the users back (supposedly the tarball is meant to be usable offline)

—
Reply to this email directly or view it on GitHub #4054 (comment).

[rofl0r]

i'm using http://www.radare.org/get/radare2-0.10.0.tar.xz

[rofl0r]

if i use —with-syscapstone , is the latest capstone release 3.0.4 sufficient ?

[Maijin]

What about using radare2 from git http://bin.rada.re/radare2-w32-0.10.0-git.zip

[Maijin]

or if you build git clone...

[rofl0r]

i'm trying to package radare2 for my linux distro, whose package system is based on release tarballs.

[rofl0r]

WARNING: Unknown flag '-—with-syscapstone'.

[rofl0r]

the problem are the quotes around "--with-syscapstone" in configure

"--with-syscapstone") WITH_CAPSTONE="1"; ;;
"--with-syszip") USE_ZIP="1"; ;;
"--without-gpl") WITH_GPL="0"; ;;
"--with-openssl") WANT_OPENSSL="1"; ;;

[radare]

????? which shell are you using? quotes in there are totally legit, and works fine in all my machines

On 29 Jan 2016, at 17:19, rofl0r notifications@github.com wrote:

the problem are the quotes around "--with-syscapstone" in configure

"--with-syscapstone") WITH_CAPSTONE="1"; ;;
"--with-syszip") USE_ZIP="1"; ;;
"--without-gpl") WITH_GPL="0"; ;;
"--with-openssl") WANT_OPENSSL="1"; ;;
—
Reply to this email directly or view it on GitHub #4054 (comment).

[rofl0r]

busybox ash, which is posix conformant
it's equivalent to debians dash

[rofl0r]

with a patched configure, and git installed, the build breaks now here

 - LIL_ENDIAN = 1
 - LIBVERSION = 0.10.0
fatal: Not a git repository (or any of the parent directories): .git

or actually no, because it continues.... till here

/src/build/radare2/radare2-0.10.0/libr/..//libr/anal/p/anal_arm_cs.c:593:21: error: 'arm_op_mem' has no member named 'lshift'
/src/build/radare2/radare2-0.10.0/libr/..//libr/anal/p/anal_arm_cs.c:595:19: error: 'arm_op_mem' has no member named 'lshift'

supposedly that's due to radare excepting a newer version of capstone than the latest release.

[radare]

this error is fixed in git, i will release 0.10.1 soon because you are using an old version of capstone installed in your system.

and in fact, there are more abi changes that will break the next release of capstone.. but i was not planning to wait more for it. can you run make dist in the latest r2 code from git and verify if this is still happening?

are those quotes the only problem for your ash to run configure?

On 29 Jan 2016, at 17:46, rofl0r notifications@github.com wrote:

with a patched configure, and git installed, the build breaks now here

• LIL_ENDIAN = 1
• LIBVERSION = 0.10.0
  fatal: Not a git repository (or any of the parent directories): .git
  or actually no, because it continues.... till here

/src/build/radare2/radare2-0.10.0/libr/..//libr/anal/p/anal_arm_cs.c:593:21: error: 'arm_op_mem' has no member named 'lshift'
/src/build/radare2/radare2-0.10.0/libr/..//libr/anal/p/anal_arm_cs.c:595:19: error: 'arm_op_mem' has no member named 'lshift'
—
Reply to this email directly or view it on GitHub #4054 (comment).

[rofl0r]

i upgraded capstone to 4.0-alpha2, and now i get

p/asm_x86_cs.c:54:33: error: 'CS_OPT_SYNTAX_MASM' undeclared (first use in this function)

[rofl0r]

optimally you could coordinate releases with capstone, but if not the solution would be to integrate the source code (however without relying on git).
re ash: i have not experienced other problems but i can go over the source quickly to spot bashisms.

[rofl0r]

using latest capstone from "next" branch, i got through the build, however now linking fails:

$ cat src/logs/build_radare2.log | grep -C 2 "undefined re"
gcc -fdata-sections -ffunction-sections -Os -g0 -fno-unwind-tables -fno-asynchronous-unwind-tables -Wa,--noexecstack -mtune=generic -MD   -fPIC -g -Wall -D__UNIX__=1 -I/src/build/radare2/radare2-0.10.0/libr/../shlr/zip/include -pie -I/src/build/radare2/radare2-0.10.0/libr/..//libr/include rasm2.o -L.. -o rasm2 -s -Wl,--gc-sections -Wl,-z,relro,-z,now -Wl,-rpath-link=/lib -ldl    -L/src/build/radare2/radare2-0.10.0/libr/asm -L/src/build/radare2/radare2-0.10.0/libr/parse -L/src/build/radare2/radare2-0.10.0/libr/syscall -L/src/build/radare2/radare2-0.10.0/libr/anal -L/src/build/radare2/radare2-0.10.0/libr/reg -L/src/build/radare2/radare2-0.10.0/libr/db -L/src/build/radare2/radare2-0.10.0/libr/flags -L/src/build/radare2/radare2-0.10.0/libr/cons -L/src/build/radare2/radare2-0.10.0/libr/util -lr_asm -lr_parse -lr_syscall -lr_anal -lr_reg -lr_db -lr_flags -lr_cons -lr_util -fPIC /src/build/radare2/radare2-0.10.0/libr/../shlr/zip/librz.a
/bin/ld: warning: libr_lang.so, needed by /src/build/radare2/radare2-0.10.0/libr/asm/libr_asm.so, not found (try using -rpath or -rpath-link)
/src/build/radare2/radare2-0.10.0/libr/anal/libr_anal.so: undefined reference to `cs_reg_name'
/src/build/radare2/radare2-0.10.0/libr/anal/libr_anal.so: undefined reference to `cs_insn_group'
/src/build/radare2/radare2-0.10.0/libr/asm/libr_asm.so: undefined reference to `cs_free'
/src/build/radare2/radare2-0.10.0/libr/asm/libr_asm.so: undefined reference to `cs_group_name'
/src/build/radare2/radare2-0.10.0/libr/asm/libr_asm.so: undefined reference to `cs_option'
/src/build/radare2/radare2-0.10.0/libr/asm/libr_asm.so: undefined reference to `cs_disasm'
/src/build/radare2/radare2-0.10.0/libr/asm/libr_asm.so: undefined reference to `cs_close'
/src/build/radare2/radare2-0.10.0/libr/asm/libr_asm.so: undefined reference to `cs_open'
collect2: error: ld returned 1 exit status

apparently there's a -Wl,-rpath-link=somewhere-in-tree missing in LDFLAGS

[rofl0r]

running make dist in a git checkout i get:

~/radare2[master]$ make dist
[ configure -nt config-user.mk ] && ./configure "--prefix=/usr/local"
make: [dist] Error 1 (ignored)
#git log $(git show-ref `git tag |tail -n1`)..HEAD > ChangeLog
git log $(git show-ref | grep 0.10.0 | awk '{print $1}')..HEAD > ChangeLog
cd shlr && make capstone-sync
make[1]: Entering directory `/home/rofl/radare2/shlr'
Makefile:2: ../config-user.mk: No such file or directory
Makefile:4: ../libr/config.mk: No such file or directory
Makefile:244: warning: overriding recipe for target `all'
Makefile:51: warning: ignoring old recipe for target `all'
make[1]: *** No rule to make target `../libr/config.mk'.  Stop.
make[1]: Leaving directory `/home/rofl/radare2/shlr'
make: *** [dist] Error 2

[radare]

i have already delayed the r2 release for 3 months, and ive been waiting a lot for capstone to get the new version released.

but, even if capstone is updated, distros will not be updated, mainly because the new capstone release will break compat with all the previous releases, so this makes the thing even worst. the r2 tarball comes with the capstone code embedded, meinly because we use a properly tested version of it. capstone doesnt have testsuite, we do, and there are some bugs that are only fixed in our branch, because capstone doesnt wants to accept those patches. and im tired of fighting to get things done of software that is not mine.

the problem is that distros dislike to use embedded versions of code, which makes sense, but its a pain for software like r2 that aims to be reliable… for example. i also ship a version of libmagic, and distros link against the system libmagic,, which is vulnerable in many distros. mine comes from openbsd, the code has been reviewed, cleaned up and fuzzed a lot to verfy this. i would say libmagic is not safe in many distros.

the only solution here is to make r2 able to compile against previous versions of the libraries.

On 29 Jan 2016, at 18:07, rofl0r notifications@github.com wrote:

optimally you could coordinate releases with capstone, but if not the solution would be to integrate the source code (however without relying on git).
re ash: i have not experienced other problems but i can go over the source quickly to spot bashisms.

—
Reply to this email directly or view it on GitHub #4054 (comment).

[radare]

you have to run configure before make dist

On 29 Jan 2016, at 18:46, rofl0r notifications@github.com wrote:

running make dist in a git checkout i get:

~/radare2[master]$ make dist
[ configure -nt config-user.mk ] && ./configure "--prefix=/usr/local"
make: [dist] Error 1 (ignored)
#git log $(git show-ref git tag |tail -n1)..HEAD > ChangeLog
git log $(git show-ref | grep 0.10.0 | awk '{print $1}')..HEAD > ChangeLog
cd shlr && make capstone-sync
make[1]: Entering directory /home/rofl/radare2/shlr' Makefile:2: ../config-user.mk: No such file or directory Makefile:4: ../libr/config.mk: No such file or directory Makefile:244: warning: overriding recipe for targetall'
Makefile:51: warning: ignoring old recipe for target all' make[1]: *** No rule to make target../libr/config.mk'. Stop.
make[1]: Leaving directory `/home/rofl/radare2/shlr'
make: *** [dist] Error 2

—
Reply to this email directly or view it on GitHub #4054 (comment).

[rofl0r]

if i understood correctly, radare2 works only right with a patched capstone. what prevents you from embedding the patched source code in the tarball release (as opposed to checking it out from git)?

re make dist:
i ran it and everything seemed to work. however there was apparently no tarball produced.
what would you want me to check?

[rofl0r]

pardon, i saw now:

r2 tarball comes with the capstone code embedded

so apparently the problem is only that it tries to do something with git if --with-syscapstone is not given ?

[rofl0r]

i got radare2 now built (natively).
i need to apply the following patch http://sprunge.us/GaGQ
and run make with the following options:
make USE_CAPSTONE=1 because even the patched configure when passed --with-syscapstone doesnt pick up capstone, because capstone didnt install a pkg-config file.
i ran configure with sh -x and put the log here: http://sprunge.us/JTJP

[rofl0r]

i'm unsure why it was required to add r_lang to bindeps of some things, though.

[radare]

Nope. R2 (from git) works fine with capstone3, capstone4(next). The patches we have are basically fixing issues from the disaasembly, those patches are already submitted to the capstone project for more than half a year in pullreqs.

Patched Capstone src is already distributed in the tarball.

The tarball is already generated. Look closer :p

On 30 Jan 2016, at 16:13, rofl0r notifications@github.com wrote:

if i understood correctly, radare2 works only right with a patched capstone. what prevents you from embedding the patched source code in the tarball release (as opposed to checking it out from git)?

re make dist:
i ran it and everything seemed to work. however there was apparently no tarball produced.
what would you want me to check?

—
Reply to this email directly or view it on GitHub.

[radare]

Theres a capstone.mk instead of doing -lcapstone

On 30 Jan 2016, at 16:34, rofl0r notifications@github.com wrote:

i got radare2 now built (natively).
i need to apply the following patch http://sprunge.us/GaGQ
and run make with the following options:
make USE_CAPSTONE=1 because even the patched configure when passed --with-syscapstone doesnt pick up capstone, because capstone didnt install a pkg-config file.
i ran configure with sh -x and put the log here: http://sprunge.us/JTJP

—
Reply to this email directly or view it on GitHub.

[radare]

Rlang shouldnt be required to build those binaries. Which was the error message?

On 30 Jan 2016, at 16:34, rofl0r notifications@github.com wrote:

i got radare2 now built (natively).
i need to apply the following patch http://sprunge.us/GaGQ
and run make with the following options:
make USE_CAPSTONE=1 because even the patched configure when passed --with-syscapstone doesnt pick up capstone, because capstone didnt install a pkg-config file.
i ran configure with sh -x and put the log here: http://sprunge.us/JTJP

—
Reply to this email directly or view it on GitHub.

[rofl0r]

/bin/ld: warning: libr_lang.so, needed by /src/build/radare2/radare2-0.10.0/libr/asm/libr_asm.so, not found (try using -rpath or -rpath-link)

[radare]

Well, thats quite explicit. And asm/Makefile refers to rlang. So yep. Your patch is ok

On 30 Jan 2016, at 16:56, rofl0r notifications@github.com wrote:

/bin/ld: warning: libr_lang.so, needed by /src/build/radare2/radare2-0.10.0/libr/asm/libr_asm.so, not found (try using -rpath or -rpath-link)
—
Reply to this email directly or view it on GitHub.

[radare]

Can you confirm the configure script fails because of the quotes and that this is the only issue in there? it will be good to patch ACR with this change in order to get it fixed when running autogen.sh. thanks

[radare]

i've added the r_lang changes in binr/*/Makefile. I would like to reproduce the capstone linking issue, because your patch doesnt looks good to me, mainly because libcapstone is suposed to be statically linked to the r_asm_*_cs plugins, (or maybe dynamically), but this is a dependency in a plugin, not in a program or a library.

[radare]

try again. the [ is an alias for test. kinda posix thing

[rofl0r]

make dist finally succeeded, trying to use the tarball i now get

Using PKGCONFIG: pkg-config                                                     
checking pkg-config flags for capstone... no                                    
sed: bad regex '\+': Repetition not preceded by valid expression                
checking for libz ... yes                                                       
sed: bad regex '\+': Repetition not preceded by valid expression                
checking for libzip ... yes                                                     
sed: bad regex '\+': Repetition not preceded by valid expression                
checking for libssl ... yes                                                     
checking pkg-config flags for openssl... no                                     
sed: bad regex '\+': Repetition not preceded by valid expression                

...

 - LIBVERSION = 0.10.0-git                                                      
make: git: Command not found                                                    
Generating r_version.h file                                                     
...

build succeeds tho.
so we have only 2 problems left

• assumption that sed is gnu sed
• usage of git to query version number. this could be improved by using it only if .git/ exists

[radare]

_CHKLIB_LIBS=$(echo "-l${_CHKLIB_NAME}" | sed 's,+, -l,g’)

this is the offending line

On 26 Feb 2016, at 15:20, rofl0r notifications@github.com wrote:

make dist finally succeeded, trying to use the tarball i now get

Using PKGCONFIG: pkg-config
checking pkg-config flags for capstone... no
sed: bad regex '+': Repetition not preceded by valid expression
checking for libz ... yes
sed: bad regex '+': Repetition not preceded by valid expression
checking for libzip ... yes
sed: bad regex '+': Repetition not preceded by valid expression
checking for libssl ... yes
checking pkg-config flags for openssl... no
sed: bad regex '+': Repetition not preceded by valid expression

...

• LIBVERSION = 0.10.0-git
  make: git: Command not found
  Generating r_version.h file
  ...
  build succeeds tho.
  so we have only 2 problems left

assumption that sed is gnu sed
usage of git to query version number. this could be improved by using it only if .git/ exists
—
Reply to this email directly or view it on GitHub #4054 (comment).

[rofl0r]

what's the sed cmd supposed to achieve ?

[radare]

does it works if you use sed -e instead of sed in this line of the configure script?

On 26 Feb 2016, at 15:20, rofl0r notifications@github.com wrote:

make dist finally succeeded, trying to use the tarball i now get

Using PKGCONFIG: pkg-config
checking pkg-config flags for capstone... no
sed: bad regex '+': Repetition not preceded by valid expression
checking for libz ... yes
sed: bad regex '+': Repetition not preceded by valid expression
checking for libzip ... yes
sed: bad regex '+': Repetition not preceded by valid expression
checking for libssl ... yes
checking pkg-config flags for openssl... no
sed: bad regex '+': Repetition not preceded by valid expression

...

• LIBVERSION = 0.10.0-git
  make: git: Command not found
  Generating r_version.h file
  ...
  build succeeds tho.
  so we have only 2 problems left

assumption that sed is gnu sed
usage of git to query version number. this could be improved by using it only if .git/ exists
—
Reply to this email directly or view it on GitHub #4054 (comment).

[rofl0r]

nope, -e does not fix it

[rofl0r]

btw are lua, openssl, and zlib hard deps (i.e. if not found an in-tree copy gets used)?

[rofl0r]

if you can explain what the regex is supposed to achieve i'll ask the implementor of musl's BRE code how it can be written in a portable way.

[radare]

there are no hard deps in r2, everything can be compiled with just gcc (and git, make..)

lua is only required if you want to run lua scripts from the r2 prompt, so it

zlib is embedded in r2, the system dependency is just if you are a distro and dont like projects to embed their own forks of code from other libraries.

openssl is not used at all, it was used for big number computations with gmp in the past, but now is only planned to be used for https and such.

On 26 Feb 2016, at 15:33, rofl0r notifications@github.com wrote:

btw are lua, openssl, and zlib hard deps (i.e. if not found an in-tree copy gets used)?

—
Reply to this email directly or view it on GitHub #4054 (comment).

[radare]

It’s strange that this expression is not supported by your musl+busybox distro. can i download that distro from somewhere to try it?

the regexp is quite obvious, just replacing the “+” char for “ -l"

On 26 Feb 2016, at 15:34, rofl0r notifications@github.com wrote:

if you can explain what the regex is supposed to achieve i'll ask the implementor of musl's BRE code how it can be written in a portable way.

—
Reply to this email directly or view it on GitHub #4054 (comment).

[rofl0r]

you could dl a 40 MB rootfs from http://ftp.barfooze.de/pub/sabotage/ (the 1.1.12 ones are the newest) and then use them with chroot (
see also https://github.com/sabotage-linux/sabotage/blob/master/enter-chroot or
and https://github.com/sabotage-linux/sabotage/blob/master/KEEP/super_chroot.c if you want to chroot without root privs)
POSIX specifies for sed strict usage of BRE as opposed to ERE

$ echo +foo | sed 's@+@-@g'
-foo
$ echo +foo | sed 's@\+@-@g'
sed: bad regex '\+': Repetition not preceded by valid expression

[rofl0r]

according to one of the musl devs:

+ is not special in BRE
\+ is not an escaped +, it's an impl-defined thing that happens to be treated like the ERE +
+ and ? are ERE-only
plain + in BRE is a literal +

so it's fine to just leave the slash away

[radare]

I will install busybox:musl docker later for testing but if u confirm that removing the backslash works for you i'll update and release a new version of acr and bump that thing.

Thanks!

On 26 Feb 2016, at 20:27, rofl0r notifications@github.com wrote:

according to one of the musl devs:

• is not special in BRE
  • is not an escaped +, it's an impl-defined thing that happens to be treated like the ERE +
• and ? are ERE-only
  plain + in BRE is a literal +
  so it's fine to just leave the slash away

—
Reply to this email directly or view it on GitHub.

[rofl0r]

yeah i can confirm that. thanks for fixing it.

[radare]

i have installed the busybox:musl docker image and the sed works fine with that + thing.. but i can confirm that it also works fine without the backslash on gnu, bsd and musl machines i have. so i have done the change in acr and pushed it. pls try again

On 26 Feb 2016, at 21:55, rofl0r notifications@github.com wrote:

yeah i can confirm that. thanks for fixing it.

—
Reply to this email directly or view it on GitHub #4054 (comment).

[radare]

On 26 Feb 2016, at 15:20, rofl0r notifications@github.com wrote:

usage of git to query version number. this could be improved by using it only if .git/ exists

where is this check done?

[radare]

pls try again ,lemme know if i fixed the git errmsg thing

On 27 Feb 2016, at 10:39, Pancake Nopcode pancake@nopcode.org wrote:

On 26 Feb 2016, at 15:20, rofl0r <notifications@github.com mailto:notifications@github.com> wrote:

usage of git to query version number. this could be improved by using it only if .git/ exists

where is this check done?

[rofl0r]

yeah, musl 1.1.14 which came out 5 days ago, added the \+ \? stuff since it's so widely assumed to be available, so it seems your docker img is bleeding edge.

[rofl0r]

when doing make dist now, i get

"/bin/sh" capstone.sh "https://github.com/aquynh/capstone.git" "next" "2af0217258b61243bcc59f32b140482ea467492b" ""
Cloning into 'capstone'...
remote: Counting objects: 18026, done.
remote: Total 18026 (delta 0), reused 0 (delta 0), pack-reused 18026
Receiving objects: 100% (18026/18026), 26.54 MiB | 142.00 KiB/s, done.
Resolving deltas: 100% (12974/12974), done.
Checking connectivity... done
[capstone] Updating capstone from git...
HEAD eebfcb8ff26727a35e1be8d65341e52c8aad7ec4
TIP 2af0217258b61243bcc59f32b140482ea467492b
fatal: ambiguous argument '@^^^': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Branch next set up to track remote branch next from origin.
Switched to a new branch 'next'

[radare]

the @^^^ issue happens because you are using a pretty old version of git... this can be fixed by using HEAD instead of @.. but cmon.. this changed 1-2 years ago.. which version of git are you using?

[radare]

just changed that, pls verify, anyway, you shouldnt be running make dist if you are packaging the tarball.

[rofl0r]

$ git --version
git version 1.8.4

[rofl0r]

ok testing now

[radare]

This is 3 years old version of git.

On 27 Feb 2016, at 15:58, rofl0r notifications@github.com wrote:

$ git --version
git version 1.8.4
—
Reply to this email directly or view it on GitHub #4054 (comment).

[rofl0r]

and ? it works well and i never felt the urge to rebuild it.

[radare]

i have changed the @ for HEAD, so it should work for you now.

it’s ok not to update if you don’t care about remote code execution and path traversal vulnerabilities.

On 27 Feb 2016, at 16:04, rofl0r notifications@github.com wrote:

and ? it works well and i never felt the urge to rebuild it.

—
Reply to this email directly or view it on GitHub #4054 (comment).

[rofl0r]

i thought those were in libcurl, openssl etc, which are up to date

[rofl0r]

thanks, it appears everything works smoothly now.
however i get a few warnings that should probably be adressed http://sprunge.us/LhPe

[radare]

i have created a new issue for those warnings, which are already tracked somewhere else, but i always forgot O:P

#4186