separate debuginfo failures #542

stanfordcox · 2019-01-28T02:11:01Z

No description provided.

stanfordcox · 2019-01-28T02:24:54Z

Some tests that pass normally fail with separate debuginfo (which is how system debuginfo is shipped in Fedora)

(I stripped the debuginfo by building and installing the testsuite, then running
cd dyninst-testsuite/install/bin/testsuite
ln -s . usr
ln -s . lib
ln -s . debug
RPM_BUILD_ROOT=$(pwd) /usr/lib/rpm/find-debuginfo.sh .
(creates *.debug for each executable and shared library)

There are some new failures
+test1_23 +test1_23 +test1_24 +test1_24 +test1_26 +test1_26 +test1_27 +test1_27 +test1_6 +test1_6 +test5_1 +test5_1 +test5_2 +test5_2 +test5_5 +test5_5 +test5_7 +test5_7 +test5_8 +test5_8 +test5_9 +test5_9 +test_driver: /work/scox/dynins +test_line_info +[test_local_var_locations.C:90] +test_local_var_locations g++ +test_local_var_lookup +test_lookup_func +test_thread_5 +test_type_info g++ none 64 create NA dynamic nonPIC FAILED (Running Test)
g++ none 64 rewriter NA dynamic nonPIC FAILED (Running Test)
g++ none 64 create NA dynamic nonPIC FAILED (Running Test)
g++ none 64 rewriter NA dynamic nonPIC FAILED (Running Test)
g++ none 64 create NA dynamic nonPIC FAILED (Running Test)
g++ none 64 rewriter NA dynamic nonPIC FAILED (Running Test)
g++ none 64 create NA dynamic nonPIC FAILED (Running Test)
g++ none 64 rewriter NA dynamic nonPIC FAILED (Running Test)
g++ none 64 create NA dynamic nonPIC FAILED (Group Teardown)
g++ none 64 rewriter NA dynamic nonPIC FAILED (Group Teardown)
g++ none 64 create NA dynamic nonPIC FAILED (Running Test)
g++ none 64 rewriter NA dynamic nonPIC FAILED (Running Test)
g++ none 64 create NA dynamic nonPIC FAILED (Running Test)
g++ none 64 rewriter NA dynamic nonPIC FAILED (Running Test)
g++ none 64 create NA dynamic nonPIC FAILED (Running Test)
g++ none 64 rewriter NA dynamic nonPIC FAILED (Running Test)
g++ none 64 create NA dynamic nonPIC FAILED (Running Test)
g++ none 64 rewriter NA dynamic nonPIC FAILED (Running Test)
g++ none 64 create NA dynamic nonPIC FAILED (Running Test)
g++ none 64 rewriter NA dynamic nonPIC FAILED (Running Test)
g++ none 64 create NA dynamic nonPIC FAILED (Running Test)
g++ none 64 rewriter NA dynamic nonPIC FAILED (Running Test)
t/src/dyninstAPI/src/BPatch_snippet.C:223: AstNodePtr generateArrayRef(const BPatch_snippet&, const BPatch_snippet&): Assertion 0' failed. +test_driver: /work/scox/dyninst/src/dyninstAPI/src/BPatch_snippet.C:223: AstNodePtr generateArrayRef(const BPatch_snippet&, const BPatch_snippet&): Assertion 0' failed.
g++ none 64 disk NA dynamic nonPIC FAILED (Running Test)
- variable (local_var_locations) not found.
none 64 disk NA dynamic nonPIC FAILED (Running Test)
g++ none 64 disk NA dynamic nonPIC FAILED (Running Test)
g++ none 64 disk NA dynamic nonPIC FAILED (Running Test)
g++ none 64 create NA dynamic nonPIC CRASHED (Running Test)
g++ none 64 disk NA dynamic nonPIC FAILED (Running Test)

sashanicolas · 2019-01-28T19:18:05Z

Hi @stanfordcox, I have seen once aborting at that line (BPatch_snippet.C:223) but couldn't reproduce it. But it wasn't with separate debuginfo files.

In your case, what binaries had their debug info separated, the system dependencies or the testsuite mutatees?

Are you using the master branch last commit? Is it x86_64?

stanfordcox · 2019-01-28T20:14:55Z

@sashanicolas So I forced the testsuite to use separate debuginfo for all the tests by using:
% rpm -q /usr/lib/rpm/find-debuginfo.sh
rpm-build-4.14.2.1-2.fc28.x86_64

find-debuginfo.sh creates separate debuginfo in /usr/lib/debug, so this forces it to create

the separate *.debug files in the working dir

% cd .../dyninst-testsuite/install/bin/testsuite
% ls -dl usr lib debug
lrwxrwxrwx. 1 scox stapusr 1 Jan 28 14:45 debug -> ./
lrwxrwxrwx. 1 scox stapusr 1 Jan 24 16:34 lib -> ./
lrwxrwxrwx. 1 scox stapusr 1 Jan 24 16:34 usr -> ./

now create the separate debuginfo for all executables and libraries in testsuite install dir

% RPM_BUILD_ROOT=$(pwd) /usr/lib/rpm/find-debuginfo.sh . >& tee /tmp/,fdbi

now all testsuite debuginfo is stripped

% ls *.debug
amd64_7_arg_call.dyn_g++_32_none_none.debug test2_14.dyn_g++_32_none_none.debug
amd64_7_arg_call.dyn_g++_32_pic_none.debug test2_14.dyn_g++_32_pic_none.debug
...

Now I just run the testsuite (results without and with separate debuginfo attached

dyn-sdbi-results.zip

(I've been looking at symtabAPI/src/dwarfWalker.C et al to try and narrow this down but I don't have a good handle yet on how this all fits together)

sashanicolas · 2019-01-29T22:43:48Z

@stanfordcox it seems that something is wrong with:
% RPM_BUILD_ROOT=$(pwd) /usr/lib/rpm/find-debuginfo.sh . >& tee /tmp/,fdbi
this >& tee creates a text file called tee with the following:

extracting debug info from <my-inicial-path>/bin/testsuite/testdriver_wrapper
Dest dir longer than base dir is not supported

It seems it's trying to start stripping them but fails. Did you mean to redirect to the program tee?

stanfordcox · 2019-01-30T01:55:18Z

Ah sorry, cut and paste error: Yes I meant to redirect via tee

RPM_BUILD_ROOT=$(pwd) /usr/lib/rpm/find-debuginfo.sh . |& tee /tmp/,fdbi

sashanicolas · 2019-01-31T00:00:32Z

@stanfordcox Yeah, I tested that before assuming you meant that. But same result.

[sasha@zatar] (11)$ RPM_BUILD_ROOT=$(pwd) /usr/lib/rpm/find-debuginfo.sh . |& tee /tmp/,fdbi
extracting debug info from <my-inicial-path>/sasha/local/bin/testsuite/testdriver_wrapper
Dest dir longer than base dir is not supported
[sasha@zatar] (12)$

stanfordcox · 2019-01-31T16:41:22Z

Heh, not sure where that is coming from. (The ln -s . usr;ln -s . lib;ln -s . debug; links are setup?)

@sashanicolas could you try:
RPM_BUILD_ROOT=$(pwd) sh -x /usr/lib/rpm/find-debuginfo.sh .

sashanicolas · 2019-01-31T21:35:57Z

@stanfordcox

[sasha@zatar] (5)$ RPM_BUILD_ROOT=$(pwd) sh -x /usr/lib/rpm/find-debuginfo.sh .
+ strip_g=false
+ strip_r=false
+ strict=false
+ BUILDDIR=.
+ out=debugfiles.list
+ nout=0
+ [ 1 -gt 0 ]
+ BUILDDIR=.
+ shift
+ break
+ i=0
/usr/lib/rpm/find-debuginfo.sh: 76: /usr/lib/rpm/find-debuginfo.sh: cannot open nout: No such file
+ i
/usr/lib/rpm/find-debuginfo.sh: 76: /usr/lib/rpm/find-debuginfo.sh: i: not found
+ LISTFILE=./debugfiles.list
+ SOURCEFILE=./debugsources.list
+ LINKSFILE=./debuglinks.list
+ 
+ 
+ 
+ debugdir=/<initial-path-hidden>/sasha/local/bin/testsuite/usr/lib/debug
+ shopt -s extglob
/usr/lib/rpm/find-debuginfo.sh: 109: /usr/lib/rpm/find-debuginfo.sh: shopt: not found
+ set -o pipefail
/usr/lib/rpm/find-debuginfo.sh: 199: set: Illegal option -o pipefail
[sasha@zatar] (6)$

stanfordcox · 2019-01-31T22:11:34Z

...shopt: not found...
...Illegal option -o pipefail...
shopt and set -o are implemented by the bash shell (see below)

% help shopt
shopt: shopt [-pqsu] [-o] [optname ...]
Set and unset shell options.
% echo $SHELL
/bin/bash

@sashanicolas Are you using the bash shell? Any chance you are using zsh or tsh? The first line of find-debuginfo.sh is
#!/bin/bash
so the shell should be interpreted by /bin/bash so if you are using another shell, I am confused why these standard bash constructs are failing.

sashanicolas · 2019-01-31T22:32:59Z

[sasha@zatar] (18)$ echo $0
-bash
[sasha@zatar] (19)$ shopt
autocd         	off
cdable_vars    	off
cdspell        	off
checkhash      	off
checkjobs      	off
checkwinsize   	on
cmdhist        	on
compat31       	off
compat32       	off
compat40       	off
compat41       	off
compat42       	off
complete_fullquote	on
direxpand      	off
dirspell       	off
dotglob        	off
execfail       	off
expand_aliases 	on
extdebug       	off
extglob        	on
extquote       	on
failglob       	off
force_fignore  	on
globstar       	off
globasciiranges	off
gnu_errfmt     	off
histappend     	off
histreedit     	off
histverify     	off
hostcomplete   	off
huponexit      	off
interactive_comments	on
lastpipe       	off
lithist        	off
login_shell    	on
mailwarn       	off
no_empty_cmd_completion	off
nocaseglob     	off
nocasematch    	off
nullglob       	off
progcomp       	on
promptvars     	on
restricted_shell	off
shift_verbose  	off
sourcepath     	on
xpg_echo       	off

sashanicolas · 2019-01-31T22:41:50Z

It seems the first error occurs at line 76
/usr/lib/rpm/find-debuginfo.sh: 76: /usr/lib/rpm/find-debuginfo.sh: cannot open nout: No such file

But nout is a variable declared before, not a file.

Command set also works.

stanfordcox · 2019-02-01T15:48:18Z

Something is odd here but it is not obvious to me what it is. What version of linux, bash and rpm-build?
% uname -a
Linux pulelehua 4.20.4-100.fc28.x86_64 #1 SMP Wed Jan 23 16:46:32 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
% rpm -q bash
bash-4.4.23-1.fc28.x86_64
% rpm -q rpm-build
rpm-build-4.14.2.1-2.fc28.x86_64

My find-debuginfo.sh is different at line 76 but based on context the complaint seems to be at the while:
i=0
while ((i < nout)); do

If that is the case then I don't know why it would think i and nout were files as opposed to shell variables. Normally shell variables are prefixed with '$' but
info bash 'bash features' 'shell arithmetic'
says
..using the '((' compound command
..shell variables may also be referenced by name without using the
parameter expansion syntax.

so ((i < nout)) should evaluate the condition and interpret i and nout as shell variables.

Does this small example work as expected?
% cat /tmp/x.sh
#! /bin/bash
i=0
nout=1
while ((i < nout)) ; do
echo "i less than nout"
break
done
% sh /tmp/x.sh
i less than nout
%

sashanicolas · 2019-02-01T17:12:25Z

Well, here's the reason.

[sasha@zatar] (3)$ sh /tmp/x.sh 
/tmp/x.sh: 4: /tmp/x.sh: cannot open nout: No such file
/tmp/x.sh: 4: /tmp/x.sh: i: not found
[sasha@zatar] (4)$ uname -a
Linux zatar 4.4.0-140-generic #166-Ubuntu SMP Wed Nov 14 20:09:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[sasha@zatar] (5)$ rpm -q bash
package bash is not installed
[sasha@zatar] (6)$ rpm -q rpm-build
package rpm-build is not installed
[sasha@zatar] (7)$

sashanicolas · 2019-02-01T19:29:21Z

@stanfordcox
When I do RPM_BUILD_ROOT=$(pwd) bash -x /usr/lib/rpm/find-debuginfo.sh .
the nout issue seems to be overcome.

+ strip_g=false
+ strip_r=false
+ strict=false
+ BUILDDIR=.
+ out=debugfiles.list
+ nout=0
+ '[' 1 -gt 0 ']'
+ case "$1" in
+ BUILDDIR=.
+ shift
+ break
+ i=0
+ (( i < nout ))
+ LISTFILE=./debugfiles.list
+ SOURCEFILE=./debugsources.list
+ LINKSFILE=./debuglinks.list
+ debugdir=/<path-hidden>/sasha/local/bin/testsuite/usr/lib/debug
+ shopt -s extglob
+ set -o pipefail
+ strict_error=ERROR
+ false
+ strict_error=WARNING
+ find /<path-hidden>/sasha/local/bin/testsuite '!' -path '/<path-hidden>/sasha/local/bin/testsuite/usr/lib/debug/*.debug' -type f '(' -perm -0100 -or -perm -0010 -or -perm -0001 ')' -print
+ file -N -f -
+ sed -n -e 's/^\(.*\):[ 	]*.*ELF.*, not stripped/\1/p'
+ read nlinks inum f
+ xargs --no-run-if-empty stat -c '%h %D_%i %n'
+ get_debugfn /<path-hidden>/sasha/local/bin/testsuite/testdriver_wrapper
++ dirname /testdriver_wrapper
+ dn=/
++ basename /<path-hidden>/sasha/local/bin/testsuite/testdriver_wrapper .debug
+ bn=testdriver_wrapper.debug
+ debugdn=/<path-hidden>/sasha/local/bin/testsuite/usr/lib/debug/
+ debugfn=/<path-hidden>/sasha/local/bin/testsuite/usr/lib/debug//testdriver_wrapper.debug
+ '[' -f /<path-hidden>/sasha/local/bin/testsuite/usr/lib/debug//testdriver_wrapper.debug ']'
+ '[' 1 -gt 1 ']'
+ echo 'extracting debug info from /<path-hidden>/sasha/local/bin/testsuite/testdriver_wrapper'
extracting debug info from /<path-hidden>/sasha/local/bin/testsuite/testdriver_wrapper
++ /usr/lib/rpm/debugedit -b '' -d /usr/src/debug -i -l ./debugsources.list /<path-hidden>/sasha/local/bin/testsuite/testdriver_wrapper
Dest dir longer than base dir is not supported
+ id=
+ exit
+ exit

stanfordcox · 2019-02-01T19:48:22Z

Hmm I am not sure about the debugedit complain. I get a similar invocation for -b and -d but no error:
++ /usr/lib/rpm/debugedit -b '' -d /usr/src/debug -i -l ./debugsources.list /PATH/dyninst-testsuite/install/bin/testsuite/testdriver_wrapper

id=bffdeedecef3802a748a7d5e3f9346645e60caaf
...

Guess I can poke at the source and try to guess what is going on

sashanicolas · 2019-02-01T19:57:05Z

I read about this dest dir and base dir fairly enough to know that it's just a way to avoid dealing with strings.
So now I made it work. The command line that I am using is:

RPM_BUILD_ROOT=$(pwd) RPM_BUILD_DIR=$(pwd)/building bash /usr/lib/rpm/find-debuginfo.sh .

My find-debuginfo.sh is somehow invoking sh which for Ubuntu is different from bash. And was complaining about the syntax use of nout.
I had to add RPM_BUILD_DIR since it will be used as the base dir, and was empty and causing the message "Dest dir longer than base dir is not supported".
Also the script use eu-* files, which are from elfutils.

sashanicolas · 2019-02-12T23:51:00Z

@stanfordcox I believe this commit will fix it c371022

stanfordcox · 2019-02-13T15:23:47Z

Aha, was not looking at the right module; seems fine now, thanks!

sashanicolas self-assigned this Jan 30, 2019

sashanicolas closed this as completed Feb 12, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

separate debuginfo failures #542

separate debuginfo failures #542

stanfordcox commented Jan 28, 2019

stanfordcox commented Jan 28, 2019

sashanicolas commented Jan 28, 2019

stanfordcox commented Jan 28, 2019

sashanicolas commented Jan 29, 2019

stanfordcox commented Jan 30, 2019

sashanicolas commented Jan 31, 2019 •

edited

stanfordcox commented Jan 31, 2019

sashanicolas commented Jan 31, 2019

stanfordcox commented Jan 31, 2019

sashanicolas commented Jan 31, 2019

sashanicolas commented Jan 31, 2019

stanfordcox commented Feb 1, 2019

sashanicolas commented Feb 1, 2019

sashanicolas commented Feb 1, 2019

stanfordcox commented Feb 1, 2019

sashanicolas commented Feb 1, 2019

sashanicolas commented Feb 12, 2019

stanfordcox commented Feb 13, 2019

separate debuginfo failures #542

separate debuginfo failures #542

Comments

stanfordcox commented Jan 28, 2019

stanfordcox commented Jan 28, 2019

sashanicolas commented Jan 28, 2019

stanfordcox commented Jan 28, 2019

find-debuginfo.sh creates separate debuginfo in /usr/lib/debug, so this forces it to create

the separate *.debug files in the working dir

now create the separate debuginfo for all executables and libraries in testsuite install dir

now all testsuite debuginfo is stripped

Now I just run the testsuite (results without and with separate debuginfo attached

sashanicolas commented Jan 29, 2019

stanfordcox commented Jan 30, 2019

sashanicolas commented Jan 31, 2019 • edited

stanfordcox commented Jan 31, 2019

sashanicolas commented Jan 31, 2019

stanfordcox commented Jan 31, 2019

sashanicolas commented Jan 31, 2019

sashanicolas commented Jan 31, 2019

stanfordcox commented Feb 1, 2019

sashanicolas commented Feb 1, 2019

sashanicolas commented Feb 1, 2019

stanfordcox commented Feb 1, 2019

sashanicolas commented Feb 1, 2019

sashanicolas commented Feb 12, 2019

stanfordcox commented Feb 13, 2019

sashanicolas commented Jan 31, 2019 •

edited