ctfmerge gets a segmentation fault when merging large set of objects on Ubuntu 12.04 X86_64 machine #92

shrikanth07 · 2014-12-10T18:43:26Z

The machine I am building on is a Ubuntu 12.0.4 X86_64 version and the CTF tools are built 64 bit. There is a large set of object files around 200+ which are compiled,linked to create a kernel module. The ctfconvert builds the .SUNW_ctf sections for each object file but when merging ctfmerge blows up, gets a segmentation fault. The command is ctfmerge -L VERSION -g -o module.ko.debug <list of object .o's>. I enabled the debugs in the environment i.e CTFMERGE_DEBUG_LEVEL 7 and CTFMERGE_DEBUG_PARSE 1 and have the following snapshot before the crash
Average: 0.35
DEBUG: 354588416: entering first barrier
DEBUG: 346195712: entering second barrier
DEBUG: 346195712: phase 1 complete
DEBUG: 371373824: entering second barrier
DEBUG: 354588416: doing work in first barrier
DEBUG: clearing slot 0 (0) (saving 5)
DEBUG: clearing slot 1 (1) (saving 5)
DEBUG: clearing slot 2 (2) (saving 5)
DEBUG: clearing slot 3 (3) (saving 5)
DEBUG: clearing slot 4 (4) (saving 5)
DEBUG: 362981120: entering second barrier
DEBUG: phase one done: donequeue has 75 items
DEBUG: 354588416: ninqueue is 149, 75 on queue
DEBUG: 354588416: entering second barrier

The crash backtrace
(gdb) bt
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:45
#1 0x00000000004029a5 in barrier_wait (bar=0x6333e0) at barrier.c:107
#2 0x0000000000403388 in worker_thread (wq=0x633280) at ctfmerge.c:547
#3 0x00007ffff7498e9a in start_thread (arg=0x7ffff5eca700) at pthread_create.c:308
#4 0x00007ffff71c53fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()

(gdb) frame 1
#1 0x00000000004029a5 in barrier_wait (bar=0x6333e0) at barrier.c:107

107
(gdb) l
102 pthread_mutex_lock(&bar->bar_lock);
103
104 if (++bar->bar_numin < bar->bar_nthr) {
105 pthread_mutex_unlock(&bar->bar_lock);
106 sem_wait(bar->bar_sem);
107
108 return (0);
109
110 } else {
111 int i;
(gdb) p bar[0]
$1 = {bar_lock = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, bar_numin = 1, bar_sem = 0x0,
bar_nthr = 5}

Further, an experimental nit in barrier.c

diff --git a/cmd/ctfconvert/barrier.c b/cmd/ctfconvert/barrier.c
index 94bb78d..8c37bff 100644
--- a/cmd/ctfconvert/barrier.c
+++ b/cmd/ctfconvert/barrier.c
@@ -90,8 +90,9 @@ void
barrier_init(barrier_t *bar, int nthreads)
{
pthread_mutex_init(&bar->bar_lock, NULL);

  bar->bar_sem = sem_open("ctfmerge_barrier", O_CREAT | O_EXCL);

  bar->bar_sem = sem_open("ctfmerge_barrier", O_CREAT);

```
   if (bar->bar_sem == SEM_FAILED)
```

           perror("sem_open failed");
 bar->bar_numin = 0;
 bar->bar_nthr = nthreads;

}

The perror reports EEXISTS.

The text was updated successfully, but these errors were encountered:

qwkslvr · 2014-12-16T00:01:36Z

I have a patch for the above said issue which involves creating a uniq name for the named-semaphore instead of a standard name. How do I submit the patch for review/commit ?

dtrace4linux · 2014-12-16T07:06:57Z

Hi

Thanks for this you can send me the paych direct.
On 16 Dec 2014 00:01, "qwkslvr" notifications@github.com wrote:

I have a patch for the above said issue which involves creating a uniq
name for the named-semaphore instead of a standard name. How do I submit
the patch for review/commit ?

—
Reply to this email directly or view it on GitHub
#92 (comment).

qwkslvr · 2014-12-16T18:03:37Z

Hi,
Sorry, I haven't submitted patches to github before. What's the recommended way? Do I send it to some email address? or just attach to this thread?

qwkslvr · 2015-01-05T19:08:24Z

Trying again. Can anyone tell me how to submit patches?

dtrace4linux · 2015-01-05T21:31:31Z

You can just do a diff -c on your tree vs the original and send it to me or
this mail thread, or do a "git push" from your tree.

tx

On 5 January 2015 at 19:08, qwkslvr notifications@github.com wrote:

Trying again. Can anyone tell me how to submit patches?

—
Reply to this email directly or view it on GitHub
#92 (comment).

qwkslvr · 2015-01-05T23:21:10Z

Just to summarize the changes, What I have done

Create dynamic semaphore names based on gettimeofday() so that parallel builds don't fail. Saved the name of the semaphore created in struct barrier
Create a new function called barrier_destroy() which cleans up the semaphores created.
Call barrier_destroy() when the program exits.

I tried doing git push, but It doesn't allow me to do. I get a
"error: The requested URL returned error: 403 while accessing https://github.com/dtrace4linux/linux.git/info/refs
fatal: HTTP request failed"

And I can't attach the patches here.
What's the email address I send this to?

qwkslvr · 2015-01-06T02:10:07Z

Hope this works. I am trying to attach the patch to this email.

Let me know if you got it.

On Mon, Jan 5, 2015 at 1:31 PM, dtrace4linux notifications@github.com
wrote:

You can just do a diff -c on your tree vs the original and send it to me
or
this mail thread, or do a "git push" from your tree.

tx

On 5 January 2015 at 19:08, qwkslvr notifications@github.com wrote:

Trying again. Can anyone tell me how to submit patches?

—
Reply to this email directly or view it on GitHub
#92 (comment).

—
Reply to this email directly or view it on GitHub
#92 (comment).

dtrace4linux · 2015-01-15T22:26:23Z

hello qwkslvr - apologies for not replying. I havent merged your
contribution yet into my tree. I was working on some nuisance reliability
issues in the 3.16 kernel, but havent finished. My last part was to
review/understand your changes (they look very reasonable given the
research you have done).

just backlogged on non-dtrace stuff.

On 15 January 2015 at 21:45, qwkslvr notifications@github.com wrote:

You didn't respond to my queries. So, pasting the patch here.
diff --git a/cmd/ctfconvert/barrier.c b/cmd/ctfconvert/barrier.c
index 94bb78d..6b8a423 100644
--- a/cmd/ctfconvert/barrier.c
+++ b/cmd/ctfconvert/barrier.c
@@ -53,6 +53,11 @@ barrier_init(barrier_t *bar, int nthreads)
bar->bar_nthr = nthreads;
}

+barrier_destroy(barrier_t *bar)
+{

/*XXX */ + +} int barrier_wait(barrier_t *bar) { @@ -83,19 +88,52 @@
barrier_wait(barrier_t *bar) #include #include #include - +#include
+#include +#include #include "barrier.h"

+#define SEM_STRING "ctfmerge"
+
void
barrier_init(barrier_t *bar, int nthreads)
{

int num;

struct timeval t; + pthread_mutex_init(&bar->bar_lock, NULL);

bar->bar_sem = sem_open("ctfmerge_barrier", O_CREAT | O_EXCL);

/* Get the timeof day as the seed to the random */

gettimeofday(&t, NULL);

srandom((unsigned int)t.tv_usec);

num = random(); +

snprintf(bar->sem_name, 20, "%s-%d", SEM_STRING, num);

bar->bar_sem = sem_open(bar->sem_name, O_CREAT|O_EXCL);
+

if(bar->bar_sem == SEM_FAILED) {

perror("sem_open failed");

exit(0);

} +

/*

* Save the random number generated, so that we can destroy on

* the exit of program.

*/

printf("sem_open for %s\n", bar->sem_name);

bar->bar_numin = 0; bar->bar_nthr = nthreads; }

+void
+barrier_destroy(barrier_t *bar)
+{

printf("Unlinking semaphore %s\n", bar->sem_name);

sem_unlink(bar->sem_name);
+}
+
int
barrier_wait(barrier_t *bar)
{
diff --git a/cmd/ctfconvert/barrier.h b/cmd/ctfconvert/barrier.h
index af9aea7..1487fb4 100644
--- a/cmd/ctfconvert/barrier.h
+++ b/cmd/ctfconvert/barrier.h
@@ -49,6 +49,7 @@ typedef struct barrier {
} barrier_t;

extern void barrier_init(barrier_t *, int);
+extern void barrier_destroy(barrier_t *);
extern int barrier_wait(barrier_t *);

#ifdef __cplusplus
@@ -63,6 +64,7 @@ typedef struct barrier {

sem_t bar_sem; / where everyone waits

/ int bar_nthr; / # of waiters to trigger release */

char sem_name[20];
} barrier_t;

extern void barrier_init(barrier_t *, int);
diff --git a/cmd/ctfconvert/ctfmerge.c b/cmd/ctfconvert/ctfmerge.c
index fffb92a..8766ed5 100644
--- a/cmd/ctfconvert/ctfmerge.c
+++ b/cmd/ctfconvert/ctfmerge.c
@@ -609,10 +609,14 @@ merge_ctf_cb(tdata_t *td, char *name, void *arg)

completion. The run time of ctfmerge can, however, be measured in
minutes

in some cases, so this is not a valid option. */ + +static
workqueue_t wq; + static void handle_sig(int sig) {

static void
@@ -630,6 +634,8 @@ terminate_cleanup(void)
fprintf(stderr, "Removing %s\n", outfile);
unlink(outfile);
}

- barrier_destroy(&(wq.wq_bar1));

barrier_destroy(&(wq.wq_bar2));
}

static void
@@ -763,7 +769,6 @@ strcompare(const void *p1, const void *p2)

into your stack to another thread is fragile at best and leads to
some

hard-to-debug failure modes. */ -static workqueue_t wq;

int
main(int argc, char *
_argv) @@ -1043,5 +1048,7 @@ main(int argc, char *_argv)
terminate("Couldn't rename output temp file %s", tmpname);

free(tmpname);

barrier_destroy(&(wq.wq_bar1));

barrier_destroy(&(wq.wq_bar2)); return (0); }

—
Reply to this email directly or view it on GitHub
#92 (comment).

qwkslvr · 2015-01-28T22:22:45Z

Ok, thanks. Let me know if you run into issues with the patch or have any questions for me. Would be interested in getting the patch committed soon.

dtrace4linux · 2015-02-21T10:36:57Z

gmail seems to have corrupted the diff output - can you send me the
relevant files you changed, and I will merge them in.

thanks

On 28 January 2015 at 22:22, qwkslvr notifications@github.com wrote:

Ok, thanks. Let me know if you run into issues with the patch or have any
questions for me. Would be interested in getting the patch committed soon.

—
Reply to this email directly or view it on GitHub
#92 (comment).

qwkslvr · 2015-02-23T04:49:47Z

I would love to, just tell me how. How do I attach the patch?

dtrace4linux · 2015-02-23T21:48:21Z

you can mail me at crispeditor/at/gmail.com

thanks

On 23 February 2015 at 04:49, qwkslvr notifications@github.com wrote:

I would love to, just tell me how. How do I attach the patch?

—
Reply to this email directly or view it on GitHub
#92 (comment).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ctfmerge gets a segmentation fault when merging large set of objects on Ubuntu 12.04 X86_64 machine #92

ctfmerge gets a segmentation fault when merging large set of objects on Ubuntu 12.04 X86_64 machine #92

shrikanth07 commented Dec 10, 2014

qwkslvr commented Dec 16, 2014

dtrace4linux commented Dec 16, 2014

qwkslvr commented Dec 16, 2014

qwkslvr commented Jan 5, 2015

dtrace4linux commented Jan 5, 2015

qwkslvr commented Jan 5, 2015

qwkslvr commented Jan 6, 2015

dtrace4linux commented Jan 15, 2015

bar->bar_sem = sem_open("ctfmerge_barrier", O_CREAT | O_EXCL);

num = random(); +

snprintf(bar->sem_name, 20, "%s-%d", SEM_STRING, num);

printf("sem_open for %s\n", bar->sem_name);

printf("Unlinking semaphore %s\n", bar->sem_name);

/ int bar_nthr; / # of waiters to trigger release */

- barrier_destroy(&(wq.wq_bar1));

free(tmpname);

qwkslvr commented Jan 28, 2015

dtrace4linux commented Feb 21, 2015

qwkslvr commented Feb 23, 2015

dtrace4linux commented Feb 23, 2015

ctfmerge gets a segmentation fault when merging large set of objects on Ubuntu 12.04 X86_64 machine #92

ctfmerge gets a segmentation fault when merging large set of objects on Ubuntu 12.04 X86_64 machine #92

Comments

shrikanth07 commented Dec 10, 2014

Further, an experimental nit in barrier.c

qwkslvr commented Dec 16, 2014

dtrace4linux commented Dec 16, 2014

qwkslvr commented Dec 16, 2014

qwkslvr commented Jan 5, 2015

dtrace4linux commented Jan 5, 2015

qwkslvr commented Jan 5, 2015

qwkslvr commented Jan 6, 2015

dtrace4linux commented Jan 15, 2015

bar->bar_sem = sem_open("ctfmerge_barrier", O_CREAT | O_EXCL);

num = random(); +

snprintf(bar->sem_name, 20, "%s-%d", SEM_STRING, num);

printf("sem_open for %s\n", bar->sem_name);

printf("Unlinking semaphore %s\n", bar->sem_name);

/ int bar_nthr; / # of waiters to trigger release */

- barrier_destroy(&(wq.wq_bar1));

free(tmpname);

qwkslvr commented Jan 28, 2015

dtrace4linux commented Feb 21, 2015

qwkslvr commented Feb 23, 2015

dtrace4linux commented Feb 23, 2015