Skip to content
This repository

[posix meterpreter] Various fixes for pre-NPTL threads and zombie reaper #792

Closed
wants to merge 3 commits into from

4 participants

M M wvu-r7 Brandon Perry James Lee
M M

various fixes for the pre-NPTL zombie reaper :

create it before the scheduler and destroy it after
every zombie, including the zombie reaper itself and the scheduler thread zombie, should be reaped and not left on the system

Brandon Perry brandonprry commented on the diff November 08, 2012
external/source/meterpreter/source/common/arch/posix/scheduler.c
@@ -50,6 +52,9 @@ DWORD scheduler_destroy( VOID )
50 52
 		thread_destroy(scheduler_thread);
51 53
 		scheduler_thread = NULL;
52 54
 
  55
+		dprintf("stopping zombie reaper");
2

debug line?

M M
mephos added a note November 08, 2012

dprintf prints debug to log file (/tmp/meterpreter.log.PID) only if debugging is enabled on meterpreter payload (which isn't by default)
it's invaluable when developping and there are this kind of debugging statements everywhere in posix meterpreter code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Brandon Perry brandonprry commented on the diff November 08, 2012
external/source/meterpreter/source/common/arch/posix/scheduler.c
@@ -50,6 +52,9 @@ DWORD scheduler_destroy( VOID )
50 52
 		thread_destroy(scheduler_thread);
51 53
 		scheduler_thread = NULL;
52 54
 
  55
+		dprintf("stopping zombie reaper");
  56
+		stop_zombie_reaper();
  57
+
53 58
 		dprintf("thread joined .. going for polltable");
1

debug line?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Brandon Perry brandonprry commented on the diff November 08, 2012
external/source/meterpreter/source/common/arch/posix/scheduler.c
@@ -81,6 +86,68 @@ DWORD scheduler_destroy( VOID )
81 86
 	return ERROR_SUCCESS;
82 87
 }
83 88
 
  89
+
  90
+/*
  91
+ * Reap child zombie threads on linux 2.4 (before NPTL)
  92
+ * each thread appears as a process and pthread_join don't necessarily reap it
  93
+ * threads are created using the clone syscall, so use special __WCLONE flag in waitpid
  94
+ */
  95
+
  96
+VOID reap_zombie_thread(void * param)
  97
+{
  98
+	pid_t pid;
  99
+	BOOL success;
  100
+	pthread_internal_t * ptr = (pthread_internal_t *)reaper_tid; 
  101
+ //       dprintf("reap_zombie_thread : getpid : %d , getppid : %d, gettid : %d, kernel_id : %d, is_kernel_24 : %d", getpid(), getppid(), gettid(), ptr->kernel_id, is_kernel_24);
2

should we remove old debug lines?

M M
mephos added a note November 08, 2012

when debugging problems, one just have to uncomment the line, recompile and test. saves times when developping for posix meterpreter

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Brandon Perry brandonprry commented on the diff November 08, 2012
external/source/meterpreter/source/common/arch/posix/scheduler.c
((34 lines not shown))
  119
+//				dprintf("tried to remove pid : %d , success : %d",pid, success);
  120
+			}
  121
+	        }
  122
+	}
  123
+	pthread_exit(0);
  124
+}
  125
+
  126
+/* 
  127
+ * ask the zombie reaper to stop
  128
+ * before stopping, the reaper will reap all remaining threads created, including the scheduler
  129
+ */
  130
+
  131
+VOID stop_zombie_reaper( VOID )
  132
+{
  133
+	pid_t pid;
  134
+	pthread_internal_t * ptr;
2

Not sure what the preferences are, but I am not sure of anyone that puts a space before and after the asterisk here.

M M
mephos added a note November 08, 2012

this is the same comment pattern as previous comments in the file and in some other posix meterpreter files

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Brandon Perry brandonprry commented on the diff November 08, 2012
external/source/meterpreter/source/common/arch/posix/scheduler.c
((36 lines not shown))
  121
+	        }
  122
+	}
  123
+	pthread_exit(0);
  124
+}
  125
+
  126
+/* 
  127
+ * ask the zombie reaper to stop
  128
+ * before stopping, the reaper will reap all remaining threads created, including the scheduler
  129
+ */
  130
+
  131
+VOID stop_zombie_reaper( VOID )
  132
+{
  133
+	pid_t pid;
  134
+	pthread_internal_t * ptr;
  135
+	// 2.6/3.x kernel don't have zombie reaper
  136
+	if (is_kernel_24 == 0)
1

if (!is_kernel_24)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Brandon Perry brandonprry commented on the diff November 08, 2012
external/source/meterpreter/source/common/arch/posix/scheduler.c
((43 lines not shown))
  128
+ * before stopping, the reaper will reap all remaining threads created, including the scheduler
  129
+ */
  130
+
  131
+VOID stop_zombie_reaper( VOID )
  132
+{
  133
+	pid_t pid;
  134
+	pthread_internal_t * ptr;
  135
+	// 2.6/3.x kernel don't have zombie reaper
  136
+	if (is_kernel_24 == 0)
  137
+		return;
  138
+	// stop zombie reaper
  139
+        stop_reaper = 1;
  140
+	ptr = (pthread_internal_t *)reaper_tid;
  141
+	// reap reaper thread itself
  142
+	pid = waitpid(ptr->kernel_id, NULL, __WCLONE);
  143
+	dprintf("zombie_reaper kernel_id : %d, ret pid : %d (should be equal)",ptr->kernel_id, pid);
1

debug line

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Brandon Perry brandonprry commented on the diff November 08, 2012
external/source/meterpreter/source/common/base.c
((9 lines not shown))
268 246
 	}
269 247
 
270 248
 	list_add( commandThreadList, thread );
271  
-
  249
+/*
  250
+#ifndef _WIN32
  251
+	pthread_internal_t * ptr = (pthread_internal_t *)(thread->pid); 
  252
+	dprintf("getpid : %d , getppid : %d, gettid : %d, kernel_id : %d", getpid(), getppid(), gettid(), ptr->kernel_id); 
1

debug line

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Brandon Perry brandonprry commented on the diff November 08, 2012
external/source/meterpreter/source/common/thread.c
@@ -390,6 +409,18 @@ THREAD * thread_create( THREADFUNK funk, LPVOID param1, LPVOID param2 )
390 409
 			return NULL;
391 410
 		}
392 411
 		// __paused_thread free's the allocated memory.
  412
+		if (is_kernel_24 == 1) {
  413
+			if( commandThreadListPID == NULL )
  414
+			{
  415
+				commandThreadListPID = list_create();
  416
+				if( commandThreadListPID == NULL )
  417
+					return NULL;
  418
+			}
  419
+			pthread_internal_t * ptr = (pthread_internal_t *)(thread->pid);
  420
+//			dprintf("list_add ptr->kernel_id : %d",ptr->kernel_id);
1

remove old debug lines?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
wvu-r7
Collaborator
wvu-r7 commented June 11, 2013

Holy crap. This one is old. @mephos???

M M
mephos commented June 14, 2013

The commit should still be good (comments were only about debugging statements present only when debugging is enabled which isn't by default). Unfortunately, I don't have much time to retest it. It's up to you now :)

wvu-r7
Collaborator
wvu-r7 commented June 15, 2013

I'll see if I can test this. If there are non-trivial issues, we may need to close this for a while. Too many PRs, and we just don't have time to get through all of them. :(

wvu-r7
Collaborator
wvu-r7 commented June 25, 2013

Might want to resubmit this over at the Meterpreter repo.

wvu-r7 wvu-r7 closed this June 25, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 3 unique commits by 2 authors.

Sep 08, 2012
[posix meterpreter] create reaper thread as child of master thread e536ecf
comment 5aa3233
Sep 16, 2012
fix zombie thread reaper ef5b17c
This page is out of date. Refresh to see the latest.
77  external/source/meterpreter/source/common/arch/posix/scheduler.c
@@ -19,6 +19,8 @@ LIST_HEAD(_WaitableEntryHead, _WaitableEntry) WEHead;
19 19
 
20 20
 THREAD *scheduler_thread;
21 21
 
  22
+int stop_reaper = 0;
  23
+
22 24
 /*
23 25
  * If there are no waitables in the queue, we wait
24 26
  * for a conditional broadcast to start it.
@@ -50,6 +52,9 @@ DWORD scheduler_destroy( VOID )
50 52
 		thread_destroy(scheduler_thread);
51 53
 		scheduler_thread = NULL;
52 54
 
  55
+		dprintf("stopping zombie reaper");
  56
+		stop_zombie_reaper();
  57
+
53 58
 		dprintf("thread joined .. going for polltable");
54 59
 
55 60
 		if(polltable)
@@ -81,6 +86,68 @@ DWORD scheduler_destroy( VOID )
81 86
 	return ERROR_SUCCESS;
82 87
 }
83 88
 
  89
+
  90
+/*
  91
+ * Reap child zombie threads on linux 2.4 (before NPTL)
  92
+ * each thread appears as a process and pthread_join don't necessarily reap it
  93
+ * threads are created using the clone syscall, so use special __WCLONE flag in waitpid
  94
+ */
  95
+
  96
+VOID reap_zombie_thread(void * param)
  97
+{
  98
+	pid_t pid;
  99
+	BOOL success;
  100
+	pthread_internal_t * ptr = (pthread_internal_t *)reaper_tid; 
  101
+ //       dprintf("reap_zombie_thread : getpid : %d , getppid : %d, gettid : %d, kernel_id : %d, is_kernel_24 : %d", getpid(), getppid(), gettid(), ptr->kernel_id, is_kernel_24);
  102
+	// when tested, this means we are on a 2.4 kernel
  103
+	if (getpid() == getppid())
  104
+		is_kernel_24 = 1;
  105
+	else
  106
+		is_kernel_24 = 0;
  107
+
  108
+	/*
  109
+	 * on a 2.4 kernel, we need the reaper
  110
+	 */
  111
+	if (is_kernel_24 == 1) {
  112
+		/* reap until asked to exit
  113
+		 * at this point, will reap any remaining zombie before exiting
  114
+		 */
  115
+		while( stop_reaper == 0 || (stop_reaper == 1 && list_count(commandThreadListPID) > 0)) {
  116
+			pid = waitpid(-1, NULL, __WCLONE);
  117
+			if (pid > 0) {
  118
+				success = list_remove(commandThreadListPID, pid);
  119
+//				dprintf("tried to remove pid : %d , success : %d",pid, success);
  120
+			}
  121
+	        }
  122
+	}
  123
+	pthread_exit(0);
  124
+}
  125
+
  126
+/* 
  127
+ * ask the zombie reaper to stop
  128
+ * before stopping, the reaper will reap all remaining threads created, including the scheduler
  129
+ */
  130
+
  131
+VOID stop_zombie_reaper( VOID )
  132
+{
  133
+	pid_t pid;
  134
+	pthread_internal_t * ptr;
  135
+	// 2.6/3.x kernel don't have zombie reaper
  136
+	if (is_kernel_24 == 0)
  137
+		return;
  138
+	// stop zombie reaper
  139
+        stop_reaper = 1;
  140
+	ptr = (pthread_internal_t *)reaper_tid;
  141
+	// reap reaper thread itself
  142
+	pid = waitpid(ptr->kernel_id, NULL, __WCLONE);
  143
+	dprintf("zombie_reaper kernel_id : %d, ret pid : %d (should be equal)",ptr->kernel_id, pid);
  144
+	// all zombies have been reaped, including scheduler one (it was inserted into commandThreadListPID)
  145
+	list_destroy(commandThreadListPID);
  146
+	commandThreadListPID = NULL;
  147
+	return;
  148
+
  149
+}
  150
+
84 151
 DWORD scheduler_initialize( Remote * remote )
85 152
 {
86 153
 	if(scheduler_thread) {
@@ -91,6 +158,16 @@ DWORD scheduler_initialize( Remote * remote )
91 158
 	pthread_mutex_init(&scheduler_mutex, NULL);
92 159
 	pthread_cond_init(&scheduler_cond, NULL);
93 160
 
  161
+	// create zombie thread reaper for pre-NPTL (in 2.4 kernels) threads
  162
+	pthread_attr_t tattr;
  163
+	pthread_attr_init(&tattr);
  164
+	pthread_attr_setdetachstate(&tattr,PTHREAD_CREATE_DETACHED);
  165
+	pthread_create(&reaper_tid, &tattr, reap_zombie_thread, NULL);
  166
+
  167
+	while (is_kernel_24 == -1) usleep(10000);
  168
+	// here, we know if we're on a 2.4 kernel or not
  169
+	
  170
+
94 171
 	scheduler_thread = thread_create(scheduler_run, remote, NULL);
95 172
 	if(! scheduler_thread) {
96 173
 		return ENOMEM;
29  external/source/meterpreter/source/common/base.c
@@ -212,23 +212,6 @@ VOID command_throtle( int maxthreads )
212 212
 }
213 213
 */
214 214
 
215  
-#ifndef _WIN32
216  
-/*
217  
- * Reap child zombie threads on linux 2.4 (before NPTL)
218  
- * each thread appears as a process and pthread_join don't necessarily reap it
219  
- * threads are created using the clone syscall, so use special __WCLONE flag in waitpid
220  
- */
221  
-
222  
-VOID reap_zombie_thread(void * param)
223  
-{
224  
-	while(1) {
225  
-		waitpid(-1, NULL, __WCLONE);
226  
-		// on 2.6 kernels, don't chew 100% CPU
227  
-		usleep(500000);
228  
-	}
229  
-}
230  
-#endif
231  
-
232 215
 /*
233 216
  * Process a single command in a seperate thread of execution.
234 217
  */
@@ -260,15 +243,15 @@ DWORD THREADCALL command_process_thread( THREAD * thread )
260 243
 		commandThreadList = list_create();
261 244
 		if( commandThreadList == NULL )
262 245
 			return ERROR_INVALID_HANDLE;
263  
-#ifndef _WIN32
264  
-		pthread_t tid;
265  
-		pthread_create(&tid, NULL, reap_zombie_thread, NULL);
266  
-		dprintf("reap_zombie_thread created, thread_id : 0x%x",tid);
267  
-#endif
268 246
 	}
269 247
 
270 248
 	list_add( commandThreadList, thread );
271  
-
  249
+/*
  250
+#ifndef _WIN32
  251
+	pthread_internal_t * ptr = (pthread_internal_t *)(thread->pid); 
  252
+	dprintf("getpid : %d , getppid : %d, gettid : %d, kernel_id : %d", getpid(), getppid(), gettid(), ptr->kernel_id); 
  253
+#endif
  254
+*/
272 255
 	__try
273 256
 	{
274 257
 		do
5  external/source/meterpreter/source/common/common.h
@@ -174,7 +174,10 @@ struct connection_table {
174 174
 #define METERPRETER_TRANSPORT_HTTP 1
175 175
 #define METERPRETER_TRANSPORT_HTTPS 2
176 176
 
177  
-#ifdef _WIN32
  177
+#ifndef _WIN32
  178
+extern LIST * commandThreadListPID;
  179
+
  180
+#else // if _WIN32
178 181
 
179 182
 
180 183
 #include <wininet.h>
31  external/source/meterpreter/source/common/thread.c
@@ -6,6 +6,24 @@
6 6
 int __futex_wait(volatile void *ftx, int val, const struct timespec *timeout);
7 7
 int __futex_wake(volatile void *ftx, int count);
8 8
 
  9
+/*
  10
+ *  are we running on a linux 2.4 kernel ?
  11
+ * if this is the case, as we're using pthread, a "few" things might not work
  12
+ * pthread_join will return immediately (sys_futex3 returns immediately as there's no support for futex in 2.4 kernels)
  13
+ * terminated threads end up as zombies in the system, we need to reap them
  14
+ * ...
  15
+ * empiric way observed during testing : if getpid() == getppid(), we're on a 2.4 kernel (didn't happen during testing on a 2.6/3.x kernel)
  16
+ */
  17
+
  18
+int is_kernel_24 = -1;
  19
+pthread_t reaper_tid = 0;
  20
+/*
  21
+ * A list of all command threads PID currenlty executing.
  22
+ */
  23
+LIST * commandThreadListPID = NULL;
  24
+
  25
+
  26
+
9 27
 #include <time.h>
10 28
 #include <signal.h>
11 29
 
@@ -316,6 +334,7 @@ void *__paused_thread(void *req)
316 334
 
317 335
 	return funk(thread);	
318 336
 }
  337
+
319 338
 #endif
320 339
 
321 340
 /*
@@ -390,6 +409,18 @@ THREAD * thread_create( THREADFUNK funk, LPVOID param1, LPVOID param2 )
390 409
 			return NULL;
391 410
 		}
392 411
 		// __paused_thread free's the allocated memory.
  412
+		if (is_kernel_24 == 1) {
  413
+			if( commandThreadListPID == NULL )
  414
+			{
  415
+				commandThreadListPID = list_create();
  416
+				if( commandThreadListPID == NULL )
  417
+					return NULL;
  418
+			}
  419
+			pthread_internal_t * ptr = (pthread_internal_t *)(thread->pid);
  420
+//			dprintf("list_add ptr->kernel_id : %d",ptr->kernel_id);
  421
+			list_add( commandThreadListPID, ptr->kernel_id );
  422
+		}
  423
+
393 424
 
394 425
 	} while(0);
395 426
 #endif
27  external/source/meterpreter/source/common/thread.h
@@ -37,6 +37,33 @@ typedef DWORD (WINAPI * NTOPENTHREAD)( PHANDLE, ACCESS_MASK, _POBJECT_ATTRIBUTES
37 37
 
38 38
 #else
39 39
 #include "pthread.h"
  40
+
  41
+/*
  42
+ *  are we running on a linux 2.4 kernel ?
  43
+ * if this is the case, as we're using pthread, a "few" things might not work
  44
+ * pthread_join will return immediately (sys_futex3 returns immediately as there's no support for futex in 2.4 kernels)
  45
+ * terminated threads end up as zombies in the system, we need to reap them
  46
+ * ...
  47
+ * empiric way observed during testing : if getpid() == getppid(), we're on a 2.4 kernel (didn't happen during testing on a 2.6/3.x kernel)
  48
+ */
  49
+
  50
+extern int is_kernel_24;
  51
+extern pthread_t reaper_tid;
  52
+
  53
+typedef struct pthread_internal_t
  54
+{
  55
+    struct pthread_internal_t*  next;
  56
+    struct pthread_internal_t** pref;
  57
+    pthread_attr_t              attr;
  58
+    pid_t                       kernel_id;
  59
+    pthread_cond_t              join_cond;
  60
+    int                         join_count;
  61
+    void*                       return_value;
  62
+    int                         intern;
  63
+    __pthread_cleanup_t*        cleanup_stack;
  64
+    void**                      tls;         /* thread-local storage area */
  65
+} pthread_internal_t;
  66
+
40 67
 #endif // _WIN32
41 68
 
42 69
 typedef struct _LOCK
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.