[Bug] Stack is not displayed when attaching to gdbserver inside a container #901

Ultimator14 · 2022-10-19T09:36:45Z

GEF+GDB version

GEF: (Standalone)
Blob Hash(/usr/share/gef/gef.py): 0653da00ca754d9580d60a330907c988d6468dee
SHA256(/usr/share/gef/gef.py): f14771fa2cafaf7bca7f8b2a53db54c85f6612f42ac8ff80aaaa476fd7553e69
GDB: 12.1
GDB-Python: 3.10

Operating System

Gentoo Linux

Describe the issue you encountered

I'm using gdbserver to debug my binaries inside a docker container.

The setup is as follows

Host:               Container:
--------------------------------
gef   -> connect -> gdbserver
 |                      |
binary  <--- volume <---

Since commit 546f4b1 the stack is not displayed anymore. The stack field just displays [!] Unmapped address. Additionally I get a [*] Omitting remote file warning (see example below).

...
Reading /usr/lib/debug/lib64//ld-linux-x86-64.so.2.debug from remote target...
Reading target:/usr/lib/debug/lib64//ld-linux-x86-64.so.2.debug from remote target...
[*] Omitting remote file '/lib64/ld-linux-x86-64.so.2'
Reading /usr/lib/debug/.build-id/8a/29dcb265d5fa49083ddb942a7581a54dade086.debug from remote target...
0x00007f4f5d224730 in ?? () from target:/lib64/ld-linux-x86-64.so.2
[ Legend: Modified register | Code | Heap | Stack | String ]
───────────────────────────────────────────────────────────────── registers ────
$rax   : 0x0               

...
$r14   : 0x0               
$r15   : 0x7f4f5d23c000    
$eflags: [ZERO carry PARITY adjust sign trap INTERRUPT direction overflow resume virtualx86 identification]
$cs: 0x33 $ss: 0x2b $ds: 0x00 $es: 0x00 $fs: 0x00 $gs: 0x00 
───────────────────────────────────────────────────────────────────── stack ────
[!] Unmapped address: '0x7ffed79afbf0'
─────────────────────────────────────────────────────────────── code:x86:64 ────
0x400689 <win+18>         ret    
...

Since commit 430d9d3, the [*] Omitting remote file '/lib64/ld-linux-x86-64.so.2' warning is changed to several warnings of the type

[*] Failed to find objfile or not a valid file format: [Errno 2] Datei oder Verzeichnis nicht gefunden: 'target:/lib64/ld-linux-x86-64.so.2'
Reading /usr/lib/debug/.build-id/8a/29dcb265d5fa49083ddb942a7581a54dade086.debug from remote target...
[*] Failed to find objfile or not a valid file format: [Errno 2] Datei oder Verzeichnis nicht gefunden: 'system-supplied DSO at 0x7ffec83c2000'

The pull request #897 removes the [*] Failed to find objfile or not a valid file format warnings but creates a Python Exception.
The issue is not fixed in HEAD.

Do you read the docs and look at previously closed issues/PRs for similar cases?

Yes

Architecture impacted

Describe your issue. Without a proper reproduction step-by-step, your issue will be ignored.

Place the binary to debug in the current directory
Start the docker container docker run --tty --interactive --rm -v "$PWD":/workdir -w /workdir ultimator14/arch-interactive /bin/bash
Start gdbserver in the container gdbserver 0.0.0.0:4101 ./mybinary
Start gef outside the container gef ./mybinary
Connect gef to gdbserver inside the container target remote 172.17.0.2:4101
See that gef does not display the stack

Minimalist test case

This happens with every binary

Additional context?

No response

The text was updated successfully, but these errors were encountered:

clubby789 · 2022-10-19T09:48:48Z

Using gef-remote should help here. target remote is specifically unsupported by GEF.

Ultimator14 · 2022-10-19T09:50:08Z

Thanks there is no issue when using gef-remote

Ultimator14 · 2022-10-20T09:03:56Z

Actually this also happens with gef-remote if the inferior process forks and the debugger attaches to the child.
The setup is exactly the same as above but with gef-remote 172.17.0.2 4141 for the connection.

Everything works fine until the inferior process passes a fork call.
Then gef drops out of remote mode and the stack displays [!] Unmapped address again.
This only happens with set follow-fork-mode child.
There are no Failed to find objfile or not a valid file format warnings with gef-remote.

Sample program for testing

#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>

int main() {
	int pid = fork();

	if (pid < 0) {
		printf("Error on fork");
		exit(1);
	}

	if (pid == 0) {
		printf("I'm the child");
	} else {
		printf("I'm the parent");
	}

	return 0;
}

Compile with: gcc -o forktest forktest.c

Steps to reproduce
1-4. from above
5. Connect gef to gdbserver inside the container gef-remote 172.17.0.2 4101
6. (remote) gef➤ set follow-fork-mode child
7. (remote) gef➤ b *main+8
8. (remote) gef➤ c
9. (everything ok at this point)
10. (remote) gef➤ ni
11. Stack displays [!] Unmapped address, gef prompt changed from (remote) gef➤ to gef➤

Log if fork is called

(remote) gef➤  
[Attaching after Thread 17.17 fork to child Thread 18.18]
[New inferior 2 (process 18)]
Reading /workdir/forktest from remote target...
Reading /workdir/forktest from remote target...
[=] [remote] in remote_objfile_handler(target:/workdir/forktest))
[Detaching after fork from parent process 17]
[Inferior 1 (process 17) detached]
Reading /lib64/ld-linux-x86-64.so.2 from remote target...
Reading /usr/lib/debug/.build-id/8a/29dcb265d5fa49083ddb942a7581a54dade086.debug from remote target...
[=] [remote] in remote_objfile_handler(system-supplied DSO at 0x7ffdec76c000))
[*] [remote] skipping 'system-supplied DSO at 0x7ffdec76c000'
gef➤  [Switching to Thread 18.18]

theguy147 · 2022-10-20T19:10:00Z

I can reproduce the issue and found the problem: GDB is triggering GEFs exit_handler when detaching from the parent process (before attaching to the child process). We therefore have to catch this case in the handler.

theguy147 · 2022-10-20T19:55:04Z

@Ultimator14 can you check if the draft PR #903 fixes the issue for you as well?

Ultimator14 · 2022-10-20T22:36:23Z

Yes, #903 fixes the issue for me.

Ultimator14 · 2022-12-19T16:20:22Z

I noticed that this bug also occurs with any binary if the current working directory contains spaces.

Steps to reproduce:

Navigate to a path containing spaces (e.g. mkdir test\ dir && cd test\ dir)
Place any binary in the current directory (for testing I used a simple hello world program)
Start the docker container with docker run --tty --interactive --rm -v "$PWD":/workdir -w /workdir ultimator14/arch-interactive /bin/bash
Start gdbserver in the container gdbserver 0.0.0.0:4101 ./mybinary
Start gef outside the container gef ./mybinary
Connect gef to gdbserver inside the container gef-remote 172.17.0.2 4141
Gef will throw the error [!] Command 'gef-remote' failed to execute properly, reason: Invalid parameters to remote get and will not display the stack

When running gdbserver directly on the host, the error also occurs but the stack is display nevertheless.

stale · 2023-02-17T21:47:21Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. You can reopen it by adding a comment to this issue.

Ultimator14 · 2023-03-01T17:58:21Z

Both #901 (comment) and #901 (comment) still occur in current dev.

stale · 2023-04-30T19:16:19Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. You can reopen it by adding a comment to this issue.

stale · 2023-07-24T22:30:22Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. You can reopen it by adding a comment to this issue.

stale · 2023-08-23T23:34:22Z

This issue has been automatically closed because it has not had recent activity. If you are the owner of this issue, you can either re-open it and provide a more complete description; or create a new issue. Thank you for your contributions.

Grazfather · 2023-08-24T15:10:10Z

Will try to fix this one, reopening.

Grazfather · 2023-08-24T18:28:43Z

Seems gef handles this now. @Ultimator14: I know this issue is super old, but if you still use gef, I would be curious if you could still reproduce on main.

Ultimator14 · 2023-08-24T21:11:02Z

I can still reproduce both issues on main (using the program from #901 (comment))

Container (same commands for both setups)

docker run --tty --interactive --rm -v "$PWD":/workdir -w /workdir ultimator14/arch-interactive /bin/bash
gdbserver 0.0.0.0:4101 ./forktest

1. Unmapped address after fork

Current directory: ~/workdir/gef
Commands on host:

gdb -x "~/workdir/gef/repo/gef.py" ./forktest
gef-remote 172.17.0.2 4101
set follow-fork-mode child
b *main+8
c
ni
ni

2. Unmapped address at any point if path contains spaces

Current directory: ~/workdir/gef/spa ces
Commands on host:

gdb -x "~/workdir/gef/repo/gef.py" ./forktest
gef-remote 172.17.0.2 4101
b *main+8
c

Grazfather · 2023-08-25T00:11:26Z

I'll focus on #2. Confirming that I can reproduce the issue, and that I cannot if I rerun from a directory without spaces.

Grazfather · 2023-08-25T00:16:32Z

Can you please try making this change?

diff --git a/gef.py b/gef.py
index 5435c79..feb4c31 100644
--- a/gef.py
+++ b/gef.py
@@ -10902,7 +10902,7 @@ class GefRemoteSessionManager(GefSessionManager):
             return True
         tgt.parent.mkdir(parents=True, exist_ok=True)
         dbg(f"[remote] downloading '{src}' -> '{tgt}'")
-        gdb.execute(f"remote get {src} {tgt.absolute()}")
+        gdb.execute(f"remote get '{src}' '{tgt.absolute()}'")
         return tgt.exists()

     def connect(self, pid: int) -> bool:

Grazfather · 2023-08-25T01:06:31Z

And for the fork issue:

diff --git a/gef.py b/gef.py
index 5435c79..a5a7d35 100644
--- a/gef.py
+++ b/gef.py
@@ -3554,19 +3554,25 @@ def new_objfile_handler(evt: Optional["gdb.NewObjFileEvent"]) -> None:
     return


-def exit_handler(_: "gdb.ExitedEvent") -> None:
+def exit_handler(e: "gdb.ExitedEvent") -> None:
     """GDB event handler for exit cases."""
     global gef
     # flush the caches
     reset_all_caches()

     # disconnect properly the remote session
-    gef.session.qemu_mode = False
-    if gef.session.remote:
-        gef.session.remote.close()
-        del gef.session.remote
-        gef.session.remote = None
-        gef.session.remote_initializing = False
+    if hasattr(e, "exit_code"):
+        gef.session.qemu_mode = False
+        if gef.session.remote:
+            gef.session.remote.close()
+            del gef.session.remote
+            gef.session.remote = None
+            gef.session.remote_initializing = False

Ultimator14 · 2023-08-25T01:53:48Z

Both patches work.
With the first one applied, the stack is also shown with spaces in path. The second one keeps the stack displayed after a fork.
I applied them both and was not able to get another Unmapped address.
TYVM

As I looked into #901, I found that we often don't quote filepaths in `gdb.execute`. This began lazy grep that tries to ID where we do this, and fixes where we can. Noticed that we cannot use quotes in the filename when setting up logging, so it instead restricts the logging path to one without spaces.

Ultimator14 added bug triage labels Oct 19, 2022

Ultimator14 closed this as completed Oct 19, 2022

Ultimator14 reopened this Oct 20, 2022

theguy147 mentioned this issue Oct 20, 2022

fix following child process in remote session #903

Closed

13 tasks

stale bot added the stale label Feb 17, 2023

stale bot removed the stale label Mar 1, 2023

stale bot added the stale label Apr 30, 2023

stale bot removed the stale label May 25, 2023

Angelo942 mentioned this issue Jun 5, 2023

Error "Unable to find the text segment base addr" if space present in binary path mahaloz/decomp2dbg#69

Closed

stale bot added the stale label Jul 24, 2023

stale bot closed this as completed Aug 23, 2023

Grazfather reopened this Aug 24, 2023

hugsy assigned Grazfather Aug 24, 2023

stale bot removed the stale label Aug 24, 2023

This was referenced Aug 25, 2023

gef-remote: Fix issue with remote path having a space #998

Merged

Fix gdb.execute not quoting paths #999

Merged

Grazfather closed this as completed in #998 Aug 25, 2023

Grazfather mentioned this issue Aug 26, 2023

[Bug] gef-remote disconnects when follow-fork-mode is set to child #1001

Closed

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug] Stack is not displayed when attaching to gdbserver inside a container #901

[Bug] Stack is not displayed when attaching to gdbserver inside a container #901

Ultimator14 commented Oct 19, 2022

clubby789 commented Oct 19, 2022

Ultimator14 commented Oct 19, 2022

Ultimator14 commented Oct 20, 2022

theguy147 commented Oct 20, 2022

theguy147 commented Oct 20, 2022

Ultimator14 commented Oct 20, 2022

Ultimator14 commented Dec 19, 2022

stale bot commented Feb 17, 2023

Ultimator14 commented Mar 1, 2023 •

edited

stale bot commented Apr 30, 2023

stale bot commented Jul 24, 2023

stale bot commented Aug 23, 2023

Grazfather commented Aug 24, 2023

Grazfather commented Aug 24, 2023

Ultimator14 commented Aug 24, 2023 •

edited

Grazfather commented Aug 25, 2023

Grazfather commented Aug 25, 2023 •

edited

Grazfather commented Aug 25, 2023 •

edited

Ultimator14 commented Aug 25, 2023

[Bug] Stack is not displayed when attaching to gdbserver inside a container #901

[Bug] Stack is not displayed when attaching to gdbserver inside a container #901

Comments

Ultimator14 commented Oct 19, 2022

GEF+GDB version

Operating System

Describe the issue you encountered

Do you read the docs and look at previously closed issues/PRs for similar cases?

Architecture impacted

Describe your issue. Without a proper reproduction step-by-step, your issue will be ignored.

Minimalist test case

Additional context?

clubby789 commented Oct 19, 2022

Ultimator14 commented Oct 19, 2022

Ultimator14 commented Oct 20, 2022

theguy147 commented Oct 20, 2022

theguy147 commented Oct 20, 2022

Ultimator14 commented Oct 20, 2022

Ultimator14 commented Dec 19, 2022

stale bot commented Feb 17, 2023

Ultimator14 commented Mar 1, 2023 • edited

stale bot commented Apr 30, 2023

stale bot commented Jul 24, 2023

stale bot commented Aug 23, 2023

Grazfather commented Aug 24, 2023

Grazfather commented Aug 24, 2023

Ultimator14 commented Aug 24, 2023 • edited

1. Unmapped address after fork

2. Unmapped address at any point if path contains spaces

Grazfather commented Aug 25, 2023

Grazfather commented Aug 25, 2023 • edited

Grazfather commented Aug 25, 2023 • edited

Ultimator14 commented Aug 25, 2023

Ultimator14 commented Mar 1, 2023 •

edited

Ultimator14 commented Aug 24, 2023 •

edited

Grazfather commented Aug 25, 2023 •

edited

Grazfather commented Aug 25, 2023 •

edited