Serial debugging failing to start

[joe-graham]

This issue was found on a physical ASA 5505 running version 9.1(6).

When enabling gdbserver on the serial interface in asafw, the gdb script generated by asadbg hangs at target extended-remote [serial port].

Turning on gdb's remote debugging shows that gdbserver continually prints this string, preventing gdb from attaching:

It looks like a fix for this would be modifying the inittab file to run /tmp/start_cmd on a different tty. This will still cause gdbserver to attach to the serial interface, but will result in the error message being printed on the other tty as opposed to over the serial interface. Manually making this change on my end results in asadbg hanging (since it's waiting for the "Remote debugging over /dev/ttyS0" string) but eventually connecting over USB:

I'm willing to write the code to implement this change, but I'll hold off on implementing and making a pull request since this is a little more involved than my previous two issues.

[saidelike]

Thanks for this.

Good finding about modifying /etc/inittab. So you effectively change: ttyS0::once:/tmp/run_cmd into ttyS1::once:/tmp/run_cmd? Automation could quite easily be tested by adding code to unpack_repack.sh in the custom function and invoking unpack_repack_bin.sh -c .... Once this works we could add a special case in the enable_gdb function for the firmware we know require that (e.g. *"asa91"*). Afaict most of the 9.1.x do require that.

I think if we can find a way to have gdb not hang at all, it will be even better. Not sure if we can tell gdb to not wait for the "Remote debugging using /dev/ttyS0" string. Tbh I didn't even know it worked like that.

What is interesting is that for asa924 for instance, we have this fix (note the comment which is not on asa91x):

# run on the console specified on the kernel start line
::once:/tmp/run_cmd

I am wondering if just changing to this would solve the gdb hanging problem.

Yes, I agree it involves a bit more changes than your previous PR but at the same time is a good exercise to be able to modify the scripts (and not too complex in this case imho) so if that is something that you are interested in, feel free to do so.

Note: In the past, we would use --bin-with-asa-to-inject to e.g. inject the /asa folder from 9.1.x into another firmware such as asa92*.bin but your way is a lot cleaner and does not change the firmware container.

[joe-graham]

gdb only hangs because of the "Remote side has terminated connection. GDBserver will reopen the connection" message being printed over the serial line repeatedly. The hang while waiting for "Remote debugging using /dev/ttyS0" is on the asadbg side, since that's one of the strings that comm.py looks for while waiting for the ASA to finish booting (under boot_router_cli_non_rooted and boot_router_cli). I'm not really sure what the fix for that would be, though I would imagine that whatever protocol gdbserver uses to communicate with gdb has a series of characters that it sends over serial to advertise that it is ready for debugging.

I didn't realize that fix already existed for 924, that would work perfectly I think. Instead of directing the output /tmp/run_cmd to another tty, it just gets directed to nowhere.

[saidelike]

Oh right thanks for explaining that after your fix, the hang is on the asadbg side, not gdb side. The reason there is a hang is because we timeout after 3 minutes in case we don't detect the "Remote debugging using /dev/ttyS0" string which indeed we don't detect following your fix. We should be able to work around that but not sure how yet either.

Good idea about the characters gdbserver may send over serial. I guess we could hexdump what we receive in comm.py instead of just printing ascii characters, to see if there is anything interesting.

In case there isn't any character send, it may be possible to check from comm.py if there is a listening gdbserver on the other end by sending special characters. So in case we read 0 bytes, we could send this special series of bytes from comm.py and check we get the right reply from the other end (gdbserver).

[joe-graham]

Definitely worth looking into. I'll get started on implementing the change for unpack_repack_bin.sh this weekend, and open an issue for the comm.py stuff over on the asadbg repo after I do a little more research into the gdbserver stuff. Thanks for your insight :)