Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Serial debugging failing to start #5

Closed
joe-graham opened this issue Oct 17, 2018 · 4 comments · Fixed by #11
Closed

Serial debugging failing to start #5

joe-graham opened this issue Oct 17, 2018 · 4 comments · Fixed by #11

Comments

@joe-graham
Copy link
Contributor

This issue was found on a physical ASA 5505 running version 9.1(6).

When enabling gdbserver on the serial interface in asafw, the gdb script generated by asadbg hangs at target extended-remote [serial port].
image
Turning on gdb's remote debugging shows that gdbserver continually prints this string, preventing gdb from attaching:
image

It looks like a fix for this would be modifying the inittab file to run /tmp/start_cmd on a different tty. This will still cause gdbserver to attach to the serial interface, but will result in the error message being printed on the other tty as opposed to over the serial interface. Manually making this change on my end results in asadbg hanging (since it's waiting for the "Remote debugging over /dev/ttyS0" string) but eventually connecting over USB:
image

I'm willing to write the code to implement this change, but I'll hold off on implementing and making a pull request since this is a little more involved than my previous two issues.

@saidelike
Copy link
Collaborator

saidelike commented Oct 18, 2018

Thanks for this.

Good finding about modifying /etc/inittab. So you effectively change: ttyS0::once:/tmp/run_cmd into ttyS1::once:/tmp/run_cmd? Automation could quite easily be tested by adding code to unpack_repack.sh in the custom function and invoking unpack_repack_bin.sh -c .... Once this works we could add a special case in the enable_gdb function for the firmware we know require that (e.g. *"asa91"*). Afaict most of the 9.1.x do require that.

I think if we can find a way to have gdb not hang at all, it will be even better. Not sure if we can tell gdb to not wait for the "Remote debugging using /dev/ttyS0" string. Tbh I didn't even know it worked like that.

What is interesting is that for asa924 for instance, we have this fix (note the comment which is not on asa91x):

# run on the console specified on the kernel start line
::once:/tmp/run_cmd

I am wondering if just changing to this would solve the gdb hanging problem.

Yes, I agree it involves a bit more changes than your previous PR but at the same time is a good exercise to be able to modify the scripts (and not too complex in this case imho) so if that is something that you are interested in, feel free to do so.

Note: In the past, we would use --bin-with-asa-to-inject to e.g. inject the /asa folder from 9.1.x into another firmware such as asa92*.bin but your way is a lot cleaner and does not change the firmware container.

@joe-graham
Copy link
Contributor Author

gdb only hangs because of the "Remote side has terminated connection. GDBserver will reopen the connection" message being printed over the serial line repeatedly. The hang while waiting for "Remote debugging using /dev/ttyS0" is on the asadbg side, since that's one of the strings that comm.py looks for while waiting for the ASA to finish booting (under boot_router_cli_non_rooted and boot_router_cli). I'm not really sure what the fix for that would be, though I would imagine that whatever protocol gdbserver uses to communicate with gdb has a series of characters that it sends over serial to advertise that it is ready for debugging.

I didn't realize that fix already existed for 924, that would work perfectly I think. Instead of directing the output /tmp/run_cmd to another tty, it just gets directed to nowhere.

@saidelike
Copy link
Collaborator

Oh right thanks for explaining that after your fix, the hang is on the asadbg side, not gdb side. The reason there is a hang is because we timeout after 3 minutes in case we don't detect the "Remote debugging using /dev/ttyS0" string which indeed we don't detect following your fix. We should be able to work around that but not sure how yet either.

Good idea about the characters gdbserver may send over serial. I guess we could hexdump what we receive in comm.py instead of just printing ascii characters, to see if there is anything interesting.

In case there isn't any character send, it may be possible to check from comm.py if there is a listening gdbserver on the other end by sending special characters. So in case we read 0 bytes, we could send this special series of bytes from comm.py and check we get the right reply from the other end (gdbserver).

@joe-graham
Copy link
Contributor Author

Definitely worth looking into. I'll get started on implementing the change for unpack_repack_bin.sh this weekend, and open an issue for the comm.py stuff over on the asadbg repo after I do a little more research into the gdbserver stuff. Thanks for your insight :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants