Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make handles not inheritable to prevent from blocking durning tab-completion #852

Merged

Conversation

iuhilnehc-ynos
Copy link

@iuhilnehc-ynos iuhilnehc-ynos commented Sep 8, 2023

to fix #851

Please refer to the #852 (comment)

@fujitatomoya
Copy link
Collaborator

(TODO: why?)

big TODO though 😅 i will take a look.

@fujitatomoya fujitatomoya self-assigned this Sep 8, 2023
@fujitatomoya fujitatomoya self-requested a review September 8, 2023 05:38
@iuhilnehc-ynos
Copy link
Author

This is definitely not the final solution as Foxy works well with the spawn_daemon.

@iuhilnehc-ynos
Copy link
Author

@fujitatomoya Sorry, I need to turn it into a draft.

@iuhilnehc-ynos iuhilnehc-ynos marked this pull request as draft September 8, 2023 05:58
@iuhilnehc-ynos
Copy link
Author

iuhilnehc-ynos commented Sep 8, 2023

Mark: It's related to #652.

@fujitatomoya
Copy link
Collaborator

@hidmic by any chance, do you have any idea why #652 leads to #851?

@hidmic
Copy link
Contributor

hidmic commented Sep 11, 2023

@fujitatomoya hmm, whatever it is, it is not obvious. Perhaps something funky about processes started to gather autocompletion options? A backtrace at the time of blocking would help.

@iuhilnehc-ynos
Copy link
Author

iuhilnehc-ynos commented Sep 13, 2023

Let me share my understanding,

  1. When the Tab key is pressed for ros2 topic echo /<tab> in the bash, the bash will fork a child bash to do the complete
gen_completion_matches

(gdb) bt
#0  __libc_fork () at ./posix/fork.c:41
#1  0x00005602d660c6c4 in make_child (command=command@entry=0x0, flags=4) at jobs.c:2175
#2  0x00005602d6619e29 in command_substitute (
    string=string@entry=0x5602d84aca10 "IFS=\"$IFS\"", ' ' <repeats 19 times>, "COMP_LINE=\"$COMP_LINE\"", ' ' <repeats 19 times>, "COMP_POINT=\"$COMP_POINT\"", ' ' <repeats 19 times>, "COMP_TYPE=\"$COMP_TYPE\"", ' ' <repeats 19 times>, "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREAKS"..., 
    quoted=quoted@entry=0, flags=flags@entry=0) at subst.c:6371
#3  0x00005602d661c8c9 in param_expand (
    string=string@entry=0x5602d81ba610 "$(IFS=\"$IFS\"", ' ' <repeats 19 times>, "COMP_LINE=\"$COMP_LINE\"", ' ' <repeats 19 times>, "COMP_POINT=\"$COMP_POINT\"", ' ' <repeats 19 times>, "COMP_TYPE=\"$COMP_TYPE\"", ' ' <repeats 19 times>, "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., 
    sindex=sindex@entry=0x7ffe56c1be54, quoted=quoted@entry=0, expanded_something=expanded_something@entry=0x7ffe56c1bf18, 
    contains_dollar_at=contains_dollar_at@entry=0x7ffe56c1be60, quoted_dollar_at_p=quoted_dollar_at_p@entry=0x7ffe56c1be58, 
    had_quoted_null_p=0x7ffe56c1be5c, pflags=0) at subst.c:9863
#4  0x00005602d661fa8c in expand_word_internal (word=<optimized out>, quoted=quoted@entry=0, isexp=isexp@entry=0, 
    contains_dollar_at=contains_dollar_at@entry=0x7ffe56c1bf1c, expanded_something=expanded_something@entry=0x7ffe56c1bf18) at subst.c:10329
#5  0x00005602d6621e1d in shell_expand_word_list (tlist=<optimized out>, eflags=30) at subst.c:11890
#6  0x00005602d6626af5 in expand_word_list_internal (eflags=30, list=0x5602d8495890) at subst.c:12014
#7  expand_words_no_vars (list=list@entry=0x5602d8495890) at subst.c:11366
#8  0x00005602d6631d60 in expand_compound_array_assignment (var=var@entry=0x5602d81b6010, 
    value=value@entry=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' <repeats 19 times>, "COMP_LINE=\"$COMP_LINE\"", ' ' <repeats 19 times>, "COMP_POINT=\"$COMP_POINT\"", ' ' <repeats 19 times>, "COMP_TYPE=\"$COMP_TYPE\"", ' ' <repeats 19 times>, "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., 
    flags=flags@entry=0) at arrayfunc.c:547
#9  0x00005602d66330ef in assign_array_var_from_string (flags=0, 
    value=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' <repeats 19 times>, "COMP_LINE=\"$COMP_LINE\"", ' ' <repeats 19 times>, "COMP_POINT=\"$COMP_POINT\"", ' ' <repeats 19 times>, "COMP_TYPE=\"$COMP_TYPE\"", ' ' <repeats 19 times>, "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., var=0x5602d81b6010)
    at arrayfunc.c:820
#10 assign_array_var_from_string (flags=0, 
    value=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' <repeats 19 times>, "COMP_LINE=\"$COMP_LINE\"", ' ' <repeats 19 times>, "COMP_POINT=\"$COMP_POINT\"", ' ' <repeats 19 times>, "COMP_TYPE=\"$COMP_TYPE\"", ' ' <repeats 19 times>, "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., var=0x5602d81b6010)
    at arrayfunc.c:810
#11 assign_array_from_string (name=name@entry=0x5602d84aba10 "COMPREPLY", 
    value=value@entry=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' <repeats 19 times>, "COMP_LINE=\"$COMP_LINE\"", ' ' <repeats 19 times>, "COMP_POINT=\"$COMP_POINT\"", ' ' <repeats 19 times>, "COMP_TYPE=\"$COMP_TYPE\"", ' ' <repeats 19 times>, "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., 
    flags=flags@entry=0) at arrayfunc.c:473
#12 0x00005602d661b70f in do_compound_assignment (flags=0, 
    value=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' <repeats 19 times>, "COMP_LINE=\"$COMP_LINE\"", ' ' <repeats 19 times>, "COMP_POINT=\"$COMP_POINT\"", ' ' <repeats 19 times>, "COMP_TYPE=\"$COMP_TYPE\"", ' ' <repeats 19 times>, "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., 
    name=0x5602d84aba10 "COMPREPLY") at subst.c:3190
--Type <RET> for more, q to quit, c to continue without paging--c
#13 do_assignment_internal (word=<optimized out>, expand=expand@entry=1) at subst.c:3298
#14 0x00005602d66269ea in do_word_assignment (flags=0, word=<optimized out>) at subst.c:3345
#15 expand_word_list_internal (eflags=31, list=<optimized out>) at subst.c:11980
#16 expand_words (list=<optimized out>) at subst.c:11357
#17 0x00005602d65f72bc in execute_simple_command (fds_to_close=0x5602d849f510, async=0, pipe_out=-1, pipe_in=-1, simple_command=<optimized out>) at execute_cmd.c:4381
#18 execute_command_internal (command=0x5602d8498fd0, asynchronous=<optimized out>, pipe_in=-1, pipe_out=<optimized out>, fds_to_close=0x5602d849f510) at execute_cmd.c:846
#19 0x00005602d65f9e88 in execute_connection (command=0x5602d849c310, asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d849f510) at execute_cmd.c:2718
#20 0x00005602d65f555a in execute_command_internal (command=0x5602d849c310, asynchronous=<optimized out>, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d849f510) at execute_cmd.c:1020
#21 0x00005602d65f8207 in execute_command (command=0x5602d849c310) at execute_cmd.c:395
#22 0x00005602d65f9e4b in execute_connection (command=0x5602d8159610, asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d849a650) at execute_cmd.c:2709
#23 0x00005602d65f555a in execute_command_internal (command=0x5602d8159610, asynchronous=<optimized out>, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d849a650) at execute_cmd.c:1020
#24 0x00005602d65f55a2 in execute_command_internal (command=0x5602d849d3d0, asynchronous=<optimized out>, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d849a650) at execute_cmd.c:1012
#25 0x00005602d65f957c in execute_function (var=var@entry=0x5602d84a9210, words=words@entry=0x5602d8499210, flags=flags@entry=0, fds_to_close=fds_to_close@entry=0x5602d849a650, async=async@entry=0, subshell=subshell@entry=0) at execute_cmd.c:5103
#26 0x00005602d65f96ec in execute_shell_function (var=var@entry=0x5602d84a9210, words=words@entry=0x5602d8499210) at execute_cmd.c:5162
#27 0x00005602d66464d5 in gen_shell_function_matches (cs=0x5602d848e010, cs=0x5602d848e010, nw=<optimized out>, foundp=<synthetic pointer>, cw=<optimized out>, lwords=0x5602d84924d0, ind=<optimized out>, line=0x5602d818e590 "ros2 topic echo /", text=0x5602d81bc9d0 "/", cmd=0x5602d848cc10 "ros2") at pcomplete.c:1155
#28 gen_compspec_completions (cs=cs@entry=0x5602d848e010, cmd=cmd@entry=0x5602d848cc10 "ros2", word=word@entry=0x5602d81bc9d0 "/", start=start@entry=0, end=end@entry=17, foundp=foundp@entry=0x7ffe56c1c998) at pcomplete.c:1422
#29 0x00005602d664702c in gen_progcomp_completions (ocmd=ocmd@entry=0x5602d848cc10 "ros2", cmd=cmd@entry=0x5602d848cc10 "ros2", word=word@entry=0x5602d81bc9d0 "/", start=start@entry=0, end=end@entry=17, foundp=foundp@entry=0x7ffe56c1c998, retryp=0x7ffe56c1c99c, lastcs=0x7ffe56c1c9a0) at pcomplete.c:1594
#30 0x00005602d66471d6 in programmable_completions (cmd=cmd@entry=0x5602d848cc10 "ros2", word=word@entry=0x5602d81bc9d0 "/", start=start@entry=0, end=end@entry=17, foundp=foundp@entry=0x7ffe56c1ca34) at pcomplete.c:1649
#31 0x00005602d663f7af in attempt_shell_completion (text=0x5602d81bc9d0 "/", start=<optimized out>, end=17) at bashline.c:1665
#32 0x00005602d667dfbc in gen_completion_matches (text=text@entry=0x5602d81bc9d0 "/", start=start@entry=16, end=end@entry=17, our_func=our_func@entry=0x5602d667b780 <rl_filename_completion_function>, found_quote=<optimized out>, quote_char=<optimized out>) at complete.c:1222
#33 0x00005602d667e1ea in rl_complete_internal (what_to_do=9) at complete.c:2031
#34 0x00005602d66737cb in _rl_dispatch_subseq (key=<optimized out>, map=0x5602d66e22e0 <emacs_standard_keymap>, got_subseq=<optimized out>) at readline.c:887
#35 0x00005602d6673d98 in _rl_dispatch (map=<optimized out>, key=<optimized out>) at readline.c:833
#36 readline_internal_char () at readline.c:645
#37 0x00005602d667471d in readline_internal_charloop () at readline.c:694
#38 readline_internal () at readline.c:706
#39 readline (prompt=<optimized out>) at readline.c:385
#40 0x00005602d65de056 in yy_readline_get () at /Users/chet/src/bash/src/parse.y:1488
#41 0x00005602d65e0ac0 in yy_getc () at /Users/chet/src/bash/src/parse.y:1422
#42 shell_getc (remove_quoted_newline=remove_quoted_newline@entry=1) at /Users/chet/src/bash/src/parse.y:2358
#43 0x00005602d65e407a in read_token (command=<optimized out>) at /Users/chet/src/bash/src/parse.y:3290
#44 0x00005602d65e75e2 in read_token (command=0) at /Users/chet/src/bash/src/parse.y:3254
#45 yylex () at /Users/chet/src/bash/src/parse.y:2797
#46 yyparse () at y.tab.c:1835
#47 0x00005602d65dd56b in parse_command () at eval.c:347
#48 0x00005602d65dd707 in read_command () at eval.c:391
#49 0x00005602d65dd92a in reader_loop () at eval.c:138
#50 0x00005602d65dc583 in main (argc=1, argv=0x7ffe56c1de38, env=0x7ffe56c1de48) at shell.c:811

  1. The child bash continue to fork to execute /usr/bin/python3 /home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2 to call the TopicNameCompleter
waitchld

(gdb) bt
#0  0x00007f4d764ea45a in __GI___wait4 (pid=pid@entry=-1, stat_loc=stat_loc@entry=0x7ffe56c1b560, options=options@entry=0, usage=usage@entry=0x0)
    at ../sysdeps/unix/sysv/linux/wait4.c:30
#1  0x00007f4d764ea41b in __GI___waitpid (pid=pid@entry=-1, stat_loc=stat_loc@entry=0x7ffe56c1b560, options=options@entry=0) at ./posix/waitpid.c:38
#2  0x00005602d660b6ca in waitchld (block=block@entry=1, wpid=<optimized out>) at jobs.c:3812
#3  0x00005602d660cf8a in wait_for (pid=2731517, flags=flags@entry=0) at jobs.c:2992
#4  0x00005602d65f7bc2 in execute_command_internal (command=0x5602d849a990, asynchronous=asynchronous@entry=0, pipe_in=pipe_in@entry=-1, 
    pipe_out=pipe_out@entry=-1, fds_to_close=fds_to_close@entry=0x5602d81bd890) at execute_cmd.c:891
#5  0x00005602d664f979 in parse_and_execute (
    string=string@entry=0x5602d84aca10 "IFS=\"$IFS\"", ' ' <repeats 19 times>, "COMP_LINE=\"$COMP_LINE\"", ' ' <repeats 19 times>, "COMP_POINT=\"$COMP_POINT\"", ' ' <repeats 19 times>, "COMP_TYPE=\"$COMP_TYPE\"", ' ' <repeats 19 times>, "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREAKS"..., 
    from_file=from_file@entry=0x5602d669eac5 "command substitution", flags=flags@entry=20) at evalstring.c:489
#6  0x00005602d661a5b4 in command_substitute (
    string=string@entry=0x5602d84aca10 "IFS=\"$IFS\"", ' ' <repeats 19 times>, "COMP_LINE=\"$COMP_LINE\"", ' ' <repeats 19 times>, "COMP_POINT=\"$COMP_POINT\"", ' ' <repeats 19 times>, "COMP_TYPE=\"$COMP_TYPE\"", ' ' <repeats 19 times>, "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREAKS"..., 
    quoted=quoted@entry=0, flags=flags@entry=0) at subst.c:6517
#7  0x00005602d661c8c9 in param_expand (
    string=string@entry=0x5602d81ba610 "$(IFS=\"$IFS\"", ' ' <repeats 19 times>, "COMP_LINE=\"$COMP_LINE\"", ' ' <repeats 19 times>, "COMP_POINT=\"$COMP_POINT\"", ' ' <repeats 19 times>, "COMP_TYPE=\"$COMP_TYPE\"", ' ' <repeats 19 times>, "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., 
    sindex=sindex@entry=0x7ffe56c1be54, quoted=quoted@entry=0, expanded_something=expanded_something@entry=0x7ffe56c1bf18, 
    contains_dollar_at=contains_dollar_at@entry=0x7ffe56c1be60, quoted_dollar_at_p=quoted_dollar_at_p@entry=0x7ffe56c1be58, 
    had_quoted_null_p=0x7ffe56c1be5c, pflags=0) at subst.c:9863
#8  0x00005602d661fa8c in expand_word_internal (word=<optimized out>, quoted=quoted@entry=0, isexp=isexp@entry=0, 
    contains_dollar_at=contains_dollar_at@entry=0x7ffe56c1bf1c, expanded_something=expanded_something@entry=0x7ffe56c1bf18) at subst.c:10329
#9  0x00005602d6621e1d in shell_expand_word_list (tlist=<optimized out>, eflags=30) at subst.c:11890
#10 0x00005602d6626af5 in expand_word_list_internal (eflags=30, list=0x5602d8495890) at subst.c:12014
#11 expand_words_no_vars (list=list@entry=0x5602d8495890) at subst.c:11366
#12 0x00005602d6631d60 in expand_compound_array_assignment (var=var@entry=0x5602d81b6010, 
    value=value@entry=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' <repeats 19 times>, "COMP_LINE=\"$COMP_LINE\"", ' ' <repeats 19 times>, "COMP_POINT=\"$COMP_POINT\"", ' ' <repeats 19 times>, "COMP_TYPE=\"$COMP_TYPE\"", ' ' <repeats 19 times>, "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., 
    flags=flags@entry=0) at arrayfunc.c:547
#13 0x00005602d66330ef in assign_array_var_from_string (flags=0, 
    value=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' <repeats 19 times>, "COMP_LINE=\"$COMP_LINE\"", ' ' <repeats 19 times>, "COMP_POINT=\"$COMP_POINT\"", ' ' <repeats 19 times>, "COMP_TYPE=\"$COMP_TYPE\"", ' ' <repeats 19 times>, "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., var=0x5602d81b6010)
    at arrayfunc.c:820
#14 assign_array_var_from_string (flags=0, 
    value=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' <repeats 19 times>, "COMP_LINE=\"$COMP_LINE\"", ' ' <repeats 19 times>, "COMP_POINT=\"$COMP_POINT\"", ' ' <repeats 19 times>, "COMP_TYPE=\"$COMP_TYPE\"", ' ' <repeats 19 times>, "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., var=0x5602d81b6010)
--Type <RET> for more, q to quit, c to continue without paging--c
    at arrayfunc.c:810
#15 assign_array_from_string (name=name@entry=0x5602d84aba10 "COMPREPLY", value=value@entry=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' <repeats 19 times>, "COMP_LINE=\"$COMP_LINE\"", ' ' <repeats 19 times>, "COMP_POINT=\"$COMP_POINT\"", ' ' <repeats 19 times>, "COMP_TYPE=\"$COMP_TYPE\"", ' ' <repeats 19 times>, "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., flags=flags@entry=0) at arrayfunc.c:473
#16 0x00005602d661b70f in do_compound_assignment (flags=0, value=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' <repeats 19 times>, "COMP_LINE=\"$COMP_LINE\"", ' ' <repeats 19 times>, "COMP_POINT=\"$COMP_POINT\"", ' ' <repeats 19 times>, "COMP_TYPE=\"$COMP_TYPE\"", ' ' <repeats 19 times>, "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., name=0x5602d84aba10 "COMPREPLY") at subst.c:3190
#17 do_assignment_internal (word=<optimized out>, expand=expand@entry=1) at subst.c:3298
#18 0x00005602d66269ea in do_word_assignment (flags=0, word=<optimized out>) at subst.c:3345
#19 expand_word_list_internal (eflags=31, list=<optimized out>) at subst.c:11980
#20 expand_words (list=<optimized out>) at subst.c:11357
#21 0x00005602d65f72bc in execute_simple_command (fds_to_close=0x5602d849f510, async=0, pipe_out=-1, pipe_in=-1, simple_command=<optimized out>) at execute_cmd.c:4381
#22 execute_command_internal (command=0x5602d8498fd0, asynchronous=<optimized out>, pipe_in=-1, pipe_out=<optimized out>, fds_to_close=0x5602d849f510) at execute_cmd.c:846
#23 0x00005602d65f9e88 in execute_connection (command=0x5602d849c310, asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d849f510) at execute_cmd.c:2718
#24 0x00005602d65f555a in execute_command_internal (command=0x5602d849c310, asynchronous=<optimized out>, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d849f510) at execute_cmd.c:1020
#25 0x00005602d65f8207 in execute_command (command=0x5602d849c310) at execute_cmd.c:395
#26 0x00005602d65f9e4b in execute_connection (command=0x5602d8159610, asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d849a650) at execute_cmd.c:2709
#27 0x00005602d65f555a in execute_command_internal (command=0x5602d8159610, asynchronous=<optimized out>, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d849a650) at execute_cmd.c:1020
#28 0x00005602d65f55a2 in execute_command_internal (command=0x5602d849d3d0, asynchronous=<optimized out>, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d849a650) at execute_cmd.c:1012
#29 0x00005602d65f957c in execute_function (var=var@entry=0x5602d84a9210, words=words@entry=0x5602d8499210, flags=flags@entry=0, fds_to_close=fds_to_close@entry=0x5602d849a650, async=async@entry=0, subshell=subshell@entry=0) at execute_cmd.c:5103
#30 0x00005602d65f96ec in execute_shell_function (var=var@entry=0x5602d84a9210, words=words@entry=0x5602d8499210) at execute_cmd.c:5162
#31 0x00005602d66464d5 in gen_shell_function_matches (cs=0x5602d848e010, cs=0x5602d848e010, nw=<optimized out>, foundp=<synthetic pointer>, cw=<optimized out>, lwords=0x5602d84924d0, ind=<optimized out>, line=0x5602d818e590 "ros2 topic echo /", text=0x5602d81bc9d0 "/", cmd=0x5602d848cc10 "ros2") at pcomplete.c:1155
#32 gen_compspec_completions (cs=cs@entry=0x5602d848e010, cmd=cmd@entry=0x5602d848cc10 "ros2", word=word@entry=0x5602d81bc9d0 "/", start=start@entry=0, end=end@entry=17, foundp=foundp@entry=0x7ffe56c1c998) at pcomplete.c:1422
#33 0x00005602d664702c in gen_progcomp_completions (ocmd=ocmd@entry=0x5602d848cc10 "ros2", cmd=cmd@entry=0x5602d848cc10 "ros2", word=word@entry=0x5602d81bc9d0 "/", start=start@entry=0, end=end@entry=17, foundp=foundp@entry=0x7ffe56c1c998, retryp=0x7ffe56c1c99c, lastcs=0x7ffe56c1c9a0) at pcomplete.c:1594
#34 0x00005602d66471d6 in programmable_completions (cmd=cmd@entry=0x5602d848cc10 "ros2", word=word@entry=0x5602d81bc9d0 "/", start=start@entry=0, end=end@entry=17, foundp=foundp@entry=0x7ffe56c1ca34) at pcomplete.c:1649
#35 0x00005602d663f7af in attempt_shell_completion (text=0x5602d81bc9d0 "/", start=<optimized out>, end=17) at bashline.c:1665
#36 0x00005602d667dfbc in gen_completion_matches (text=text@entry=0x5602d81bc9d0 "/", start=start@entry=16, end=end@entry=17, our_func=our_func@entry=0x5602d667b780 <rl_filename_completion_function>, found_quote=<optimized out>, quote_char=<optimized out>) at complete.c:1222
#37 0x00005602d667e1ea in rl_complete_internal (what_to_do=9) at complete.c:2031
#38 0x00005602d66737cb in _rl_dispatch_subseq (key=<optimized out>, map=0x5602d66e22e0 <emacs_standard_keymap>, got_subseq=<optimized out>) at readline.c:887
#39 0x00005602d6673d98 in _rl_dispatch (map=<optimized out>, key=<optimized out>) at readline.c:833
#40 readline_internal_char () at readline.c:645
#41 0x00005602d667471d in readline_internal_charloop () at readline.c:694
#42 readline_internal () at readline.c:706
#43 readline (prompt=<optimized out>) at readline.c:385
#44 0x00005602d65de056 in yy_readline_get () at /Users/chet/src/bash/src/parse.y:1488
#45 0x00005602d65e0ac0 in yy_getc () at /Users/chet/src/bash/src/parse.y:1422
#46 shell_getc (remove_quoted_newline=remove_quoted_newline@entry=1) at /Users/chet/src/bash/src/parse.y:2358
#47 0x00005602d65e407a in read_token (command=<optimized out>) at /Users/chet/src/bash/src/parse.y:3290
#48 0x00005602d65e75e2 in read_token (command=0) at /Users/chet/src/bash/src/parse.y:3254
#49 yylex () at /Users/chet/src/bash/src/parse.y:2797
#50 yyparse () at y.tab.c:1835
#51 0x00005602d65dd56b in parse_command () at eval.c:347
#52 0x00005602d65dd707 in read_command () at eval.c:391
#53 0x00005602d65dd92a in reader_loop () at eval.c:138
#54 0x00005602d65dc583 in main (argc=1, argv=0x7ffe56c1de38, env=0x7ffe56c1de48) at shell.c:811

  1. And then TopicNameCompleter will call the spawn_daemon
call spawn_daemon

(gdb) bt
#0  0x00007fe55b71b7ed in __GI___select (nfds=nfds@entry=0, readfds=readfds@entry=0x0, writefds=writefds@entry=0x0, exceptfds=exceptfds@entry=0x0, 
    timeout=timeout@entry=0x7ffcb21d6600) at ../sysdeps/unix/sysv/linux/select.c:69
#1  0x0000560fb71b6766 in pysleep (secs=<optimized out>) at ../Modules/timemodule.c:2076
#2  time_sleep (self=<optimized out>, obj=<optimized out>) at ../Modules/timemodule.c:370
#3  0x0000560fb709a854 in cfunction_vectorcall_O (func=<built-in method sleep of module object at remote 0x7fe55b50fe20>, args=0x560fb97abb28, 
    nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/methodobject.c:516
#4  0x0000560fb70898a2 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x560fb97abb28, 
    callable=<built-in method sleep of module object at remote 0x7fe55b50fe20>, tstate=0x560fb90c0840) at ../Include/cpython/abstract.h:114
#5  PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x560fb97abb28, 
    callable=<built-in method sleep of module object at remote 0x7fe55b50fe20>) at ../Include/cpython/abstract.h:123
#6  call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffcb21d6790, tstate=<optimized out>)
    at ../Python/ceval.c:5893
#7  _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:4181
#8  0x0000560fb709b70c in _PyEval_EvalFrame (throwflag=0, 
    f=Frame 0x560fb97ab940, for file /home/chenlh/Projects/ROS2/ros2-master/build/ros2cli/ros2cli/daemon/daemonize.py, line 138, in daemonize (callable_=<functools.partial at remote 0x7fe530b815d0>, tags={'name': 'ros2-daemon', 'ros_domain_id': 28, 'rmw_implementation': 'rmw_fastrtps_cpp'}, timeout=None, debug=False, prog='from ros2cli.daemon.daemonize import main; main()', cmd=['/usr/bin/python3', '-c', 'from ros2cli.daemon.daemonize import main; main()', '--name', 'ros2-daemon', '--ros-domain-id', '28', '--rmw-implementation', 'rmw_fastrtps_cpp'], name='rmw_implementation', value='rmw_fastrtps_cpp', flag='--rmw-implementation', kwargs={'stdin': -1, 'stdout': -3, 'stderr': -3, 'close_fds': False}, process=<Popen(_waitpid_lock=<_thread.lock at remote 0x7fe55ad07100>, _input=None, _communication_started=False, args=[...], stdin=<_io.BufferedWriter at remote 0x7fe530b771c0>, stdout=None, stderr=None, pid=2731550, returncode=None, encoding=None, errors=None, pipesize=-1, _devnull=6, text_mode=None, _sigint_wait_secs=<...(truncated), tstate=0x560fb90c0840)
    at ../Include/internal/pycore_ceval.h:46
#9  _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7fe530b7a720, tstate=0x560fb90c0840)
    at ../Python/ceval.c:5067
#10 _PyFunction_Vectorcall (func=<function at remote 0x7fe530b7a710>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>)
    at ../Objects/call.c:342
#11 0x0000560fb70850d1 in _PyObject_VectorcallTstate (kwnames=('tags', 'timeout', 'debug'), nargsf=<optimized out>, args=<optimized out>, 
    callable=<function at remote 0x7fe530b7a710>, tstate=0x560fb90c0840) at ../Include/cpython/abstract.h:114
#12 PyObject_Vectorcall (kwnames=('tags', 'timeout', 'debug'), nargsf=<optimized out>, args=<optimized out>, 
    callable=<function at remote 0x7fe530b7a710>) at ../Include/cpython/abstract.h:123
#13 call_function (kwnames=('tags', 'timeout', 'debug'), oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffcb21d6970, 
    tstate=<optimized out>) at ../Python/ceval.c:5893
#14 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:4231
#15 0x0000560fb709b70c in _PyEval_EvalFrame (throwflag=0, 
    f=Frame 0x7fe559c15e40, for file /home/chenlh/Projects/ROS2/ros2-master/build/ros2cli/ros2cli/node/daemon.py, line 139, in spawn_daemon (args=<Namespace(use_python_default_buffering=False,  command='topic', include_hidden_topics=False,  verb='echo', _command=<TopicCommand(NAME='topic', _subparser=<MonkeyPatchedIntrospectiveArgumentParser(description='Various topic related sub-commands', argument_default=None, prefix_chars='-', conflict_handler='error'--Type <RET> for more, q to quit, c to continue without paging--c
, _registries={'action': {None: <type at remote 0x560fb9271b00>, 'store': <type at remote 0x560fb9271b00>, 'store_const': <type at remote 0x560fb9271ec0>, 'store_true': <type at remote 0x560fb9272280>, 'store_false': <type at remote 0x560fb9272640>, 'append': <type at remote 0x560fb9272a00>, 'append_const': <type at remote 0x560fb9272dc0>, 'count': <type at remote 0x560fb9273180>, 'help': <type at remote 0x560fb9273540>, 'version': <type at remote 0x560fb9273900>, 'parsers': <type at remote 0x560fb9274570>, 'extend': <type at remote 0x560fb9274930>}, 'type': {None:...(truncated), tstate=0x560fb90c0840) at ../Include/internal/pycore_ceval.h:46
#16 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7fe530b7aba0, tstate=0x560fb90c0840) at ../Python/ceval.c:5067
#17 _PyFunction_Vectorcall (func=<function at remote 0x7fe530b7ab90>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:342
#18 0x0000560fb7083e0d in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fe5575877c0, callable=<function at remote 0x7fe530b7ab90>, tstate=0x560fb90c0840) at ../Include/cpython/abstract.h:114
#19 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fe5575877c0, callable=<function at remote 0x7fe530b7ab90>) at ../Include/cpython/abstract.h:123
#20 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffcb21d6b50, tstate=<optimized out>) at ../Python/ceval.c:5893
#21 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:4213
#22 0x0000560fb7090784 in _PyEval_EvalFrame (throwflag=0, f=Frame 0x7fe557587640, for file /home/chenlh/Projects/ROS2/ros2-master/build/ros2cli/ros2cli/node/strategy.py, line 35, in __init__ (self=<NodeStrategy at remote 0x7fe55ae91c60>, args=<Namespace(use_python_default_buffering=False,  command='topic', include_hidden_topics=False,  verb='echo', _command=<TopicCommand(NAME='topic', _subparser=<MonkeyPatchedIntrospectiveArgumentParser(description='Various topic related sub-commands', argument_default=None, prefix_chars='-', conflict_handler='error', _registries={'action': {None: <type at remote 0x560fb9271b00>, 'store': <type at remote 0x560fb9271b00>, 'store_const': <type at remote 0x560fb9271ec0>, 'store_true': <type at remote 0x560fb9272280>, 'store_false': <type at remote 0x560fb9272640>, 'append': <type at remote 0x560fb9272a00>, 'append_const': <type at remote 0x560fb9272dc0>, 'count': <type at remote 0x560fb9273180>, 'help': <type at remote 0x560fb9273540>, 'version': <type at remote 0x560fb9273900>, 'parsers': <type at remote 0x560fb9274570>, 'extend': <type...(truncated), tstate=0x560fb90c0840) at ../Include/internal/pycore_ceval.h:46
#23 _PyEval_Vector (kwnames=0x0, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7fe530b7acc0, tstate=0x560fb90c0840) at ../Python/ceval.c:5067
#24 _PyFunction_Vectorcall (kwnames=0x0, nargsf=<optimized out>, stack=<optimized out>, func=<function at remote 0x7fe530b7acb0>) at ../Objects/call.c:342
#25 _PyObject_FastCallDictTstate (tstate=0x560fb90c0840, callable=<function at remote 0x7fe530b7acb0>, args=<optimized out>, nargsf=<optimized out>, kwargs=<optimized out>) at ../Objects/call.c:142
#26 0x0000560fb70a5744 in _PyObject_Call_Prepend (kwargs=0x0, args=(<Namespace(use_python_default_buffering=False,  command='topic', include_hidden_topics=False,  verb='echo', _command=<TopicCommand(NAME='topic', _subparser=<MonkeyPatchedIntrospectiveArgumentParser(description='Various topic related sub-commands', argument_default=None, prefix_chars='-', conflict_handler='error', _registries={'action': {None: <type at remote 0x560fb9271b00>, 'store': <type at remote 0x560fb9271b00>, 'store_const': <type at remote 0x560fb9271ec0>, 'store_true': <type at remote 0x560fb9272280>, 'store_false': <type at remote 0x560fb9272640>, 'append': <type at remote 0x560fb9272a00>, 'append_const': <type at remote 0x560fb9272dc0>, 'count': <type at remote 0x560fb9273180>, 'help': <type at remote 0x560fb9273540>, 'version': <type at remote 0x560fb9273900>, 'parsers': <type at remote 0x560fb9274570>, 'extend': <type at remote 0x560fb9274930>}, 'type': {None: <function at remote 0x7fe55b019d80>}}, _actions=[<IntrospectAction(option_strings=['-h', '--help'], dest='help', nargs=0, const=None, defa...(truncated), obj=<NodeStrategy at remote 0x7fe55ae91c60>, callable=<function at remote 0x7fe530b7acb0>, tstate=0x560fb90c0840) at ../Objects/call.c:431
#27 slot_tp_init (self=self@entry=<NodeStrategy at remote 0x7fe55ae91c60>, args=args@entry=(<Namespace(use_python_default_buffering=False,  command='topic', include_hidden_topics=False,  verb='echo', _command=<TopicCommand(NAME='topic', _subparser=<MonkeyPatchedIntrospectiveArgumentParser(description='Various topic related sub-commands', argument_default=None, prefix_chars='-', conflict_handler='error', _registries={'action': {None: <type at remote 0x560fb9271b00>, 'store': <type at remote 0x560fb9271b00>, 'store_const': <type at remote 0x560fb9271ec0>, 'store_true': <type at remote 0x560fb9272280>, 'store_false': <type at remote 0x560fb9272640>, 'append': <type at remote 0x560fb9272a00>, 'append_const': <type at remote 0x560fb9272dc0>, 'count': <type at remote 0x560fb9273180>, 'help': <type at remote 0x560fb9273540>, 'version': <type at remote 0x560fb9273900>, 'parsers': <type at remote 0x560fb9274570>, 'extend': <type at remote 0x560fb9274930>}, 'type': {None: <function at remote 0x7fe55b019d80>}}, _actions=[<IntrospectAction(option_strings=['-h', '--help'], dest='help', nargs=0, const=None, defa...(truncated), kwds=kwds@entry=0x0) at ../Objects/typeobject.c:7734
#28 0x0000560fb709158c in type_call (kwds=0x0, args=(<Namespace(use_python_default_buffering=False,  command='topic', include_hidden_topics=False,  verb='echo', _command=<TopicCommand(NAME='topic', _subparser=<MonkeyPatchedIntrospectiveArgumentParser(description='Various topic related sub-commands', argument_default=None, prefix_chars='-', conflict_handler='error', _registries={'action': {None: <type at remote 0x560fb9271b00>, 'store': <type at remote 0x560fb9271b00>, 'store_const': <type at remote 0x560fb9271ec0>, 'store_true': <type at remote 0x560fb9272280>, 'store_false': <type at remote 0x560fb9272640>, 'append': <type at remote 0x560fb9272a00>, 'append_const': <type at remote 0x560fb9272dc0>, 'count': <type at remote 0x560fb9273180>, 'help': <type at remote 0x560fb9273540>, 'version': <type at remote 0x560fb9273900>, 'parsers': <type at remote 0x560fb9274570>, 'extend': <type at remote 0x560fb9274930>}, 'type': {None: <function at remote 0x7fe55b019d80>}}, _actions=[<IntrospectAction(option_strings=['-h', '--help'], dest='help', nargs=0, const=None, defa...(truncated), type=<optimized out>) at ../Objects/typeobject.c:1135
#29 _PyObject_MakeTpCall (tstate=0x560fb90c0840, callable=<type at remote 0x560fb97ebe40>, args=<optimized out>, nargs=<optimized out>, keywords=0x0) at ../Objects/call.c:215
#30 0x0000560fb7089c66 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fe5595d7290, callable=<optimized out>, tstate=<optimized out>) at ../Include/cpython/abstract.h:112
#31 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fe5595d7290, callable=<optimized out>, tstate=<optimized out>) at ../Include/cpython/abstract.h:99
#32 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fe5595d7290, callable=<optimized out>) at ../Include/cpython/abstract.h:123
#33 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffcb21d6e60, tstate=<optimized out>) at ../Python/ceval.c:5893
#34 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:4213
#35 0x0000560fb709b70c in _PyEval_EvalFrame (throwflag=0, f=Frame 0x7fe5595d7100, for file /home/chenlh/Projects/ROS2/ros2-master/build/ros2topic/ros2topic/api/__init__.py, line 74, in __call__ (self=<TopicNameCompleter(include_hidden_topics_key='include_hidden_topics') at remote 0x7fe530b4e590>, prefix='/', parsed_args=<Namespace(use_python_default_buffering=False,  command='topic', include_hidden_topics=False,  verb='echo', _command=<TopicCommand(NAME='topic', _subparser=<MonkeyPatchedIntrospectiveArgumentParser(description='Various topic related sub-commands', argument_default=None, prefix_chars='-', conflict_handler='error', _registries={'action': {None: <type at remote 0x560fb9271b00>, 'store': <type at remote 0x560fb9271b00>, 'store_const': <type at remote 0x560fb9271ec0>, 'store_true': <type at remote 0x560fb9272280>, 'store_false': <type at remote 0x560fb9272640>, 'append': <type at remote 0x560fb9272a00>, 'append_const': <type at remote 0x560fb9272dc0>, 'count': <type at remote 0x560fb9273180>, 'help': <type at remote 0x560fb9273540>, 'version': <type at remo...(truncated), tstate=0x560fb90c0840) at ../Include/internal/pycore_ceval.h:46
#36 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7fe530b88ef0, tstate=0x560fb90c0840) at ../Python/ceval.c:5067
#37 _PyFunction_Vectorcall (func=<function at remote 0x7fe530b88ee0>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:342
#38 0x0000560fb709082d in _PyObject_FastCallDictTstate (tstate=0x560fb90c0840, callable=<function at remote 0x7fe530b88ee0>, args=<optimized out>, nargsf=<optimized out>, kwargs=<optimized out>) at ../Objects/call.c:153
#39 0x0000560fb70a654c in _PyObject_Call_Prepend (tstate=0x560fb90c0840, callable=<function at remote 0x7fe530b88ee0>, obj=<optimized out>, args=<optimized out>, kwargs={'prefix': '/', 'action': <IntrospectAction(option_strings=[], dest='topic_name', nargs=None, const=None, default=None, type=None, choices=None, required=True, help="Name of the ROS topic to listen to (e.g. '/chatter')", metavar=None, container=<_ArgumentGroup(description=None, argument_default=None, prefix_chars='-', conflict_handler='error', _registries={'action': {None: <type at remote 0x560fb9271b00>, 'store': <type at remote 0x560fb9271b00>, 'store_const': <type at remote 0x560fb9271ec0>, 'store_true': <type at remote 0x560fb9272280>, 'store_false': <type at remote 0x560fb9272640>, 'append': <type at remote 0x560fb9272a00>, 'append_const': <type at remote 0x560fb9272dc0>, 'count': <type at remote 0x560fb9273180>, 'help': <type at remote 0x560fb9273540>, 'version': <type at remote 0x560fb9273900>, 'parsers': <type at remote 0x560fb9274570>, 'extend': <type at remote 0x560fb9274930>}, 'type': {None: <function at remote 0x7fe55b01aef0>}}, _actions=[<IntrospectAction(option_strings=['-h', '--help'], dest='he...(truncated)) at ../Objects/call.c:431
#40 0x0000560fb71bf1e0 in slot_tp_call (self=<TopicNameCompleter(include_hidden_topics_key='include_hidden_topics') at remote 0x7fe530b4e590>, args=(), kwds={'prefix': '/', 'action': <IntrospectAction(option_strings=[], dest='topic_name', nargs=None, const=None, default=None, type=None, choices=None, required=True, help="Name of the ROS topic to listen to (e.g. '/chatter')", metavar=None, container=<_ArgumentGroup(description=None, argument_default=None, prefix_chars='-', conflict_handler='error', _registries={'action': {None: <type at remote 0x560fb9271b00>, 'store': <type at remote 0x560fb9271b00>, 'store_const': <type at remote 0x560fb9271ec0>, 'store_true': <type at remote 0x560fb9272280>, 'store_false': <type at remote 0x560fb9272640>, 'append': <type at remote 0x560fb9272a00>, 'append_const': <type at remote 0x560fb9272dc0>, 'count': <type at remote 0x560fb9273180>, 'help': <type at remote 0x560fb9273540>, 'version': <type at remote 0x560fb9273900>, 'parsers': <type at remote 0x560fb9274570>, 'extend': <type at remote 0x560fb9274930>}, 'type': {None: <function at remote 0x7fe55b01aef0>}}, _actions=[<IntrospectAction(option_strings=['-h', '--help'], dest='he...(truncated)) at ../Objects/typeobject.c:7494
#41 0x0000560fb70915eb in _PyObject_MakeTpCall (tstate=0x560fb90c0840, callable=<TopicNameCompleter(include_hidden_topics_key='include_hidden_topics') at remote 0x7fe530b4e590>, args=<optimized out>, nargs=<optimized out>, keywords=('prefix', 'action', 'parser', 'parsed_args')) at ../Objects/call.c:215
#42 0x0000560fb708a908 in _PyObject_VectorcallTstate (kwnames=('prefix', 'action', 'parser', 'parsed_args'), nargsf=<optimized out>, args=<optimized out>, callable=<TopicNameCompleter(include_hidden_topics_key='include_hidden_topics') at remote 0x7fe530b4e590>, tstate=0x560fb90c0840) at ../Include/cpython/abstract.h:112
#43 _PyObject_VectorcallTstate (kwnames=('prefix', 'action', 'parser', 'parsed_args'), nargsf=<optimized out>, args=<optimized out>, callable=<TopicNameCompleter(include_hidden_topics_key='include_hidden_topics') at remote 0x7fe530b4e590>, tstate=0x560fb90c0840) at ../Include/cpython/abstract.h:99
#44 PyObject_Vectorcall (kwnames=('prefix', 'action', 'parser', 'parsed_args'), nargsf=<optimized out>, args=<optimized out>, callable=<TopicNameCompleter(include_hidden_topics_key='include_hidden_topics') at remote 0x7fe530b4e590>) at ../Include/cpython/abstract.h:123
#45 call_function (kwnames=('prefix', 'action', 'parser', 'parsed_args'), oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffcb21d71e0, tstate=<optimized out>) at ../Python/ceval.c:5893
#46 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:4231
#47 0x0000560fb709b70c in _PyEval_EvalFrame (throwflag=0, f=Frame 0x560fb9812a30, for file /home/chenlh/.local/lib/python3.10/site-packages/argcomplete/finders.py, line 384, in _complete_active_option (self=<CompletionFinder(_parser=<MonkeyPatchedIntrospectiveArgumentParser(description='ros2 is an extensible command-line tool for ROS 2.', argument_default=None, prefix_chars='-', conflict_handler='error', _registries={'action': {None: <type at remote 0x560fb9271b00>, 'store': <type at remote 0x560fb9271b00>, 'store_const': <type at remote 0x560fb9271ec0>, 'store_true': <type at remote 0x560fb9272280>, 'store_false': <type at remote 0x560fb9272640>, 'append': <type at remote 0x560fb9272a00>, 'append_const': <type at remote 0x560fb9272dc0>, 'count': <type at remote 0x560fb9273180>, 'help': <type at remote 0x560fb9273540>, 'version': <type at remote 0x560fb9273900>, 'parsers': <type at remote 0x560fb9274570>, 'extend': <type at remote 0x560fb9274930>}, 'type': {None: <function at remote 0x7fe55b47acb0>}}, _actions=[<IntrospectAction(option_strings=['-h', '--help'], dest='...(truncated), tstate=0x560fb90c0840) at ../Include/internal/pycore_ceval.h:46
#48 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7fe55aeb39b0, tstate=0x560fb90c0840) at ../Python/ceval.c:5067
#49 _PyFunction_Vectorcall (func=<function at remote 0x7fe55aeb39a0>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:342
#50 0x0000560fb7083f52 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fe55aecc1e0, callable=<function at remote 0x7fe55aeb39a0>, tstate=0x560fb90c0840) at ../Include/cpython/abstract.h:114
#51 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fe55aecc1e0, callable=<function at remote 0x7fe55aeb39a0>) at ../Include/cpython/abstract.h:123
#52 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffcb21d73c0, tstate=<optimized out>) at ../Python/ceval.c:5893
#53 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:4198
#54 0x0000560fb709b70c in _PyEval_EvalFrame (throwflag=0, f=Frame 0x7fe55aecc040, for file /home/chenlh/.local/lib/python3.10/site-packages/argcomplete/finders.py, line 438, in collect_completions (self=<CompletionFinder(_parser=<MonkeyPatchedIntrospectiveArgumentParser(description='ros2 is an extensible command-line tool for ROS 2.', argument_default=None, prefix_chars='-', conflict_handler='error', _registries={'action': {None: <type at remote 0x560fb9271b00>, 'store': <type at remote 0x560fb9271b00>, 'store_const': <type at remote 0x560fb9271ec0>, 'store_true': <type at remote 0x560fb9272280>, 'store_false': <type at remote 0x560fb9272640>, 'append': <type at remote 0x560fb9272a00>, 'append_const': <type at remote 0x560fb9272dc0>, 'count': <type at remote 0x560fb9273180>, 'help': <type at remote 0x560fb9273540>, 'version': <type at remote 0x560fb9273900>, 'parsers': <type at remote 0x560fb9274570>, 'extend': <type at remote 0x560fb9274930>}, 'type': {None: <function at remote 0x7fe55b47acb0>}}, _actions=[<IntrospectAction(option_strings=['-h', '--help'], dest='help...(truncated), tstate=0x560fb90c0840) at ../Include/internal/pycore_ceval.h:46
#55 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7fe55aeb3a40, tstate=0x560fb90c0840) at ../Python/ceval.c:5067
#56 _PyFunction_Vectorcall (func=<function at remote 0x7fe55aeb3a30>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:342
#57 0x0000560fb7083f52 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x560fb94af438, callable=<function at remote 0x7fe55aeb3a30>, tstate=0x560fb90c0840) at ../Include/cpython/abstract.h:114
#58 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x560fb94af438, callable=<function at remote 0x7fe55aeb3a30>) at ../Include/cpython/abstract.h:123
#59 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffcb21d75a0, tstate=<optimized out>) at ../Python/ceval.c:5893
#60 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:4198
#61 0x0000560fb709b70c in _PyEval_EvalFrame (throwflag=0, f=Frame 0x560fb94af280, for file /home/chenlh/.local/lib/python3.10/site-packages/argcomplete/finders.py, line 212, in _get_completions (self=<CompletionFinder(_parser=<MonkeyPatchedIntrospectiveArgumentParser(description='ros2 is an extensible command-line tool for ROS 2.', argument_default=None, prefix_chars='-', conflict_handler='error', _registries={'action': {None: <type at remote 0x560fb9271b00>, 'store': <type at remote 0x560fb9271b00>, 'store_const': <type at remote 0x560fb9271ec0>, 'store_true': <type at remote 0x560fb9272280>, 'store_false': <type at remote 0x560fb9272640>, 'append': <type at remote 0x560fb9272a00>, 'append_const': <type at remote 0x560fb9272dc0>, 'count': <type at remote 0x560fb9273180>, 'help': <type at remote 0x560fb9273540>, 'version': <type at remote 0x560fb9273900>, 'parsers': <type at remote 0x560fb9274570>, 'extend': <type at remote 0x560fb9274930>}, 'type': {None: <function at remote 0x7fe55b47acb0>}}, _actions=[<IntrospectAction(option_strings=['-h', '--help'], dest='help', ...(truncated), tstate=0x560fb90c0840) at ../Include/internal/pycore_ceval.h:46
#62 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7fe55aeb3650, tstate=0x560fb90c0840) at ../Python/ceval.c:5067
#63 _PyFunction_Vectorcall (func=<function at remote 0x7fe55aeb3640>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:342
#64 0x0000560fb7083f52 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x560fb92bf8c8, callable=<function at remote 0x7fe55aeb3640>, tstate=0x560fb90c0840) at ../Include/cpython/abstract.h:114
#65 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x560fb92bf8c8, callable=<function at remote 0x7fe55aeb3640>) at ../Include/cpython/abstract.h:123
#66 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffcb21d7780, tstate=<optimized out>) at ../Python/ceval.c:5893
#67 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:4198
#68 0x0000560fb709b70c in _PyEval_EvalFrame (throwflag=0, f=Frame 0x560fb92bf6a0, for file /home/chenlh/.local/lib/python3.10/site-packages/argcomplete/finders.py, line 176, in __call__ (argument_parser=<MonkeyPatchedIntrospectiveArgumentParser(description='ros2 is an extensible command-line tool for ROS 2.', argument_default=None, prefix_chars='-', conflict_handler='error', _registries={'action': {None: <type at remote 0x560fb9271b00>, 'store': <type at remote 0x560fb9271b00>, 'store_const': <type at remote 0x560fb9271ec0>, 'store_true': <type at remote 0x560fb9272280>, 'store_false': <type at remote 0x560fb9272640>, 'append': <type at remote 0x560fb9272a00>, 'append_const': <type at remote 0x560fb9272dc0>, 'count': <type at remote 0x560fb9273180>, 'help': <type at remote 0x560fb9273540>, 'version': <type at remote 0x560fb9273900>, 'parsers': <type at remote 0x560fb9274570>, 'extend': <type at remote 0x560fb9274930>}, 'type': {None: <function at remote 0x7fe55b47acb0>}}, _actions=[<IntrospectAction(option_strings=['-h', '--help'], dest='help', nargs=0, const=None, de...(truncated), tstate=0x560fb90c0840) at ../Include/internal/pycore_ceval.h:46
#69 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7fe55aeb35c0, tstate=0x560fb90c0840) at ../Python/ceval.c:5067
#70 _PyFunction_Vectorcall (func=<function at remote 0x7fe55aeb35b0>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:342
#71 0x0000560fb709082d in _PyObject_FastCallDictTstate (tstate=0x560fb90c0840, callable=<function at remote 0x7fe55aeb35b0>, args=<optimized out>, nargsf=<optimized out>, kwargs=<optimized out>) at ../Objects/call.c:153
#72 0x0000560fb70a654c in _PyObject_Call_Prepend (tstate=0x560fb90c0840, callable=<function at remote 0x7fe55aeb35b0>, obj=<optimized out>, args=<optimized out>, kwargs={'exclude': ['-h', '--help']}) at ../Objects/call.c:431
#73 0x0000560fb71bf1e0 in slot_tp_call (self=<CompletionFinder(_parser=<MonkeyPatchedIntrospectiveArgumentParser(description='ros2 is an extensible command-line tool for ROS 2.', argument_default=None, prefix_chars='-', conflict_handler='error', _registries={'action': {None: <type at remote 0x560fb9271b00>, 'store': <type at remote 0x560fb9271b00>, 'store_const': <type at remote 0x560fb9271ec0>, 'store_true': <type at remote 0x560fb9272280>, 'store_false': <type at remote 0x560fb9272640>, 'append': <type at remote 0x560fb9272a00>, 'append_const': <type at remote 0x560fb9272dc0>, 'count': <type at remote 0x560fb9273180>, 'help': <type at remote 0x560fb9273540>, 'version': <type at remote 0x560fb9273900>, 'parsers': <type at remote 0x560fb9274570>, 'extend': <type at remote 0x560fb9274930>}, 'type': {None: <function at remote 0x7fe55b47acb0>}}, _actions=[<IntrospectAction(option_strings=['-h', '--help'], dest='help', nargs=0, const=None, default='==SUPPRESS==', type=None, choices=None, required=False, help='show this help message and exit', metavar=None, ...(truncated), args=(<MonkeyPatchedIntrospectiveArgumentParser(description='ros2 is an extensible command-line tool for ROS 2.', argument_default=None, prefix_chars='-', conflict_handler='error', _registries={'action': {None: <type at remote 0x560fb9271b00>, 'store': <type at remote 0x560fb9271b00>, 'store_const': <type at remote 0x560fb9271ec0>, 'store_true': <type at remote 0x560fb9272280>, 'store_false': <type at remote 0x560fb9272640>, 'append': <type at remote 0x560fb9272a00>, 'append_const': <type at remote 0x560fb9272dc0>, 'count': <type at remote 0x560fb9273180>, 'help': <type at remote 0x560fb9273540>, 'version': <type at remote 0x560fb9273900>, 'parsers': <type at remote 0x560fb9274570>, 'extend': <type at remote 0x560fb9274930>}, 'type': {None: <function at remote 0x7fe55b47acb0>}}, _actions=[<IntrospectAction(option_strings=['-h', '--help'], dest='help', nargs=0, const=None, default='==SUPPRESS==', type=None, choices=None, required=False, help='show this help message and exit', metavar=None, container=<_ArgumentGroup...(truncated), kwds={'exclude': ['-h', '--help']}) at ../Objects/typeobject.c:7494
#74 0x0000560fb70915eb in _PyObject_MakeTpCall (tstate=0x560fb90c0840, callable=<CompletionFinder(_parser=<MonkeyPatchedIntrospectiveArgumentParser(description='ros2 is an extensible command-line tool for ROS 2.', argument_default=None, prefix_chars='-', conflict_handler='error', _registries={'action': {None: <type at remote 0x560fb9271b00>, 'store': <type at remote 0x560fb9271b00>, 'store_const': <type at remote 0x560fb9271ec0>, 'store_true': <type at remote 0x560fb9272280>, 'store_false': <type at remote 0x560fb9272640>, 'append': <type at remote 0x560fb9272a00>, 'append_const': <type at remote 0x560fb9272dc0>, 'count': <type at remote 0x560fb9273180>, 'help': <type at remote 0x560fb9273540>, 'version': <type at remote 0x560fb9273900>, 'parsers': <type at remote 0x560fb9274570>, 'extend': <type at remote 0x560fb9274930>}, 'type': {None: <function at remote 0x7fe55b47acb0>}}, _actions=[<IntrospectAction(option_strings=['-h', '--help'], dest='help', nargs=0, const=None, default='==SUPPRESS==', type=None, choices=None, required=False, help='show this help message and exit', metavar=None, ...(truncated), args=<optimized out>, nargs=<optimized out>, keywords=('exclude',)) at ../Objects/call.c:215
#75 0x0000560fb708a908 in _PyObject_VectorcallTstate (kwnames=('exclude',), nargsf=<optimized out>, args=<optimized out>, callable=<CompletionFinder(_parser=<MonkeyPatchedIntrospectiveArgumentParser(description='ros2 is an extensible command-line tool for ROS 2.', argument_default=None, prefix_chars='-', conflict_handler='error', _registries={'action': {None: <type at remote 0x560fb9271b00>, 'store': <type at remote 0x560fb9271b00>, 'store_const': <type at remote 0x560fb9271ec0>, 'store_true': <type at remote 0x560fb9272280>, 'store_false': <type at remote 0x560fb9272640>, 'append': <type at remote 0x560fb9272a00>, 'append_const': <type at remote 0x560fb9272dc0>, 'count': <type at remote 0x560fb9273180>, 'help': <type at remote 0x560fb9273540>, 'version': <type at remote 0x560fb9273900>, 'parsers': <type at remote 0x560fb9274570>, 'extend': <type at remote 0x560fb9274930>}, 'type': {None: <function at remote 0x7fe55b47acb0>}}, _actions=[<IntrospectAction(option_strings=['-h', '--help'], dest='help', nargs=0, const=None, default='==SUPPRESS==', type=None, choices=None, required=False, help='show this help message and exit', metavar=None, ...(truncated), tstate=0x560fb90c0840) at ../Include/cpython/abstract.h:112
#76 _PyObject_VectorcallTstate (kwnames=('exclude',), nargsf=<optimized out>, args=<optimized out>, callable=<CompletionFinder(_parser=<MonkeyPatchedIntrospectiveArgumentParser(description='ros2 is an extensible command-line tool for ROS 2.', argument_default=None, prefix_chars='-', conflict_handler='error', _registries={'action': {None: <type at remote 0x560fb9271b00>, 'store': <type at remote 0x560fb9271b00>, 'store_const': <type at remote 0x560fb9271ec0>, 'store_true': <type at remote 0x560fb9272280>, 'store_false': <type at remote 0x560fb9272640>, 'append': <type at remote 0x560fb9272a00>, 'append_const': <type at remote 0x560fb9272dc0>, 'count': <type at remote 0x560fb9273180>, 'help': <type at remote 0x560fb9273540>, 'version': <type at remote 0x560fb9273900>, 'parsers': <type at remote 0x560fb9274570>, 'extend': <type at remote 0x560fb9274930>}, 'type': {None: <function at remote 0x7fe55b47acb0>}}, _actions=[<IntrospectAction(option_strings=['-h', '--help'], dest='help', nargs=0, const=None, default='==SUPPRESS==', type=None, choices=None, required=False, help='show this help message and exit', metavar=None, ...(truncated), tstate=0x560fb90c0840) at ../Include/cpython/abstract.h:99
#77 PyObject_Vectorcall (kwnames=('exclude',), nargsf=<optimized out>, args=<optimized out>, callable=<CompletionFinder(_parser=<MonkeyPatchedIntrospectiveArgumentParser(description='ros2 is an extensible command-line tool for ROS 2.', argument_default=None, prefix_chars='-', conflict_handler='error', _registries={'action': {None: <type at remote 0x560fb9271b00>, 'store': <type at remote 0x560fb9271b00>, 'store_const': <type at remote 0x560fb9271ec0>, 'store_true': <type at remote 0x560fb9272280>, 'store_false': <type at remote 0x560fb9272640>, 'append': <type at remote 0x560fb9272a00>, 'append_const': <type at remote 0x560fb9272dc0>, 'count': <type at remote 0x560fb9273180>, 'help': <type at remote 0x560fb9273540>, 'version': <type at remote 0x560fb9273900>, 'parsers': <type at remote 0x560fb9274570>, 'extend': <type at remote 0x560fb9274930>}, 'type': {None: <function at remote 0x7fe55b47acb0>}}, _actions=[<IntrospectAction(option_strings=['-h', '--help'], dest='help', nargs=0, const=None, default='==SUPPRESS==', type=None, choices=None, required=False, help='show this help message and exit', metavar=None, ...(truncated)) at ../Include/cpython/abstract.h:123
#78 call_function (kwnames=('exclude',), oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffcb21d7b00, tstate=<optimized out>) at ../Python/ceval.c:5893
#79 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:4231
#80 0x0000560fb709b70c in _PyEval_EvalFrame (throwflag=0, f=Frame 0x560fb92aa470, for file /home/chenlh/Projects/ROS2/ros2-master/build/ros2cli/ros2cli/cli.py, line 62, in main (script_name='ros2', argv=None, description='ros2 is an extensible command-line tool for ROS 2.', extension=None, parser=<MonkeyPatchedIntrospectiveArgumentParser(description='ros2 is an extensible command-line tool for ROS 2.', argument_default=None, prefix_chars='-', conflict_handler='error', _registries={'action': {None: <type at remote 0x560fb9271b00>, 'store': <type at remote 0x560fb9271b00>, 'store_const': <type at remote 0x560fb9271ec0>, 'store_true': <type at remote 0x560fb9272280>, 'store_false': <type at remote 0x560fb9272640>, 'append': <type at remote 0x560fb9272a00>, 'append_const': <type at remote 0x560fb9272dc0>, 'count': <type at remote 0x560fb9273180>, 'help': <type at remote 0x560fb9273540>, 'version': <type at remote 0x560fb9273900>, 'parsers': <type at remote 0x560fb9274570>, 'extend': <type at remote 0x560fb9274930>}, 'type': {None: <function at remote 0x7fe55b47acb0>}}, _a...(truncated), tstate=0x560fb90c0840) at ../Include/internal/pycore_ceval.h:46
#81 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7fe55b0490a0, tstate=0x560fb90c0840) at ../Python/ceval.c:5067
#82 _PyFunction_Vectorcall (func=<function at remote 0x7fe55b049090>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:342
#83 0x0000560fb7083e0d in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fe55b5439b8, callable=<function at remote 0x7fe55b049090>, tstate=0x560fb90c0840) at ../Include/cpython/abstract.h:114
#84 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fe55b5439b8, callable=<function at remote 0x7fe55b049090>) at ../Include/cpython/abstract.h:123
#85 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffcb21d7ce0, tstate=<optimized out>) at ../Python/ceval.c:5893
#86 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:4213
#87 0x0000560fb7174e56 in _PyEval_EvalFrame (throwflag=0, f=Frame 0x7fe55b543840, for file /home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2, line 33, in <module> (), tstate=0x560fb90c0840) at ../Include/internal/pycore_ceval.h:46
#88 _PyEval_Vector (tstate=0x560fb90c0840, con=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:5067
#89 0x0000560fb7174cf6 in PyEval_EvalCode (co=<code at remote 0x7fe55b41cf50>, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2') at remote 0x7fe55b578190>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fe55b920950>, '__file__': '/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2', '__cached__': None, 're': <module at remote 0x7fe55b438fe0>, 'sys': <module at remote 0x7fe55b90e390>, '__requires__': 'ros2cli', 'distribution': <function at remote 0x7fe55b47ad40>, 'importlib_load_entry_point': <function at remote 0x7fe55b8eff40>, 'load_entry_point': <function at remote 0x7fe55b8eff40>}, locals=<optimized out>) at ../Python/ceval.c:1134
#90 0x0000560fb719f7d8 in run_eval_code_obj (tstate=0x560fb90c0840, co=0x7fe55b41cf50, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2') at remote 0x7fe55b578190>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fe55b920950>, '__file__': '/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2', '__cached__': None, 're': <module at remote 0x7fe55b438fe0>, 'sys': <module at remote 0x7fe55b90e390>, '__requires__': 'ros2cli', 'distribution': <function at remote 0x7fe55b47ad40>, 'importlib_load_entry_point': <function at remote 0x7fe55b8eff40>, 'load_entry_point': <function at remote 0x7fe55b8eff40>}, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2') at remote 0x7fe55b578190>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fe55b920950>, '__file__': '/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2', '__cached__': None, 're': <module at remote 0x7fe55b438fe0>, 'sys': <module at remote 0x7fe55b90e390>, '__requires__': 'ros2cli', 'distribution': <function at remote 0x7fe55b47ad40>, 'importlib_load_entry_point': <function at remote 0x7fe55b8eff40>, 'load_entry_point': <function at remote 0x7fe55b8eff40>}) at ../Python/pythonrun.c:1291
#91 0x0000560fb71990bb in run_mod (mod=<optimized out>, filename=<optimized out>, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2') at remote 0x7fe55b578190>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fe55b920950>, '__file__': '/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2', '__cached__': None, 're': <module at remote 0x7fe55b438fe0>, 'sys': <module at remote 0x7fe55b90e390>, '__requires__': 'ros2cli', 'distribution': <function at remote 0x7fe55b47ad40>, 'importlib_load_entry_point': <function at remote 0x7fe55b8eff40>, 'load_entry_point': <function at remote 0x7fe55b8eff40>}, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2') at remote 0x7fe55b578190>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fe55b920950>, '__file__': '/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2', '__cached__': None, 're': <module at remote 0x7fe55b438fe0>, 'sys': <module at remote 0x7fe55b90e390>, '__requires__': 'ros2cli', 'distribution': <function at remote 0x7fe55b47ad40>, 'importlib_load_entry_point': <function at remote 0x7fe55b8eff40>, 'load_entry_point': <function at remote 0x7fe55b8eff40>}, flags=<optimized out>, arena=<optimized out>) at ../Python/pythonrun.c:1312
#92 0x0000560fb719f525 in pyrun_file (fp=fp@entry=0x560fb909e530, filename=filename@entry='/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2', start=start@entry=257, globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2') at remote 0x7fe55b578190>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fe55b920950>, '__file__': '/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2', '__cached__': None, 're': <module at remote 0x7fe55b438fe0>, 'sys': <module at remote 0x7fe55b90e390>, '__requires__': 'ros2cli', 'distribution': <function at remote 0x7fe55b47ad40>, 'importlib_load_entry_point': <function at remote 0x7fe55b8eff40>, 'load_entry_point': <function at remote 0x7fe55b8eff40>}, locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2') at remote 0x7fe55b578190>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fe55b920950>, '__file__': '/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2', '__cached__': None, 're': <module at remote 0x7fe55b438fe0>, 'sys': <module at remote 0x7fe55b90e390>, '__requires__': 'ros2cli', 'distribution': <function at remote 0x7fe55b47ad40>, 'importlib_load_entry_point': <function at remote 0x7fe55b8eff40>, 'load_entry_point': <function at remote 0x7fe55b8eff40>}, closeit=closeit@entry=1, flags=0x7ffcb21d7f88) at ../Python/pythonrun.c:1208
#93 0x0000560fb719ea08 in _PyRun_SimpleFileObject (fp=0x560fb909e530, filename='/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2', closeit=1, flags=0x7ffcb21d7f88) at ../Python/pythonrun.c:456
#94 0x0000560fb719e653 in _PyRun_AnyFileObject (fp=0x560fb909e530, filename='/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2', closeit=1, flags=0x7ffcb21d7f88) at ../Python/pythonrun.c:90
#95 0x0000560fb719141e in pymain_run_file_obj (skip_source_first_line=0, filename='/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2', program_name='/usr/bin/python3') at ../Modules/main.c:353
#96 pymain_run_file (config=0x560fb909f3a0) at ../Modules/main.c:372
#97 pymain_run_python (exitcode=0x7ffcb21d7f84) at ../Modules/main.c:587
#98 Py_RunMain () at ../Modules/main.c:666
#99 0x0000560fb7167cad in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at ../Modules/main.c:720
#100 0x00007fe55b629d90 in __libc_start_call_main (main=main@entry=0x560fb7167c70 <main>, argc=argc@entry=2, argv=argv@entry=0x7ffcb21d8198) at ../sysdeps/nptl/libc_start_call_main.h:58
#101 0x00007fe55b629e40 in __libc_start_main_impl (main=0x560fb7167c70 <main>, argc=2, argv=0x7ffcb21d8198, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffcb21d8188) at ../csu/libc-start.c:392
#102 0x0000560fb7167ba5 in _start ()

  1. The daemon is spawned with inheriting the handles from the parent process.
wait pickler to load data and call callable_

(gdb) bt
#0  0x00007fc026718d7f in __GI___poll (fds=0x7fc02658bf60, nfds=1, timeout=timeout@entry=7200000) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007fc0267366c2 in __poll_chk (fds=<optimized out>, nfds=<optimized out>, timeout=timeout@entry=7200000, 
    fdslen=fdslen@entry=18446744073709551615) at ./debug/poll_chk.c:27
#2  0x000055c2494c622d in poll (__timeout=<optimized out>, __nfds=<optimized out>, __fds=<optimized out>)
    at /usr/include/x86_64-linux-gnu/bits/poll2.h:39
#3  select_poll_poll_impl (timeout_obj=<optimized out>, self=0x7fbff4c683f0) at ../Modules/selectmodule.c:638
#4  select_poll_poll (self=0x7fbff4c683f0, args=<optimized out>, nargs=<optimized out>) at ../Modules/clinic/selectmodule.c.h:223
#5  0x000055c249388b28 in method_vectorcall_FASTCALL (func=<method_descriptor at remote 0x7fc0261f5580>, args=0x7fbffba9d3e8, nargsf=<optimized out>, 
    kwnames=0x0) at ../Objects/descrobject.c:386
#6  0x000055c249361f52 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fbffba9d3e8, 
    callable=<method_descriptor at remote 0x7fc0261f5580>, tstate=0x55c24b5d5170) at ../Include/cpython/abstract.h:114
#7  PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fbffba9d3e8, callable=<method_descriptor at remote 0x7fc0261f5580>)
    at ../Include/cpython/abstract.h:123
#8  call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffcafd3da90, tstate=<optimized out>)
    at ../Python/ceval.c:5893
#9  _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:4198
#10 0x000055c24937970c in _PyEval_EvalFrame (throwflag=0, 
    f=Frame 0x7fbffba9d240, for file /usr/lib/python3.10/selectors.py, line 416, in select (self=<PollSelector(_fd_to_key={3: <SelectorKey at remote 0x7fbffbe9b650>}, _map=<_SelectorMapping(_selector=<...>) at remote 0x7fbff4c683d0>, _selector=<select.poll at remote 0x7fbff4c683f0>) at remote 0x7fbff4c2bc70>, timeout=7200000, ready=[]), tstate=0x55c24b5d5170) at ../Include/internal/pycore_ceval.h:46
#11 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7fc026211400, tstate=0x55c24b5d5170)
    at ../Python/ceval.c:5067
#12 _PyFunction_Vectorcall (func=<function at remote 0x7fc0262113f0>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>)
    at ../Objects/call.c:342
#13 0x000055c249361f52 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fbff4c6c1d8, 
    callable=<function at remote 0x7fc0262113f0>, tstate=0x55c24b5d5170) at ../Include/cpython/abstract.h:114
#14 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fbff4c6c1d8, callable=<function at remote 0x7fc0262113f0>)
    at ../Include/cpython/abstract.h:123
#15 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffcafd3dc70, tstate=<optimized out>)
    at ../Python/ceval.c:5893
#16 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:4198
#17 0x000055c24937970c in _PyEval_EvalFrame (throwflag=0, 
    f=Frame 0x7fbff4c6c040, for file /usr/lib/python3.10/socketserver.py, line 294, in handle_request (self=<LocalXMLRPCServer(logRequests=False, funcs={'system.listMethods': <method at remote 0x7fbff4c43600>, 'system.methodSignature': <method at remote 0x7fbff4c43900>, 'system.methodHelp': <method at remote 0x7fbff4c43a80>, 'get_name': <function at remote 0x7fbff4c47d90>, 'get_namespace': <function at remote 0x7fbff4c47e20>, 'get_node_names_and_namespaces': <function at remote 0x7fbff4c47eb0>, 'get_node_names_and_namespaces_with_enclaves': <function at remote 0x7fbff4c47f40>, 'get_topic_names_and_types': <function at remote 0x7fbff4c64040>, 'get_service_names_and_types': <function at remote 0x7fbff4c640d0>, 'get_action_names_and_types': <function at remo--Type <RET> for more, q to quit, c to continue without paging--c
te 0x7fbff4c64160>, 'get_publisher_names_and_types_by_node': <function at remote 0x7fbff4c641f0>, 'get_publishers_info_by_topic': <function at remote 0x7fbff4c64280>, 'get_subscriber_names_and_types_by_node': <function at remote 0x7fbff4c64310>, 'get_subscriptions_...(truncated), tstate=0x55c24b5d5170) at ../Include/internal/pycore_ceval.h:46
#18 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7fbffbc13380, tstate=0x55c24b5d5170) at ../Python/ceval.c:5067
#19 _PyFunction_Vectorcall (func=<function at remote 0x7fbffbc13370>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:342
#20 0x000055c249361f52 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x55c24bc8d3a0, callable=<function at remote 0x7fbffbc13370>, tstate=0x55c24b5d5170) at ../Include/cpython/abstract.h:114
#21 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x55c24bc8d3a0, callable=<function at remote 0x7fbffbc13370>) at ../Include/cpython/abstract.h:123
#22 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffcafd3de50, tstate=<optimized out>) at ../Python/ceval.c:5893
#23 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:4198
#24 0x000055c249374ca8 in _PyEval_EvalFrame (throwflag=0, f=Frame 0x55c24bc8d1e0, for file /home/chenlh/Projects/ROS2/ros2-master/build/ros2cli/ros2cli/daemon/__init__.py, line 127, in serve (server=<LocalXMLRPCServer(logRequests=False, funcs={'system.listMethods': <method at remote 0x7fbff4c43600>, 'system.methodSignature': <method at remote 0x7fbff4c43900>, 'system.methodHelp': <method at remote 0x7fbff4c43a80>, 'get_name': <function at remote 0x7fbff4c47d90>, 'get_namespace': <function at remote 0x7fbff4c47e20>, 'get_node_names_and_namespaces': <function at remote 0x7fbff4c47eb0>, 'get_node_names_and_namespaces_with_enclaves': <function at remote 0x7fbff4c47f40>, 'get_topic_names_and_types': <function at remote 0x7fbff4c64040>, 'get_service_names_and_types': <function at remote 0x7fbff4c640d0>, 'get_action_names_and_types': <function at remote 0x7fbff4c64160>, 'get_publisher_names_and_types_by_node': <function at remote 0x7fbff4c641f0>, 'get_publishers_info_by_topic': <function at remote 0x7fbff4c64280>, 'get_subscriber_names_and_types_by_node': <function at remote...(truncated), tstate=0x55c24b5d5170) at ../Include/internal/pycore_ceval.h:46
#25 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7fbffba87800, tstate=0x55c24b5d5170) at ../Python/ceval.c:5067
#26 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=<optimized out>, func=<function at remote 0x7fbffba877f0>) at ../Objects/call.c:342
#27 _PyObject_VectorcallTstate (tstate=0x55c24b5d5170, callable=<function at remote 0x7fbffba877f0>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Include/cpython/abstract.h:114
#28 0x000055c249361e0d in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fc02652eeb0, callable=<functools.partial at remote 0x7fbffbc7a340>, tstate=0x55c24b5d5170) at ../Include/cpython/abstract.h:114
#29 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fc02652eeb0, callable=<functools.partial at remote 0x7fbffbc7a340>) at ../Include/cpython/abstract.h:123
#30 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffcafd3e040, tstate=<optimized out>) at ../Python/ceval.c:5893
#31 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:4213
#32 0x000055c24937970c in _PyEval_EvalFrame (throwflag=0, f=Frame 0x7fc02652ed40, for file /home/chenlh/Projects/ROS2/ros2-master/build/ros2cli/ros2cli/daemon/daemonize.py, line 81, in main (callable_=<functools.partial at remote 0x7fbffbc7a340>), tstate=0x55c24b5d5170) at ../Include/internal/pycore_ceval.h:46
#33 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7fbffba87ad0, tstate=0x55c24b5d5170) at ../Python/ceval.c:5067
#34 _PyFunction_Vectorcall (func=<function at remote 0x7fbffba87ac0>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:342
#35 0x000055c249361e0d in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fc02652e188, callable=<function at remote 0x7fbffba87ac0>, tstate=0x55c24b5d5170) at ../Include/cpython/abstract.h:114
#36 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fc02652e188, callable=<function at remote 0x7fbffba87ac0>) at ../Include/cpython/abstract.h:123
#37 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffcafd3e220, tstate=<optimized out>) at ../Python/ceval.c:5893
#38 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:4213
#39 0x000055c249452e56 in _PyEval_EvalFrame (throwflag=0, f=Frame 0x7fc02652e020, for file <string>, line 1, in <module> (), tstate=0x55c24b5d5170) at ../Include/internal/pycore_ceval.h:46
#40 _PyEval_Vector (tstate=0x55c24b5d5170, con=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:5067
#41 0x000055c249452cf6 in PyEval_EvalCode (co=<code at remote 0x7fc0264333c0>, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x55c24b5e6d00>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fc026594950>, 'main': <function at remote 0x7fbffba87ac0>}, locals=<optimized out>) at ../Python/ceval.c:1134
#42 0x000055c24947d7d8 in run_eval_code_obj (tstate=0x55c24b5d5170, co=0x7fc0264333c0, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x55c24b5e6d00>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fc026594950>, 'main': <function at remote 0x7fbffba87ac0>}, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x55c24b5e6d00>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fc026594950>, 'main': <function at remote 0x7fbffba87ac0>}) at ../Python/pythonrun.c:1291
#43 0x000055c2494770bb in run_mod (mod=<optimized out>, filename=<optimized out>, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x55c24b5e6d00>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fc026594950>, 'main': <function at remote 0x7fbffba87ac0>}, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x55c24b5e6d00>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fc026594950>, 'main': <function at remote 0x7fbffba87ac0>}, flags=<optimized out>, arena=<optimized out>) at ../Python/pythonrun.c:1312
#44 0x000055c249470461 in PyRun_StringFlags (str=<optimized out>, start=257, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x55c24b5e6d00>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fc026594950>, 'main': <function at remote 0x7fbffba87ac0>}, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x55c24b5e6d00>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fc026594950>, 'main': <function at remote 0x7fbffba87ac0>}, flags=0x7ffcafd3e440) at ../Python/pythonrun.c:1183
#45 0x000055c249470311 in PyRun_SimpleStringFlags (command=0x7fc02645e390 "from ros2cli.daemon.daemonize import main; main()\n", flags=0x7ffcafd3e440) at ../Python/pythonrun.c:503
#46 0x000055c24946f4d5 in pymain_run_command (command=<optimized out>) at ../Modules/main.c:248
#47 pymain_run_python (exitcode=0x7ffcafd3e434) at ../Modules/main.c:578
#48 Py_RunMain () at ../Modules/main.c:666
#49 0x000055c249445cad in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at ../Modules/main.c:720
#50 0x00007fc026629d90 in __libc_start_call_main (main=main@entry=0x55c249445c70 <main>, argc=argc@entry=9, argv=argv@entry=0x7ffcafd3e648) at ../sysdeps/nptl/libc_start_call_main.h:58
#51 0x00007fc026629e40 in __libc_start_main_impl (main=0x55c249445c70 <main>, argc=9, argv=0x7ffcafd3e648, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffcafd3e638) at ../csu/libc-start.c:392
#52 0x000055c249445ba5 in _start ()

(NOTE: build bash(source) and add time.sleep(60) somewhere in the daemonize to know every phase you want.)

Q: Why does the strace log show the ERESTARTSYS?
A: The bash parent process expects the child process to exit with a SIGINT and bash parent process can read the pipe handle to exit if the peer pipe is closed. If a read operation continues to read again caused by a SIGINT, the strace could show the ERESTARTSYS if the pipe handle is still alive and there is no data in the handle.

bash parent process is blocked at read operation

(gdb) bt
#0  0x00007f4d76514992 in __GI___libc_read (fd=fd@entry=3, buf=buf@entry=0x7ffe56c1ba80, nbytes=nbytes@entry=512)
    at ../sysdeps/unix/sysv/linux/read.c:26
#1  0x00005602d666aa69 in read (__nbytes=512, __buf=0x7ffe56c1ba80, __fd=3) at /usr/x86_64-linux-gnu/include/bits/unistd.h:38
#2  zread (fd=fd@entry=3, buf=buf@entry=0x7ffe56c1ba80 "/parameter_events\v/rosout/\263O~\207\371\362\240\273\301V", len=len@entry=512) at zread.c:61
#3  0x00005602d661a27e in read_comsub (rflag=<synthetic pointer>, flags=0, quoted=0, fd=3) at subst.c:6203
#4  command_substitute (
    string=string@entry=0x5602d84ac810 "IFS=\"$IFS\"", ' ' <repeats 19 times>, "COMP_LINE=\"$COMP_LINE\"", ' ' <repeats 19 times>, "COMP_POINT=\"$COMP_POINT\"", ' ' <repeats 19 times>, "COMP_TYPE=\"$COMP_TYPE\"", ' ' <repeats 19 times>, "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREAKS"..., 
    quoted=quoted@entry=0, flags=flags@entry=0) at subst.c:6547
#5  0x00005602d661c8c9 in param_expand (
    string=string@entry=0x5602d81ba610 "$(IFS=\"$IFS\"", ' ' <repeats 19 times>, "COMP_LINE=\"$COMP_LINE\"", ' ' <repeats 19 times>, "COMP_POINT=\"$COMP_POINT\"", ' ' <repeats 19 times>, "COMP_TYPE=\"$COMP_TYPE\"", ' ' <repeats 19 times>, "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., 
    sindex=sindex@entry=0x7ffe56c1be54, quoted=quoted@entry=0, expanded_something=expanded_something@entry=0x7ffe56c1bf18, 
    contains_dollar_at=contains_dollar_at@entry=0x7ffe56c1be60, quoted_dollar_at_p=quoted_dollar_at_p@entry=0x7ffe56c1be58, 
    had_quoted_null_p=0x7ffe56c1be5c, pflags=0) at subst.c:9863
#6  0x00005602d661fa8c in expand_word_internal (word=<optimized out>, quoted=quoted@entry=0, isexp=isexp@entry=0, 
    contains_dollar_at=contains_dollar_at@entry=0x7ffe56c1bf1c, expanded_something=expanded_something@entry=0x7ffe56c1bf18) at subst.c:10329
#7  0x00005602d6621e1d in shell_expand_word_list (tlist=<optimized out>, eflags=30) at subst.c:11890
#8  0x00005602d6626af5 in expand_word_list_internal (eflags=30, list=0x5602d8494d50) at subst.c:12014
#9  expand_words_no_vars (list=list@entry=0x5602d8494d50) at subst.c:11366
#10 0x00005602d6631d60 in expand_compound_array_assignment (var=var@entry=0x5602d81b6010, 
    value=value@entry=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' <repeats 19 times>, "COMP_LINE=\"$COMP_LINE\"", ' ' <repeats 19 times>, "COMP_POINT=\"$COMP_POINT\"", ' ' <repeats 19 times>, "COMP_TYPE=\"$COMP_TYPE\"", ' ' <repeats 19 times>, "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., 
    flags=flags@entry=0) at arrayfunc.c:547
#11 0x00005602d66330ef in assign_array_var_from_string (flags=0, 
    value=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' <repeats 19 times>, "COMP_LINE=\"$COMP_LINE\"", ' ' <repeats 19 times>, "COMP_POINT=\"$COMP_POINT\"", ' ' <repeats 19 times>, "COMP_TYPE=\"$COMP_TYPE\"", ' ' <repeats 19 times>, "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., var=0x5602d81b6010)
    at arrayfunc.c:820
#12 assign_array_var_from_string (flags=0, 
    value=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' <repeats 19 times>, "COMP_LINE=\"$COMP_LINE\"", ' ' <repeats 19 times>, "COMP_POINT=\"$COMP_POINT\"", ' ' <repeats 19 times>, "COMP_TYPE=\"$COMP_TYPE\"", ' ' <repeats 19 times>, "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., var=0x5602d81b6010)
    at arrayfunc.c:810
#13 assign_array_from_string (name=name@entry=0x5602d84aba10 "COMPREPLY", 
    value=value@entry=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' <repeats 19 times>, "COMP_LINE=\"$COMP_LINE\"", ' ' <repeats 19 times>, "COMP_POINT=\"$COMP_POINT\"", ' ' <repeats 19 times>, "COMP_TYPE=\"$COMP_TYPE\"", ' ' <repeats 19 times>, "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., 
    flags=flags@entry=0) at arrayfunc.c:473
#14 0x00005602d661b70f in do_compound_assignment (flags=0, 
--Type <RET> for more, q to quit, c to continue without paging--c
    value=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' <repeats 19 times>, "COMP_LINE=\"$COMP_LINE\"", ' ' <repeats 19 times>, "COMP_POINT=\"$COMP_POINT\"", ' ' <repeats 19 times>, "COMP_TYPE=\"$COMP_TYPE\"", ' ' <repeats 19 times>, "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., name=0x5602d84aba10 "COMPREPLY") at subst.c:3190
#15 do_assignment_internal (word=<optimized out>, expand=expand@entry=1) at subst.c:3298
#16 0x00005602d66269ea in do_word_assignment (flags=0, word=<optimized out>) at subst.c:3345
#17 expand_word_list_internal (eflags=31, list=<optimized out>) at subst.c:11980
#18 expand_words (list=<optimized out>) at subst.c:11357
#19 0x00005602d65f72bc in execute_simple_command (fds_to_close=0x5602d81b71d0, async=0, pipe_out=-1, pipe_in=-1, simple_command=<optimized out>) at execute_cmd.c:4381
#20 execute_command_internal (command=0x5602d849d3d0, asynchronous=<optimized out>, pipe_in=-1, pipe_out=<optimized out>, fds_to_close=0x5602d81b71d0) at execute_cmd.c:846
#21 0x00005602d65f9e88 in execute_connection (command=0x5602d848cb90, asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d81b71d0) at execute_cmd.c:2718
#22 0x00005602d65f555a in execute_command_internal (command=0x5602d848cb90, asynchronous=<optimized out>, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d81b71d0) at execute_cmd.c:1020
#23 0x00005602d65f8207 in execute_command (command=0x5602d848cb90) at execute_cmd.c:395
#24 0x00005602d65f9e4b in execute_connection (command=0x5602d8499890, asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d83e9890) at execute_cmd.c:2709
#25 0x00005602d65f555a in execute_command_internal (command=0x5602d8499890, asynchronous=<optimized out>, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d83e9890) at execute_cmd.c:1020
#26 0x00005602d65f55a2 in execute_command_internal (command=0x5602d83ec450, asynchronous=<optimized out>, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d83e9890) at execute_cmd.c:1012
#27 0x00005602d65f957c in execute_function (var=var@entry=0x5602d84a9210, words=words@entry=0x5602d848fc90, flags=flags@entry=0, fds_to_close=fds_to_close@entry=0x5602d83e9890, async=async@entry=0, subshell=subshell@entry=0) at execute_cmd.c:5103
#28 0x00005602d65f96ec in execute_shell_function (var=var@entry=0x5602d84a9210, words=words@entry=0x5602d848fc90) at execute_cmd.c:5162
#29 0x00005602d66464d5 in gen_shell_function_matches (cs=0x5602d8154590, cs=0x5602d8154590, nw=<optimized out>, foundp=<synthetic pointer>, cw=<optimized out>, lwords=0x5602d849eed0, ind=<optimized out>, line=0x5602d849a110 "ros2 topic echo /", text=0x5602d81bb5f0 "/", cmd=0x5602d81bb5d0 "ros2") at pcomplete.c:1155
#30 gen_compspec_completions (cs=cs@entry=0x5602d8154590, cmd=cmd@entry=0x5602d81bb5d0 "ros2", word=word@entry=0x5602d81bb5f0 "/", start=start@entry=0, end=end@entry=17, foundp=foundp@entry=0x7ffe56c1c998) at pcomplete.c:1422
#31 0x00005602d664702c in gen_progcomp_completions (ocmd=ocmd@entry=0x5602d81bb5d0 "ros2", cmd=cmd@entry=0x5602d81bb5d0 "ros2", word=word@entry=0x5602d81bb5f0 "/", start=start@entry=0, end=end@entry=17, foundp=foundp@entry=0x7ffe56c1c998, retryp=0x7ffe56c1c99c, lastcs=0x7ffe56c1c9a0) at pcomplete.c:1594
#32 0x00005602d66471d6 in programmable_completions (cmd=cmd@entry=0x5602d81bb5d0 "ros2", word=word@entry=0x5602d81bb5f0 "/", start=start@entry=0, end=end@entry=17, foundp=foundp@entry=0x7ffe56c1ca34) at pcomplete.c:1649
#33 0x00005602d663f7af in attempt_shell_completion (text=0x5602d81bb5f0 "/", start=<optimized out>, end=17) at bashline.c:1665
#34 0x00005602d667dfbc in gen_completion_matches (text=text@entry=0x5602d81bb5f0 "/", start=start@entry=16, end=end@entry=17, our_func=our_func@entry=0x5602d667b780 <rl_filename_completion_function>, found_quote=<optimized out>, quote_char=<optimized out>) at complete.c:1222
#35 0x00005602d667e1ea in rl_complete_internal (what_to_do=9) at complete.c:2031
#36 0x00005602d66737cb in _rl_dispatch_subseq (key=<optimized out>, map=0x5602d66e22e0 <emacs_standard_keymap>, got_subseq=<optimized out>) at readline.c:887
#37 0x00005602d6673d98 in _rl_dispatch (map=<optimized out>, key=<optimized out>) at readline.c:833
#38 readline_internal_char () at readline.c:645
#39 0x00005602d667471d in readline_internal_charloop () at readline.c:694
#40 readline_internal () at readline.c:706
#41 readline (prompt=<optimized out>) at readline.c:385
#42 0x00005602d65de056 in yy_readline_get () at /Users/chet/src/bash/src/parse.y:1488
#43 0x00005602d65e0ac0 in yy_getc () at /Users/chet/src/bash/src/parse.y:1422
#44 shell_getc (remove_quoted_newline=remove_quoted_newline@entry=1) at /Users/chet/src/bash/src/parse.y:2358
#45 0x00005602d65e407a in read_token (command=<optimized out>) at /Users/chet/src/bash/src/parse.y:3290
#46 0x00005602d65e75e2 in read_token (command=0) at /Users/chet/src/bash/src/parse.y:3254
#47 yylex () at /Users/chet/src/bash/src/parse.y:2797
#48 yyparse () at y.tab.c:1835
#49 0x00005602d65dd56b in parse_command () at eval.c:347
#50 0x00005602d65dd707 in read_command () at eval.c:391
#51 0x00005602d65dd92a in reader_loop () at eval.c:138
#52 0x00005602d65dc583 in main (argc=1, argv=0x7ffe56c1de38, env=0x7ffe56c1de48) at shell.c:811

Q: Why is the pipe handle alive?
A: Because the pipe handle from bash parent process is inherited into the daemon. (bash -> bash -> python3(ros2) -> python3(daemon))

pipe handle is not closed (show the final state)

(ros2 NOT SETUP)  
[09:58:14 OptiPlex-7080] /home/chenlh/Projects/bash/debian/bash-5.1 (main modified) 
chenlh bash-5.1 $ ps aux | grep bash
chenlh   1256201  0.0  0.0  10792  7652 pts/5    S+   Sep12   0:00 ./bash
chenlh   2835661  0.0  0.0   6556  2348 pts/18   S+   09:58   0:00 grep bash
(ros2 NOT SETUP)  
[09:58:20 OptiPlex-7080] /home/chenlh/Projects/bash/debian/bash-5.1 (main modified) 
chenlh bash-5.1 $ ps aux | grep ros2
chenlh     50931  0.2  0.5 38691908 191560 ?     SLsl Sep12   3:32 /usr/share/code/code --enable-crashpad ./ros2-master.code-workspace
chenlh   2794616  0.3  0.2 1292928 78980 pts/5   Sl+  09:51   0:01 /usr/bin/python3 -c from ros2cli.daemon.daemonize import main; main() --name ros2-daemon --ros-domain-id 28 --rmw-implementation rmw_fastrtps_cpp
chenlh   2836266  0.0  0.0   6556  2444 pts/18   S+   09:58   0:00 grep ros2
(ros2 NOT SETUP)  
[09:58:31 OptiPlex-7080] /home/chenlh/Projects/bash/debian/bash-5.1 (main modified) 
chenlh bash-5.1 $ ls -l /proc/1256201/fd
total 0
lrwx------ 1 chenlh chenlh 64 Sep 12 17:17 0 -> /dev/pts/5
lrwx------ 1 chenlh chenlh 64 Sep 12 17:17 1 -> /dev/pts/5
lrwx------ 1 chenlh chenlh 64 Sep 12 17:17 2 -> /dev/pts/5
lrwx------ 1 chenlh chenlh 64 Sep 12 17:17 255 -> /dev/pts/5
lr-x------ 1 chenlh chenlh 64 Sep 12 17:17 3 -> 'pipe:[13549234]'
(ros2 NOT SETUP)  
[09:58:42 OptiPlex-7080] /home/chenlh/Projects/bash/debian/bash-5.1 (main modified) 
chenlh bash-5.1 $ ls -l /proc/2794616/fd
total 0
l-wx------ 1 chenlh chenlh 64 Sep 13 09:58 0 -> /home/chenlh/.ros/log/python3_2794616_1694569867039.log
lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 1 -> /dev/null
lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 10 -> 'socket:[13556055]'
lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 11 -> 'socket:[13556056]'
lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 12 -> 'socket:[13556057]'
lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 13 -> 'socket:[13556058]'
lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 14 -> 'anon_inode:[eventfd]'
lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 15 -> 'anon_inode:[eventpoll]'
lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 16 -> 'anon_inode:[timerfd]'
lr-x------ 1 chenlh chenlh 64 Sep 13 09:58 17 -> /dev/shm/fastrtps_c777f11058a465d2_el
lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 18 -> 'socket:[13556074]'
lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 19 -> 'socket:[13556080]'
lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 2 -> /dev/null
lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 20 -> 'socket:[13556083]'
lr-x------ 1 chenlh chenlh 64 Sep 13 09:58 21 -> /dev/shm/fastrtps_port14413_el
lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 22 -> 'socket:[13556089]'
lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 23 -> 'socket:[13556090]'
lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 24 -> 'socket:[13556091]'
lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 3 -> 'socket:[13549244]'
lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 4 -> 'socket:[13556051]'
lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 5 -> 'socket:[13556052]'
lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 6 -> 'socket:[13556053]'
lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 7 -> 'socket:[13556054]'
l-wx------ 1 chenlh chenlh 64 Sep 13 09:58 8 -> 'pipe:[13549234]'              <this one is still possessed in daemon>
lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 9 -> /dev/pts/5

Notice that the pipe handle id (8) is from register-python-argcomplete3 / register-python-argcomplete,
which is not a random number.

pipe handle id (8)

    COMPREPLY=( $(IFS="$IFS" \
                  COMP_LINE="$COMP_LINE" \
                  COMP_POINT="$COMP_POINT" \
                  COMP_TYPE="$COMP_TYPE" \
                  _ARGCOMPLETE_COMP_WORDBREAKS="$COMP_WORDBREAKS" \
                  _ARGCOMPLETE=1 \
                  _ARGCOMPLETE_SUPPRESS_SPACE=$SUPPRESS_SPACE \
                  "$1" 8>&1 9>&2 1>/dev/null 2>/dev/null) )
                  
or

__python_argcomplete_run_inner() {
    if [[ -z "${_ARC_DEBUG-}" ]]; then
        "$@" 8>&1 9>&2 1>/dev/null 2>&1
    else
        "$@" 8>&1 9>&2 1>&9 2>&1
    fi
}

The easiest way to fix this issue is to close the pipe handle (8) in daemon at the beginning.

diff --git a/ros2cli/ros2cli/daemon/daemonize.py b/ros2cli/ros2cli/daemon/daemonize.py
index d8994c4..0b0e632 100644
--- a/ros2cli/ros2cli/daemon/daemonize.py
+++ b/ros2cli/ros2cli/daemon/daemonize.py
@@ -76,6 +76,11 @@ def main():
     callable_ = pickle.load(sys.stdin.buffer)
     sys.stdin.close()
     os.close(0)  # force C stream close
+    try:
+        # force pipe handle(8) to close if available
+        os.close(8)
+    except OSError:
+        pass
     return callable_()

I'd like to hear your opinion.

@clalancette
Copy link
Contributor

I'd like to hear your opinion.

@iuhilnehc-ynos Wow, thanks for the amazing analysis and finding the problem.

I guess my issue with just closing handle 8 is that it seems like this number could move around on different operating systems. For instance, when we switch from Ubuntu 22.04 to 24.04, or when we are on RHEL. It just seems fragile to rely on that.

Now that we understand the problem, it seems to me it is better to skip spawning the daemon for the completer. It will probably improve completion performance (a bit), and will avoid this problem. What do you think?

@@ -31,7 +31,7 @@ def __init__(self, args):
self._direct_node = DirectNode(args)
self._daemon_node = None
else:
if use_daemon:
if use_daemon and getattr(args, 'spawn_daemon', True):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This attribute seems a bit redundant; we already have the use_daemon attribute. Can we just refactor this logic a bit instead? Something like:

use_daemon = not getattr(args, 'no_daemon', False)
if use_daemon:
  if not is_daemon_running(args):
    spawn_daemon()
  # Do DaemonNode stuff
else:
  # Do DirectNode stuff

Copy link
Author

@iuhilnehc-ynos iuhilnehc-ynos Sep 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's good to see that we can get the consensus to skip spawning the daemon for the completer (As there is still a bug in it even in the foxy: After running the ros2 topic echo /<tab>, there is a _ros2_daemon respawned, but if ctrl+c in the current shell any time, the _ros2_daemon exit. This is not the daemon feature I expected).

Something like:

I am confused by the refactor. There is a case:
If ros2 daemon start was run, do we expect the ros2 topic echo /<tab> can use DaemonNode to get data from daemon?

That's the reason why I added an extra spawn_daemon attribute.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After running the ros2 topic echo /, there is a _ros2_daemon respawned, but if ctrl+c in the current shell any time, the _ros2_daemon exit. This is not the daemon feature I expected

@iuhilnehc-ynos if i understand correctly, you are saying this is the behavior after your patch is applied? and i did not expect that behavior either.

Copy link
Author

@iuhilnehc-ynos iuhilnehc-ynos Sep 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Even if the patch mentioned at #852 (comment) is applied for rolling, this behavior happens as well.

NOTE: I found this behavior in Foxy first.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@clalancette Updated in aba8fe5

@hidmic
Copy link
Contributor

hidmic commented Sep 13, 2023

That was a deep, deep rabbit hole @iuhilnehc-ynos. Impressive archaeology work.

I guess my issue with just closing handle 8 is that it seems like this number could move around on different operating systems.

Yeah, it's brittle.

Now that we understand the problem, it seems to me it is better to skip spawning the daemon for the completer. It will probably improve completion performance (a bit), and will avoid this problem.

Well... Spawning the daemon may be slow, and I've seen the daemon end up in weird, inconsistent states that do not match the actual ROS graph, but without it you are exposed to DDS discovery latency.

Looking at subprocess.Popen documentation again, it appears there is a way to close all file descriptors but a few. See pass_fds on Linux and handle_list on Windows. You'll have to explicitly pass the socket around to daemonize, but IMO that'd be the future-proof solution. I missed this completely when I was working on #652, my bad. 🤦

@fujitatomoya
Copy link
Collaborator

build bash(source) and add time.sleep(60) somewhere in the daemonize to know every phase you want.

wow..., that is deep... amazing analysis @iuhilnehc-ynos 🚀 thanks!

it seems to me it is better to skip spawning the daemon for the completer. It will probably improve completion performance (a bit), and will avoid this problem. What do you think?
inconsistent states that do not match the actual ROS graph
without it you are exposed to DDS discovery latency.

all these are true. i second the idea to skip spawning the daemon for the auto-completer. besides, after the command is issued, daemon will start running.

@iuhilnehc-ynos iuhilnehc-ynos marked this pull request as ready for review September 19, 2023 04:41
Copy link
Collaborator

@fujitatomoya fujitatomoya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@iuhilnehc-ynos
Copy link
Author

I tested the tab completer for most ros2cli commands but not all.

@fujitatomoya
Copy link
Collaborator

@iuhilnehc-ynos can you start the CI? i think this is really good to have.

@iuhilnehc-ynos
Copy link
Author

CI:

  • Linux Build Status
  • Linux-aarch64 Build Status
  • Windows Build Status

@clalancette
Copy link
Contributor

After looking at this again, I think my previous comments were wrong. I'm very sorry about this, but I think this needs to change direction. Let me explain why.

When we are trying to do completion for topic names, node names, service names, or action names, we really do want to get the information from the daemon. That's because the daemon will have the most up-to-date information from the graph, and we really want to rely on that. If we never spawn the daemon here, then we can easily get into a situation where tab-completion will never discover some of these graph items. That means that there will be a difference between what tab-completion shows, and what ros2 topic list (for instance) shows, since the latter will actually spawn the daemon.

Let me enumerate a few different ways to fix the problem in the attached issue:

  1. What is currently in this PR, which essentially never uses the daemon for completion. This solves the blocking problem, but since it never uses the daemon it can easily not show the proper information for completion.
  2. The previous solution that @iuhilnehc-ynos had, which would use the daemon if it was around, but wouldn't spawn it if it wasn't. I think this is better than the first solution, since we'll at least use the daemon if it is there. But I don't think this is the best solution, since we never spawn the daemon at all.
  3. The original solution that @iuhilnehc-ynos proposed in make handles not inheritable to prevent from blocking durning tab-completion #852 (comment) , where we close the specific handle causing the issue. Conceptually I like that solution, but I don't like the hard-coded file descriptor number; that will be easy to break in the future.
  4. Something similar to 3, where before we spawn the daemon, we close all unnecessary file handles. What I'm thinking about here is that we change
    kwargs['close_fds'] = False
    to True (https://docs.python.org/3/library/subprocess.html#subprocess.Popen). Since the inheritable flag for the server socket is already set to True, in theory this should work. But there are probably some additional complications I don't know about.

Thoughts about all of this?

@iuhilnehc-ynos
Copy link
Author

That's because the daemon will have the most up-to-date information from the graph, and we really want to rely on that. If we never spawn the daemon here, then we can easily get into a situation where tab-completion will never discover some of these graph items. That means that there will be a difference between what tab-completion shows, and what ros2 topic list (for instance) shows, since the latter will actually spawn the daemon.

I totally agree on this.

4. Something similar to 3, where before we spawn the daemon, we close all unnecessary file handles.

In the 3, the patch is to close the pipe handle(8) in the spawned daemon process.
I don't think we can close the pipe handle(8) before we spawn the daemon, at least it doesn't work at the first time.

4. What I'm thinking about here is that we change

kwargs['close_fds'] = False

to True (https://docs.python.org/3/library/subprocess.html#subprocess.Popen). Since the inheritable flag for the server socket is already set to True, in theory this should work. But there are probably some additional complications I don't know about.

I am not certain if you just suggested me to set kwargs['close_fds'] with True. It seems to work on the surface, but the daemon is NOT spawned successfully. I think that the inheritable flag should work with close_fds(false).

https://docs.python.org/3/library/subprocess.html#subprocess.Popen

If close_fds is true, all file descriptors except 0, 1 and 2 will be closed before the child process is executed. Otherwise when close_fds is false, file descriptors obey their inheritable flag as described in [Inheritance of File Descriptors](https://docs.python.org/3/library/os.html#fd-inheritance).

https://docs.python.org/3/library/os.html#fd-inheritance

Using the [subprocess](https://docs.python.org/3/library/subprocess.html#module-subprocess) module, all file descriptors except standard streams are closed, and inheritable handles are only inherited if the close_fds parameter is False.

@clalancette
Copy link
Contributor

I am not certain if you just suggested me to set kwargs['close_fds'] with True. It seems to work on the surface, but the daemon is NOT spawned successfully. I think that the inheritable flag should work with close_fds(false).

Yeah, you are totally right about this. I misread the documentation.

In the 3, the patch is to close the pipe handle(8) in the spawned daemon process. I don't think we can close the pipe handle(8) before we spawn the daemon, at least it doesn't work at the first time.

Hm, I see. OK.

So that leaves us with either solution 2 (use the daemon if it is there, but don't spawn), or 3 (close the handle after spawning).

I think we should think about option 3 a little further, and maybe go back to how this whole thing is designed. Starting from

def spawn_daemon(args, timeout=None, debug=False):
, we do something like the following:

  1. During the execution of something like ros2 topic echo, attempt to create the XMLRPC server in this process. If that fails with EADDRINUSE, the daemon already exists and we have no more work to do. If that succeeds, then we'll proceed.
  2. Call daemonize() to create the new daemon process. In turn, this will:
    a. Setup a bunch of kwargs (including setting close_fds to False)
    b. Call subprocess.Popen to create a new process
    c. Pickle the current state of this process.
    d. Send that pickle over to the new process
  3. Close the server socket in this process. Since the new daemon has inherited the FD, it now has control of the XMLRPC server.

So I'm thinking about a few different ways to tackle this. These are all ideas/questions, nothing concrete:

  • Do we really have to create the XMLRPC server in the ros2 topic echo process? The comments say that it is to prevent a TOCTTOU race, but maybe there is another way we could prevent that race, and still create the XMLRPC server in the daemonized process instead?
  • If we do want to continue to create the XMLRPC server in the ros2 topic echo process, maybe we can reduce the FDs we are passing around. That is, maybe we can set close_fds to True, and somehow only send over the FDs we care about? https://stackoverflow.com/questions/2989823/how-to-pass-file-descriptors-from-parent-to-child-in-python suggests it is possible, but I haven't played with it.
  • The process of daemonizing doesn't seem to be exactly what I remember, at least in Unix. https://stackoverflow.com/questions/17954432/creating-a-daemon-in-linux/17955149#17955149 has a good overview; maybe we need to do more of that here?
  • Finally, one last thought; we could go with the solution of closing the fd(8), if we were certain we would know if it changed via a test or something. Maybe we can somehow get some meta-information about that file descriptor?

Anyway, this is all just food for thought. Let me know if you have additional ideas.

@iuhilnehc-ynos
Copy link
Author

  • maybe there is another way we could prevent that race, and still create the XMLRPC server in the daemonized process instead?
  • The process of daemonizing doesn't seem to be exactly what I remember, at least in Unix.

I think what we can do are the following things.

  • use lockfile(e.g. $HOME/.ros/daemon-$ROS_DOMAIN_ID.lock) to avoid the TOC/TOU.
  • make the XMLRPC server created in the daemonized process.
  • use a better way to create a daemon in Unix-like.
  • Finally, one last thought; we could go with the solution of closing the fd(8)

To close the fd(8) is a bad solution, please forget it.

@clalancette
Copy link
Contributor

I had one more thought about this over the weekend.

We are currently setting close_fds to False when we do the subprocess.Popen. That means that all fds that are not marked inheritable should be closed. So before the subprocess.Popen, we could go through and mark every fd except for 0, 1, 2, and the XMLRPC socket as non-inheritable, and they should get closed when the subprocess.Popen happens. There is a slight complication with Windows (see https://docs.python.org/3/library/os.html#fd-inheritance), but we can handle that.

Otherwise, if you don't think that is a good idea or it doesn't work for some reason, we can go with your proposal of a lockfile. If we go that way, the lockfile name should include both the DOMAIN_ID, as well as the RMW_IMPLEMENTATION in use. And we'll also have to see how lockfiles work on Windows.

@iuhilnehc-ynos
Copy link
Author

iuhilnehc-ynos commented Dec 12, 2023

That means that all fds that are not marked inheritable should be closed. So before the subprocess.Popen, we could go through and mark every fd except for 0, 1, 2, and the XMLRPC socket as non-inheritable, and they should get closed when the subprocess.Popen happens.

👍

There is a slight complication with Windows (see https://docs.python.org/3/library/os.html#fd-inheritance), but we can handle that.

As I have no windows development environment at company, I'll address it later.

Otherwise, if you don't think that is a good idea or it doesn't work for some reason, we can go with your proposal of a lockfile. If we go that way, the lockfile name should include both the DOMAIN_ID, as well as the RMW_IMPLEMENTATION in use. And we'll also have to see how lockfiles work on Windows.

I'll leave it in the future.

@iuhilnehc-ynos iuhilnehc-ynos force-pushed the topic-not-use-daemon-in-completer branch from d8f9dae to fc253dc Compare December 12, 2023 02:46
@iuhilnehc-ynos iuhilnehc-ynos changed the title avoid using daemon while completer called in bash complete make handles not inheritable to prevent from blocking durning tab-completion Dec 12, 2023
@iuhilnehc-ynos
Copy link
Author

It seems a bit rude, but it does work currently.

Chen Lihui added 2 commits December 12, 2023 12:44
Signed-off-by: Chen Lihui <lihui.chen@sony.com>
Signed-off-by: Chen Lihui <lihui.chen@sony.com>
Signed-off-by: Chen Lihui <lihui.chen@sony.com>
@iuhilnehc-ynos
Copy link
Author

I am not sure if the tab-completion of ros2 in Windows is supported or not.
I did the test last night, but it doesn't work, so I decided to remove the set not inheritable actions for windows.

@fujitatomoya
Copy link
Collaborator

@clalancette friendly ping, lgtm for me. i guess windows should work?

Copy link
Contributor

@clalancette clalancette left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One small change to the comment. Once that is done, I'll be happy with this with green CI. Thanks!

ros2cli/ros2cli/node/daemon.py Outdated Show resolved Hide resolved
Co-authored-by: Chris Lalancette <clalancette@gmail.com>
Signed-off-by: Chen Lihui <lihui.chen@sony.com>
@iuhilnehc-ynos
Copy link
Author

CI:

  • Linux Build Status
  • Linux-aarch64 Build Status
  • Windows Build Status

Copy link
Contributor

@clalancette clalancette left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fantastic. Thank you for all of your work here. With green CI, I'm going to merge this.

@clalancette clalancette merged commit d32d820 into ros2:rolling Dec 21, 2023
3 checks passed
@fujitatomoya
Copy link
Collaborator

@Mergifyio backport iron humble

Copy link

mergify bot commented Dec 26, 2023

backport iron humble

✅ Backports have been created

mergify bot pushed a commit that referenced this pull request Dec 26, 2023
…pletion (#852)

* make file handles not inheritable before spawn the daemon

Signed-off-by: Chen Lihui <lihui.chen@sony.com>
Co-authored-by: Chris Lalancette <clalancette@gmail.com>
(cherry picked from commit d32d820)
mergify bot pushed a commit that referenced this pull request Dec 26, 2023
…pletion (#852)

* make file handles not inheritable before spawn the daemon

Signed-off-by: Chen Lihui <lihui.chen@sony.com>
Co-authored-by: Chris Lalancette <clalancette@gmail.com>
(cherry picked from commit d32d820)
fujitatomoya pushed a commit that referenced this pull request Dec 27, 2023
…pletion (#852) (#873)

* make file handles not inheritable before spawn the daemon

Signed-off-by: Chen Lihui <lihui.chen@sony.com>
Co-authored-by: Chris Lalancette <clalancette@gmail.com>
(cherry picked from commit d32d820)

Co-authored-by: Chen Lihui <lihui.chen@sony.com>
fujitatomoya pushed a commit that referenced this pull request Dec 27, 2023
…pletion (#852) (#872)

* make file handles not inheritable before spawn the daemon

Signed-off-by: Chen Lihui <lihui.chen@sony.com>
Co-authored-by: Chris Lalancette <clalancette@gmail.com>
(cherry picked from commit d32d820)

Co-authored-by: Chen Lihui <lihui.chen@sony.com>
@akssri-sony
Copy link
Contributor

akssri-sony commented Mar 6, 2024

Looks like this is causing ros2 topic daemon to hang in arch-linux, but works fine on Ubuntu-20.04.
Unclear why.

Edit: Bug report,
ros2/ros2#1531

@akssri-sony
Copy link
Contributor

akssri-sony commented Mar 6, 2024

The limit value (soft in L143) is way higher in Arch (1073741816) vs Ubuntu (1048576 - ~1000x).

Edit:
Can confirm it's getting stuck in this loop.

Is there a better way to do this and/or limit the bounds of the loop ?

(Arch is at 6.7.8-arch-1; Ubuntu ~ 5.15).

@akssri-sony
Copy link
Contributor

akssri-sony commented Mar 6, 2024

Looks like this is something that is set by (newer ? some ?) versions of systemd to a higher value (~2^30) by default.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Autocomplete hangs before calling NodeStrategy
5 participants