Call c++ global variables in nxtask_startup #1341

xiaoxiang781216 · 2020-06-30T18:56:35Z

Summary

To avoid the similar code spread around the apps code space

Impact

Testing

patacongo · 2020-07-01T01:07:25Z

It is an incorrect design to call C++ constructors or destructors from the OS in kernel mode (PROTECTED and KERNEL builds) or with interrupts disabled. Calling the constructors.destructors introduces new problems similar to those discussed in Issue #1263 and which I am working toward fixing in PR #1328

Executing the constructors/destructors in kernel mode is a security violation. Running the constructors/destructors with interrupts disabled is just wrong. What if they need to wait for an event in a busy loop? No user code should ever run with interrupts disabled:

This change should not be done. You should consider contributing to the correct fix that does not introduce additional problems of this nature. I would recommend that this change not be merged is it is not correct. It is expedient... but it is wrong. That is forbidden in the INVIOLABLES.txt:

The Enemies
===========

No Short Cuts
-------------

  o Doing things the easy way instead of the correct way.
  o Reducing effort at the expense of Quality, Portability, or
    Consistency.
  o Focus on the values of the organization, not the values of the Open
    Source project.  Need to support both.
  o It takes work to support the Inviolables.  There are no shortcuts.

Let's do things right.

There is also a bug already listed in the to-level TODO list (This is also Issue #1265 ):

  Title:       C++ CONSTRUCTORS HAVE TOO MANY PRIVILEGES (PROTECTED MODE)
  Description: When a C++ ELF module is loaded, its C++ constructors are called
               via sched/task_starthook.c logic.  This logic runs in protected mode.
               The is a security hole because the user code runs with kernel-
              privileges when the constructor executes.

               Destructors likely have the opposite problem.  The probably try to
               execute some kernel logic in user mode?  Obviously this needs to
               be investigated further.
  Status:      Open
  Priority:    Low (unless you need build a secure C++ system).

patacongo · 2020-07-01T01:09:25Z

Think of how this is done in Linux: Each task has a wrapper function called __start in crt0.S and will handle both constructors and destructors. And the first layer or exit logic is part of the C library.

patacongo · 2020-07-01T01:28:53Z

Look how I did the pthread startup function in PR #1328. This is EXACTLY the same issue. I created a user-space pthread_startup (NOT and OS startup function) in libs/libc/pthread/pthread_create.c. This is exactly the kind of user-space startup function that you need: It runs in user mode with interrupts enabled.

xiaoxiang781216 · 2020-07-01T03:35:08Z

It is an incorrect design to call C++ constructors or destructors from the OS in kernel mode (PROTECTED and KERNEL builds) or with interrupts disabled. Calling the constructors.destructors introduces new problems similar to those discussed in Issue #1263 and which I am working toward fixing in PR #1328

nxtask_startup is the part of libc in PROTECTED and run in the user mode:

nxtask_start->up_task_start->switch to user mode->nxtask_startup->c++ constructors

KERNEL mode doesn't go this path.
Note: the constructors here is the global c++ objects which is part of nuttx.bin and nuttx_user.bin(NOT in elf binary).
You can see there are many places call gnu_cxxinitialize:
apache/nuttx-apps#316
This patch try to centralize the builtin constructors in one place, but don't break the rule: call constructor should happen inside the user mode without the interrupt disable.

There is also a bug already listed in the to-level TODO list (This is also Issue #1265 ):

  Title:       C++ CONSTRUCTORS HAVE TOO MANY PRIVILEGES (PROTECTED MODE)
  Description: When a C++ ELF module is loaded, its C++ constructors are called
               via sched/task_starthook.c logic.  This logic runs in protected mode.
               The is a security hole because the user code runs with kernel-
              privileges when the constructor executes.

               Destructors likely have the opposite problem.  The probably try to
               execute some kernel logic in user mode?  Obviously this needs to
               be investigated further.
  Status:      Open
  Priority:    Low (unless you need build a secure C++ system).

This patch move the construction inside nuttx.bin(FLAT build)/nuttx_user.bin(PROTECTED) to nxtask_startup. it's different from ELF binary problem.

Think of how this is done in Linux: Each task has a wrapper function called __start in crt0.S and will handle both constructors and destructors. And the first layer or exit logic is part of the C library.

To follow the Linux approach, the major change is the build system:
1.Statically link crt0.o into each ELF binary
2.__start call constructors, and main, destructors and exit in order
3.Remove all stuff related to elf_loadctors/elf_loaddtors
Yes, this method is better than the current solution:
1.Don't need parse ELF to extract .ctor/.dtor section
2.Don't need startup hook anymore
3.Work with other binfmt
If you agree with this direction, I will provide a patch later, but builtin apps with global c++ object still need this patch.

Look how I did the pthread startup function in PR #1328. This is EXACTLY the same issue. I created a user-space pthread_startup (NOT and OS startup function) in libs/libc/pthread/pthread_create.c. This is exactly the kind of user-space startup function that you need: It runs in user mode with interrupts enabled.

The same approach is workable for ELF binary. But how do we handle the builtin apps, do you have the idea to avoid nxtask_start in this case?

xiaoxiang781216 · 2020-07-01T03:54:45Z

@patacongo to indicate nxtask_startup isn't used in KERNEL build, I add the guard like this:

#ifndef CONFIG_BUILD_KERNEL
void nxtask_startup(main_t entrypt, int argc, FAR char *argv[]);
#endif

to follow the naming convention Signed-off-by: Xiang Xiao <xiaoxiang@xiaomi.com> Change-Id: I3594d12a65e8cacea99bc295d622628304c3f9f8

to avoid the similar code spread around each application Signed-off-by: Xiang Xiao <xiaoxiang@xiaomi.com> Change-Id: I8967d647eaf2ecae47f29f83e7fa322ef1b42a02

patacongo · 2020-07-01T13:52:40Z

To follow the Linux approach, the major change is the build system:
1.Statically link crt0.o into each ELF binary
2.__start call constructors, and main, destructors and exit in order
3.Remove all stuff related to elf_loadctors/elf_loaddtors

The KERNEL build already works this way. crt0.c currently exists only at ./arm/src/armv7-a/crt0.c

patacongo · 2020-07-01T13:55:28Z

I will go ahead and merge this change. I do not think it is a worthwhile change since it does not improve the OS architecture except in some special cases. So although I do not think it has value, I will go ahead and merge because neither can I find any of the most serious that I feared.

Think of how much more we could accomplished if we worked together in a common roadmap to achieve the same end goals. We could do much more together that if you constantly go in a different direction.

patacongo · 2020-07-01T14:11:33Z

Look how I did the pthread startup function in PR #1328. This is EXACTLY the same issue. I created a user-space pthread_startup (NOT and OS startup function) in libs/libc/pthread/pthread_create.c. This is exactly the kind of user-space startup function that you need: It runs in user mode with interrupts enabled.

The same approach is workable for ELF binary. But how do we handle the builtin apps, do you have the idea to avoid nxtask_start in this case?

In the FLAT build, there is only one set constructors. It does not matter how many built-in tasks there are, the constructors only have to be called once. Each separately linked blob will have one set of C++ constructors and must have its C++ constructors called exactly once.

Currently the C++ constructors may be initialized in many places under apps/:

examples/elf/elf_main.c: up_cxxinitialize();
examples/helloxx/helloxx_main.cxx: up_cxxinitialize();
examples/nxterm/nxterm_main.c: up_cxxinitialize();
graphics/nxwm/src/nxwm_main.cxx: up_cxxinitialize();
graphics/twm4nx/src/twm4nx_main.cxx: up_cxxinitialize();

Since the constructors will always be called in nxtask_startup(), I think these are now all bugs. We should also get rid of all occurrences of up_cxxinitialize().

@xiaoxiang781216 apps/platform/gnu/ should be removed too.

patacongo · 2020-07-01T15:27:32Z

@xiaoxiang781216 I think this change then resolves Issue #1265. Is that correct? If so then that issue should be closed.

patacongo · 2020-07-01T15:41:19Z

libs/libxx/Kconfig

+	---help---
+		The platform-specific logic includes support for initialization
+		of static C++ instances for this architecture and for the selected
+		toolchain (via up_cxxinitialize()).


@xiaoxiang781216 up_cxxinitialize() should be removed. cxx_initialize() is the only system level C++ initializer in the FLAT build.

Here: apache/nuttx-apps#316

xiaoxiang781216

Look how I did the pthread startup function in PR #1328. This is EXACTLY the same issue. I created a user-space pthread_startup (NOT and OS startup function) in libs/libc/pthread/pthread_create.c. This is exactly the kind of user-space startup function that you need: It runs in user mode with interrupts enabled.

The same approach is workable for ELF binary. But how do we handle the builtin apps, do you have the idea to avoid nxtask_start in this case?

In the FLAT build, there is only one set constructors. It does not matter how many built-in tasks there are, the constructors only have to be called once. Each separately linked blob will have one set of C++ constructors and must have its C++ constructors called exactly once.

Currently the C++ constructors may be initialized in many places under apps/:

examples/elf/elf_main.c: up_cxxinitialize();
examples/helloxx/helloxx_main.cxx: up_cxxinitialize();
examples/nxterm/nxterm_main.c: up_cxxinitialize();
graphics/nxwm/src/nxwm_main.cxx: up_cxxinitialize();
graphics/twm4nx/src/twm4nx_main.cxx: up_cxxinitialize();

Since the constructors will always be called in nxtask_startup(), I think these are now all bugs. We should also get rid of all occurrences of up_cxxinitialize().

@xiaoxiang781216 apps/platform/gnu/ should be removed too.

Yes, the patch is here: apache/nuttx-apps#316

@xiaoxiang781216 I think this change then resolves Issue #1265. Is that correct? If so then that issue should be closed.

No, this patch just centralize the builtin(FLAT and PROTECTED) constructor in one place. I will provide a patch for ELF binary this week.

patacongo · 2020-07-01T16:55:20Z

libs/libc/sched/task_startup.c

+      inited = 1;
+    }
+#endif
+}


@xiaoxiang781216 Is this logic specific to the GNU compiler? I don't know how other C++ compilers provide C++ constructor data. It could work completely differently. I wonder if this does not belong in libs/libc/machine/*/gnu

@xiaoxiang781216 Is this logic specific to the GNU compiler? I don't know how other C++ compilers provide C++ constructor data.

Not too specific as far as I know, compiler(gcc, clang, ceva) always put the constructor points in a special section.

It could work completely differently. I wonder if this does not belong in libs/libc/machine/*/gnu

Yes, if some compiler work differently, we need put this stuff into the compiler specific folder, but libc/compiler/gnu may better than libc/machine/*/gnu since how to invoke the constructor is only compiler specific not machine specific.
BTW, on the other hand, the exception unwind is the compiler and machine specific, so could you review this patch?
#1315.
The code can't even compile on x86/sim platform when c++ exception is enabled.

xiaoxiang781216 force-pushed the cpp3 branch from f7dea20 to 628be66 Compare July 1, 2020 00:56

xiaoxiang781216 force-pushed the cpp3 branch from 628be66 to 85e52e1 Compare July 1, 2020 03:52

xiaoxiang781216 mentioned this pull request Jul 1, 2020

apps: Remove all stuff related to CONFIG_xxx_CXXINITIALIZE apache/nuttx-apps#316

Merged

sched: Rename task_startup to nxtask_startup

3e70dcc

to follow the naming convention Signed-off-by: Xiang Xiao <xiaoxiang@xiaomi.com> Change-Id: I3594d12a65e8cacea99bc295d622628304c3f9f8

xiaoxiang781216 force-pushed the cpp3 branch 2 times, most recently from 0031690 to 1093077 Compare July 1, 2020 06:11

sched: Call c++ global variables constructor inside nxtask_startup

096c016

to avoid the similar code spread around each application Signed-off-by: Xiang Xiao <xiaoxiang@xiaomi.com> Change-Id: I8967d647eaf2ecae47f29f83e7fa322ef1b42a02

xiaoxiang781216 force-pushed the cpp3 branch from 1093077 to 096c016 Compare July 1, 2020 06:16

patacongo merged commit 8153e31 into apache:master Jul 1, 2020

patacongo reviewed Jul 1, 2020

View reviewed changes

xiaoxiang781216 commented Jul 1, 2020

View reviewed changes

patacongo reviewed Jul 1, 2020

View reviewed changes

xiaoxiang781216 mentioned this pull request Oct 8, 2020

sched: nxtask_start should call entry point directly for kernel thread #1922

Merged

btashton added this to To-Add in Release Notes - 10.0.0 Oct 14, 2020

btashton moved this from To-Add to Minor in Release Notes - 10.0.0 Oct 19, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Call c++ global variables in nxtask_startup #1341

Call c++ global variables in nxtask_startup #1341

xiaoxiang781216 commented Jun 30, 2020

patacongo commented Jul 1, 2020 •

edited

patacongo commented Jul 1, 2020 •

edited

patacongo commented Jul 1, 2020

xiaoxiang781216 commented Jul 1, 2020 •

edited

xiaoxiang781216 commented Jul 1, 2020

patacongo commented Jul 1, 2020

patacongo commented Jul 1, 2020

patacongo commented Jul 1, 2020 •

edited

patacongo commented Jul 1, 2020

patacongo Jul 1, 2020 •

edited

xiaoxiang781216 Jul 1, 2020

xiaoxiang781216 left a comment •

edited

patacongo Jul 1, 2020

xiaoxiang781216 Jul 2, 2020 •

edited

Call c++ global variables in nxtask_startup #1341

Call c++ global variables in nxtask_startup #1341

Conversation

xiaoxiang781216 commented Jun 30, 2020

Summary

Impact

Testing

patacongo commented Jul 1, 2020 • edited

patacongo commented Jul 1, 2020 • edited

patacongo commented Jul 1, 2020

xiaoxiang781216 commented Jul 1, 2020 • edited

xiaoxiang781216 commented Jul 1, 2020

patacongo commented Jul 1, 2020

patacongo commented Jul 1, 2020

patacongo commented Jul 1, 2020 • edited

patacongo commented Jul 1, 2020

patacongo Jul 1, 2020 • edited

Choose a reason for hiding this comment

xiaoxiang781216 Jul 1, 2020

Choose a reason for hiding this comment

xiaoxiang781216 left a comment • edited

Choose a reason for hiding this comment

patacongo Jul 1, 2020

Choose a reason for hiding this comment

xiaoxiang781216 Jul 2, 2020 • edited

Choose a reason for hiding this comment

patacongo commented Jul 1, 2020 •

edited

patacongo commented Jul 1, 2020 •

edited

xiaoxiang781216 commented Jul 1, 2020 •

edited

patacongo commented Jul 1, 2020 •

edited

patacongo Jul 1, 2020 •

edited

xiaoxiang781216 left a comment •

edited

xiaoxiang781216 Jul 2, 2020 •

edited