Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why linker strips about 90% of used files / classes? #21725

Open
Honya2000 opened this issue Apr 9, 2024 · 6 comments
Open

Why linker strips about 90% of used files / classes? #21725

Honya2000 opened this issue Apr 9, 2024 · 6 comments

Comments

@Honya2000
Copy link

Honya2000 commented Apr 9, 2024

Hello,

I'm currently working on the project which is using proto buffers, which register descriptors globally this way:

PROTOBUF_ATTRIBUTE_INIT_PRIORITY2 static ::std::false_type
    _static_init2_ PROTOBUF_UNUSED =
        (::_pbi::AddDescriptors(&descriptor_table_mediapipe_2fcalculators_2ftensor_2ftensors_5fto_5fsegmentation_5fcalculator_2eproto),
        ::_pbi::ExtensionSet::RegisterMessageExtension(
            &::mediapipe::CalculatorOptions::default_instance(), 374311106, 11,
            false, false, &::mediapipe::TensorsToSegmentationCalculatorOptions::default_instance(),
            nullptr, ::_pbi::LazyAnnotation::kUndefined),
         ::std::false_type{});

Here ::_pbi::AddDescriptors - just registeres the class in the global factory.
There are over 100 such .proto files / classes.

But during parsing .pbtxt files parser throws lot of errors regarding missing extensions.
It turns out linker just strips majority of my classes with self registration.
Currently i log all the registered descriptors and I was surprized that over 90% of my classes just stripped away.

Why linker strips the code which is potentially used in the final executable?

I checked generated ninja file and those .o files are linked to the static library libmediaPipe.a which is linked to final executable. So those classes should not be stripped.

@sbc100
Copy link
Collaborator

sbc100 commented Apr 9, 2024

If you want your static constructors to run the linker needs to have some reason to include the object file from the library.

You can force the linker to include all of the objects in the library using -Wl,--whole-archive libmediaPipe.a -Wl,--no-whole-archive.

Or you add references to the symbol in the object files that you want from some other part of the application (e.g. main.c).

This is the save behaviour as native/desktop linkers.

@Honya2000
Copy link
Author

so if i use -Wl,--whole-archive - it will not strip anything ? Even if class is never referenced ?

@sbc100
Copy link
Collaborator

sbc100 commented Apr 9, 2024

It will force the object files to be included, which will in turn force any static constructors to run.

If you just want to include what you need then you should just reference what you need and should not be stripped. The linker will never strip code that is referenced.

@sbc100
Copy link
Collaborator

sbc100 commented Apr 9, 2024

Using --whole-archive is the the same as if you specified each of the object files form the library on the command line. The linker can still do dead code elimination in this case, it just won't ever skip over entire object files like it does (by default) for object files that live in library archives.

@Honya2000
Copy link
Author

Will nested static libs also work?

For example mediaPipe static lib includes lot of other libs:

target_link_libraries(mediaPipe PRIVATE
tensorflow-lite
libprotobuf
yuv
zlibstatic
glog
absl_statusor
absl_cord
absl_strings
absl_time
absl_hash
absl_log_sink
absl_base
absl_flags_parse
absl_die_if_null
absl_log_internal_check_op
absl_log_internal_conditions
)

Will then final linking stage use all obj files in nested libs as well?
target_link_libraries(NativeAI PRIVATE
-Wl,--whole-archive mediaPipe
)

@sbc100
Copy link
Collaborator

sbc100 commented Apr 9, 2024

There is no such thing as a nested library for static ar archives so you would need to check what you build system is doing here.
Are those other libraries being included directly into libmediaPipe.a? You can check this using ar t libmediaPipe.a.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants