Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking Issue for feature(packed_bundled_libs) #108081

Open
1 of 3 tasks
petrochenkov opened this issue Feb 15, 2023 · 6 comments
Open
1 of 3 tasks

Tracking Issue for feature(packed_bundled_libs) #108081

petrochenkov opened this issue Feb 15, 2023 · 6 comments
Labels
A-linkage Area: linking into static, shared libraries and binaries C-tracking-issue Category: A tracking issue for an RFC or an unstable feature. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@petrochenkov
Copy link
Contributor

This is a tracking issue for the fix for issue "Linking modifier whole-archive and link time cfg are not supported on bundled static libraries" (#99429).
The feature gate for the issue is #![feature(packed_bundled_libs)].

Currently to combine +bundle native libraries with +whole-archive or cfg a new representation of bundled native libraries in a rlib used, in which the library is packed into the archive as a whole (and wrapped into an object file for technical reasons), and then unpacked when you need to link to it.

Other representations are also possible, e.g. keeping the bundled lib in the old "unpacked" format as a set of object files, and adding some description file telling which object files belong to which library, it would be more complex though.

It may be quite reasonable to support both packed and unpacked representations, both have their uses

  • the packed representation minimizes differences with classic non-bundled (-bundle) static libraries
  • the unpacked representation is needed when the rlib actually provides missing symbols to the native static lib, not just depends on it, as can be seen from the breakage report in [WIP] Native library linking changes for crater testing #102832

Stabilizing the +bundle,+whole-archive combination doesn't need specifying any of these details, we just need to promise that it somehow works.

About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.

Steps

Unresolved Questions

None yet.

Implementation history

@petrochenkov petrochenkov added the C-tracking-issue Category: A tracking issue for an RFC or an unstable feature. label Feb 15, 2023
@petrochenkov
Copy link
Contributor Author

Future directions:

  • We currently have an unstable option -Zpacked-bundled-libs that switches representation of all bundled static native libs to packed, it should be changed to a native lib modifier +/-packed because it's something that can and should be controllable in a more fine-grained way.
  • +packed representation doesn't require the bundled file to be a valid archive, so it's actually a good opportunity to support bundling non-archived object files directly into rlibs, like
    #[link(name = "myobject.o", modifiers="+packed,+verbatim", /* etc */)]
    , we just need to avoid wrapping the packed file into an object file it it's already an object file.

@petrochenkov
Copy link
Contributor Author

So I think we need to eventually switch everything to using the new (packed) format, e.g. on 2024 edition, because it's a slightly breaking change.

I think the main goal is to fully preserve static library semantic, in that case

  • all the modifiers specific to static libraries will work
  • crates providing both bundling a library and searching it on the system as alternatives will work identically in both cases
  • overrides like -Zlink-native-libraries or -Zlink-directives will work without changes if some bundled library is replaced by a library from the system

For dealing with the breakage in #102832 we need:

  • Some kind of attribute (or non-attribute) #[requires_symbol(foo)] that is put on foreign blocks and says that the linked library requires some symbol to be provided by downstream.
  • Support for bundling object files into rlibs (basically already exists, needs minor tweaks Tracking Issue for feature(packed_bundled_libs) #108081 (comment)), often we don't need libraries but just need to compile some C files and put the resulting objects into our rlib (example - compiler-builtins).

@Be-ing
Copy link
Contributor

Be-ing commented Jun 29, 2023

We have an interesting use case for +bundle,+whole-archive in CXX-Qt. We have a crate cxx-qt-build which is used by downstream build.rs scripts. Under the hood, cxx-qt-build generates and compiles C++ code (using the cc crate) that contains static variables that aren't referenced from within main (for several purposes, including Qt's resource system and QML plugin registration), so that code needs to be linked with +whole-archive to be included in the final staticlib/bin. Typically cxx-qt-build is used in a single staticlib crate, which works fine (except on Rust 1.69 due to #110912 which was fixed in 1.70). However, we'd like to be able to support having multiple rlib crates using cxx-qt-build in the Rust dependency graph. A proof-of-concept of such an arragement (KDAB/cxx-qt#598) fails to build with:

error: link modifiers combination `+bundle,+whole-archive` is unstable when generating rlibs

I'm not sure how the discussion above about packed versus unpacked rlibs relates to this. Somehow, we need a way to link a library (or object files) built by the cc crate in build.rs to an rlib crate and have those symbols included in the final downstream staticlib and bin products produced by Cargo.

@petrochenkov
Copy link
Contributor Author

I'm not sure how the discussion above about packed versus unpacked rlibs relates to this.

"Packed" supports +bundle,+whole-archive, "unpacked" do not.

The discussion above is mainly about enabling "packed" by default for everything.
For the +bundle,+whole-archive and +bundle, cfg(...) cases we mostly can just stabilize it, from what I remember.

@Be-ing
Copy link
Contributor

Be-ing commented Jun 29, 2023

For the +bundle,+whole-archive and +bundle, cfg(...) cases we mostly can just stabilize it, from what I remember.

Do you mean automatically turning on packed_bundled_libs when +whole-archive and +bundle are used together?

@petrochenkov
Copy link
Contributor Author

@Be-ing
Yes.
This case is not affected by any backward compatibility issues because it produced errors previously.

Be-ing added a commit to Be-ing/rust that referenced this issue Aug 15, 2023
Currently, combining +bundle and +whole-archive works only with
 #![feature(packed_bundled_libs)]
This crate feature is independent of the -Zpacked-bundled-libs
command line option.

This commit stabilizes the #![feature(packed_bundled_libs)] crate
feature and implicitly enables it only when the +bundle and
+whole-archive link modifiers are combined. This allows rlib
crates to use the +whole-archive link modifier with native
libraries and have all symbols included in the linked library
to be included in downstream staticlib crates that use the rlib as
a dependency. Other cases requiring the packed_bundled_libs
behavior still require the -Zpacked-bundled-libs command line
option, which can be stabilized independently in the future.

Per discussion on rust-lang#108081
there is no risk of regression stabilizing the crate feature in
this way because the combination of +bundle,+whole-archive link
modifiers was previously not allowed.
bors added a commit to rust-lang-ci/rust that referenced this issue Sep 29, 2023
…e, r=petrochenkov

stabilize combining +bundle and +whole-archive link modifiers

Per discussion on rust-lang#108081 combining +bundle and +whole-archive already works and can be stabilized independently of other aspects of the packed_bundled_libs feature. There is no risk of regression because this was not previously allowed.

r? `@petrochenkov`
RalfJung pushed a commit to RalfJung/miri that referenced this issue Sep 30, 2023
…ochenkov

stabilize combining +bundle and +whole-archive link modifiers

Per discussion on rust-lang/rust#108081 combining +bundle and +whole-archive already works and can be stabilized independently of other aspects of the packed_bundled_libs feature. There is no risk of regression because this was not previously allowed.

r? `@petrochenkov`
lnicola pushed a commit to lnicola/rust-analyzer that referenced this issue Apr 7, 2024
…ochenkov

stabilize combining +bundle and +whole-archive link modifiers

Per discussion on rust-lang/rust#108081 combining +bundle and +whole-archive already works and can be stabilized independently of other aspects of the packed_bundled_libs feature. There is no risk of regression because this was not previously allowed.

r? `@petrochenkov`
RalfJung pushed a commit to RalfJung/rust-analyzer that referenced this issue Apr 27, 2024
…ochenkov

stabilize combining +bundle and +whole-archive link modifiers

Per discussion on rust-lang/rust#108081 combining +bundle and +whole-archive already works and can be stabilized independently of other aspects of the packed_bundled_libs feature. There is no risk of regression because this was not previously allowed.

r? `@petrochenkov`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-linkage Area: linking into static, shared libraries and binaries C-tracking-issue Category: A tracking issue for an RFC or an unstable feature. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

No branches or pull requests

3 participants