Implement -static-stdlib for Linux #1891
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| assert(dynamicListFile); | ||
| int result = fputs(dynamicListContents, dynamicListFile); | ||
| assert(result != EOF); | ||
| fclose(dynamicListFile); |
There was a problem hiding this comment.
@jrose-apple You know if there's an LLVM utility API we could use for writing a string to a temporary file here, instead of using bare stdio?
|
@jckarter Good catch. Updated to use |
|
This looks reasonable to me, though the driver is still primarily @jrose-apple's domain, so he might want to look over it too. |
|
Rebased against #1817 |
|
ping @jrose-apple |
|
Bump. We're following the status of the bug and referenced PRs over at: emscripten-core/emscripten#2427. There's an effort to get swift compiled to javascript, I'm sure many would appreciate an update on this PR's status. Thanks. |
This commit adds the flags -static-stdlib and -no-static-stdlib to create programs statically linked (or not) with the standard library. Not is the default, which is also the current behavior. These flags are currently placebos on non-Darwin platforms.
|
Rebased on #1817 Anticipated @jrose-apple's style nitpick |
2 tasks
| @@ -32,6 +32,7 @@ | |||
| #include "llvm/Support/Path.h" | |||
| #include "llvm/Support/Process.h" | |||
| #include "llvm/Support/Program.h" | |||
| #include <stdio.h> | |||
Implement the -static-stdlib driver flag for Linux, allowing the static linking of the standard library. This implementation largely follows the Darwin implementation in #1817, although some pecularities warrant extended discussion. The original "link with stdlib" implementation had some redundancies with getRuntimeLibraryPath; these redundancies are resolved, and the implementation alternates between getRuntimeLibraryPath and getStaticRuntimeLibraryPath cleanly as appropriate. A variety of libraries are required to link statically on Linux. The implementation currently dynamically links with them. We should probably support static linking of those as well, but I think that is beyond the scope of a -static-stdlib flag. The test coverage uses ldd here, as otool is not available on Linux. As a result, we currently have separate tests for Linux vs the other platforms; that isn't ideal, but it seems necessary. Perhaps the oddest part, and the one worth the most discussion, is the use of --dynamic-list. Inside stdlib/public/runtime/ProtocolConformances.cpp appears the following code: #elif defined(__ELF__) static int _addImageProtocolConformances(struct dl_phdr_info *info, size_t size, void *data) { // inspectArgs contains addImage*Block function and the section name InspectArgs *inspectArgs = reinterpret_cast<InspectArgs *>(data); void *handle; if (!info->dlpi_name || info->dlpi_name[0] == '\0') { handle = dlopen(nullptr, RTLD_LAZY); } else handle = dlopen(info->dlpi_name, RTLD_LAZY | RTLD_NOLOAD); auto conformances = reinterpret_cast<const uint8_t*>( dlsym(handle, inspectArgs->sectionName)); The effect of this is to search for protocol_conformances_start inside the images. However, dlsym only finds symbols that exist in the dynamic table. Failure to find the protocol conformances can be diagnosed by a "hello world" program printing String(_core: Swift._StringCore(_baseAddress: Swift.OpaquePointer(_rawValue: (Opaque Value)), _countAndFlags: Swift.UInt(_value: (Opaque Value)), _owner: Swift.Optional<Swift.AnyObject>.none)) instead of "hello world". (And also by the test coverage in this commit.) Surprisingly, this behavior can still occur on ELF platforms even if `objdump -t` reports a valid `.protocol_conformances_start`. This is because `objdump -t` searches the global table, not the dynamic table, while dlsym only searches the dynamic table. To configure objdump to search only the dynamic table, use `-T`. Inquiring minds may wonder whether dynamically-linked programs (e.g. all Linux binaries up until now) also have a broken protocol conformance table on ELF. The answer is, surprisingly, no; I checked, and ordinary ELF programs are fine. The distinction is probably this, from the ld manpage: > the dynamic symbol table will normally contain only those symbols which are referenced by some dynamic object mentioned in the link. I think the linker sees `.protocol_conformances_start` inside libswiftCore.so and erroneously concludes the one in *the executable* is "referenced by some dynamic object" (e.g. the standard library). This behavior seems to save the dyanmically-linked executable from a broken protocol conformance table. I wonder if it would be wise to apply a similar fix to dynamically-linked programs to avoid relying on the linker "helping" us here, but that's out of scope of this commit. The linker manpage reflects that many people have been bitten by dlsym "surprise", and encourages the use of `--export-dynamic`: > If you use "dlopen" to load a dynamic object which needs to refer back > to the symbols defined by the program, rather than some other dynamic > object, then you will probably need to use [--export-dynamic] when > linking the program itself. However in this situation, the use of `--export-dynamic` causes the entire stdlib to be exported, which is not ideal. However, by combining with the `--exclude-libs ALL` argument, we avoid exporting the entire stdlib.
|
@jrose-apple Good catch, it was part of an abandoned solution @jckarter and I were studying, but it's no longer necessary with the present approach |
|
@swift-ci Please test |
|
Test failure unrelated |
lmihalkovic
pushed a commit
to lmihalkovic/swift
that referenced
this pull request
May 29, 2016
[Driver] implement -static-stdlib for Linux Implement the -static-stdlib driver flag for Linux, allowing the static linking of the standard library. This implementation largely follows the Darwin implementation in apple#1817, although some pecularities warrant extended discussion. The original "link with stdlib" implementation had some redundancies with getRuntimeLibraryPath; these redundancies are resolved, and the implementation alternates between getRuntimeLibraryPath and getStaticRuntimeLibraryPath cleanly as appropriate. A variety of libraries are required to link statically on Linux. The implementation currently dynamically links with them. We should probably support static linking of those as well, but I think that is beyond the scope of a -static-stdlib flag. The test coverage uses ldd here, as otool is not available on Linux. As a result, we currently have separate tests for Linux vs the other platforms; that isn't ideal, but it seems necessary. Perhaps the oddest part, and the one worth the most discussion, is the use of --dynamic-list. Inside stdlib/public/runtime/ProtocolConformances.cpp appears the following code: #elif defined(__ELF__) static int _addImageProtocolConformances(struct dl_phdr_info *info, size_t size, void *data) { // inspectArgs contains addImage*Block function and the section name InspectArgs *inspectArgs = reinterpret_cast<InspectArgs *>(data); void *handle; if (!info->dlpi_name || info->dlpi_name[0] == '\0') { handle = dlopen(nullptr, RTLD_LAZY); } else handle = dlopen(info->dlpi_name, RTLD_LAZY | RTLD_NOLOAD); auto conformances = reinterpret_cast<const uint8_t*>( dlsym(handle, inspectArgs->sectionName)); The effect of this is to search for protocol_conformances_start inside the images. However, dlsym only finds symbols that exist in the dynamic table. Failure to find the protocol conformances can be diagnosed by a "hello world" program printing String(_core: Swift._StringCore(_baseAddress: Swift.OpaquePointer(_rawValue: (Opaque Value)), _countAndFlags: Swift.UInt(_value: (Opaque Value)), _owner: Swift.Optional<Swift.AnyObject>.none)) instead of "hello world". (And also by the test coverage in this commit.) Surprisingly, this behavior can still occur on ELF platforms even if `objdump -t` reports a valid `.protocol_conformances_start`. This is because `objdump -t` searches the global table, not the dynamic table, while dlsym only searches the dynamic table. To configure objdump to search only the dynamic table, use `-T`. Inquiring minds may wonder whether dynamically-linked programs (e.g. all Linux binaries up until now) also have a broken protocol conformance table on ELF. The answer is, surprisingly, no; I checked, and ordinary ELF programs are fine. The distinction is probably this, from the ld manpage: > the dynamic symbol table will normally contain only those symbols which are referenced by some dynamic object mentioned in the link. I think the linker sees `.protocol_conformances_start` inside libswiftCore.so and erroneously concludes the one in *the executable* is "referenced by some dynamic object" (e.g. the standard library). This behavior seems to save the dyanmically-linked executable from a broken protocol conformance table. I wonder if it would be wise to apply a similar fix to dynamically-linked programs to avoid relying on the linker "helping" us here, but that's out of scope of this commit. The linker manpage reflects that many people have been bitten by dlsym "surprise", and encourages the use of `--export-dynamic`: > If you use "dlopen" to load a dynamic object which needs to refer back > to the symbols defined by the program, rather than some other dynamic > object, then you will probably need to use [--export-dynamic] when > linking the program itself. However in this situation, the use of `--export-dynamic` causes the entire stdlib to be exported, which is not ideal. However, by combining with the `--exclude-libs ALL` argument, we avoid exporting the entire stdlib.
MaxDesiatov
pushed a commit
that referenced
this pull request
Oct 19, 2020
[pull] swiftwasm from main
finagolfin
added a commit
to finagolfin/swift
that referenced
this pull request
Dec 9, 2020
The target was added for Unix toolchains in apple#901, but a later pull apple#1891 added it again. Since clang only uses the last target flag that's passed in, all customization done for the first one was unused these last 4+ years, so remove it and change tests that look for custom strings passed by the first one.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What's in this pull request?
Depends on #1817, #1806.
This provides the Linux implementation of the
-static-stdlibflag, allowing static linking of the standard library on Linux.This follows the Darwin implementation in #1817 where possible, but there are a number of very dark corners in ELF that are involved in the implementation choices. These are discussed in more detail in the commit message, but, very briefly, we have to take several unusual steps to make protocol conformance tables work in static executables.
We introduce separate test coverage for Linux, that (in addition to the protocol conformance tables check) uses
lddto verify the linkage on the final binary.Special thanks to @jckarter @hpux735 for their amazing help running down these dark corners.
Resolved bug number: (SR-730)
Before merging this pull request to apple/swift repository:
Triggering Swift CI
The swift-ci is triggered by writing a comment on this PR addressed to the GitHub user @swift-ci. Different tests will run depending on the specific comment that you use. The currently available comments are:
Smoke Testing
Validation Testing
Note: Only members of the Apple organization can trigger swift-ci.