Allow unit test targets to import and link executable targets

[abertelrud]

Allow unit test targets to import and link executable targets. This allows them to test everything except the main function in those targets without having to run the executable as a subprocess. With these changes they can still run the subprocess if they want to.

Motivation:

Many package authors don’t want to split their code into executable targets vs library targets, but still want to test data structures and algorithms in their executables.

Modifications:

This uses a new Swift compiler flag introduced in swiftlang/swift#35595 to compile the main symbol to a different name than _main, so that it doesn't cause symbol collisions when linked into a test executable. When linking the actual executable, the entry point symbol is then renamed back to to _main. This is done using linker flags for both Darwin and Linux in this implementation, but for linkers that don't support such options, a fallback would be to generate a stub source file along the lines of the compiler PR.

This feature is guarded with a tools version check, since packages written this way won't be testable on older toolchains.

This PR supersedes the one at #3103, which modified object files after compilation. That was a much messier solution but the best possible without support from either the compiler or linker. Now that there is compiler support, we can do better.

Modifications:

• when compiling the main module of an executable, use a Swift flag to set the name of the entry point to a name that's based on the module name
• when linking an executable, use a linker flag to rename the renamed entry point back to _main
• undo previous changes to prevent the modules of executables from being found by the Swift compiler
• added PackageGraph lookup methods for mapping targets and products back to packages (for the tools version)
• pipe through the tools version in BuildPlan so that logic that generates flags can take it into account

Result:

Unit test targets can now link executable targets that are implemented in Swift. Executables implemented using Clang targets would require a corresponding Clang flag, and are therefore currently not supported.

rdar://58122395

[abertelrud]

@swift-ci please smoke test

[abertelrud]

This restores the code from before a previous change to make it more obvious that tests couldn't import executable modules, by moving those modules into subdirectories of the main build directory.

[abertelrud]

Because Clang targets don't produce a module but instead use a module map in the public headers directory, the more obvious error is preserved when trying to import a Clang executable module from a test target. It is not preserved when importing an executable module from another one, however. We should consider moving the module back into its subdirectory and instead adding additional search paths to test targets that declare a dependency on the executable target, to preserve the good error message if an executable tries to import another executable's module.

This would be good to do in general, so that a dependency between two targets needs to be declared before the module can be imported (otherwise it leads to hard-to-diagnose linker errors rather than a more obvious cannot-import-module error).

[abertelrud]

@swift-ci please smoke test

[abertelrud]

@swift-ci please smoke test

[tomerd]

This should possibly be guarded by a tools version check, since packages written this way won't be testable on older toolchains.

makes sense

I also need to check whether this will work with the Windows linker.

cc @compnerd

[abertelrud]

Some of the macOS failures are because of unit tests that need to be updated, but there's a real issue with how gold is getting invoked that I'll need to look into.

[abertelrud]

So with the nightly toolchain the general principle seems to work:

❯ echo 'print("hello main")' > foo.swift
❯ swiftc -c -Xfrontend -entry-point-function-name -Xfrontend foo foo.swift -o foo.o
❯ nm foo.o
0000000000000000 T foo
0000000000000000 r l_entry_point
0000000000000000 r .L__unnamed_1
000000000000000d r .L__unnamed_2
000000000000000b r .L__unnamed_3
                 U $ss27_allocateUninitializedArrayySayxG_BptBwlFyp_Tg5
00000000000000c0 W $ss27_finalizeUninitializedArrayySayxGABnlF
                 U $ss5print_9separator10terminatoryypd_S2StF
0000000000000100 W $ss5print_9separator10terminatoryypd_S2StFfA0_
0000000000000120 W $ss5print_9separator10terminatoryypd_S2StFfA1_
0000000000000140 W $sSa12_endMutationyyF
                 U $sSaMa
                 U $sSS21_builtinStringLiteral17utf8CodeUnitCount7isASCIISSBp_BwBi1_tcfC
                 U $sSSN
                 U swift_bridgeObjectRelease
0000000000000000 V __swift_reflection_version
                 U swift_release
                 U $sypN
❯ swiftc foo.o -o foo
/usr/lib/gcc/x86_64-linux-gnu/5.4.0/../../../x86_64-linux-gnu/Scrt1.o:function _start: error: undefined reference to 'main'
clang-10: error: linker command failed with exit code 1 (use -v to see invocation)
<unknown>:0: error: link command failed with exit code 1 (use -v to see invocation)
❯ swiftc foo.o -Xlinker --defsym -Xlinker main=foo -o foo
❯ ./foo
hello main

[abertelrud]

I also need to check whether this will work with the Windows linker.

cc @compnerd

AFAIK, the Windows toolchain currently uses lld — is that right? If so, then --defsym should be supported. Worst case, we can always do the generated source file suggested in swiftlang/swift#35595, but a linker flag seems a little cleaner.

[abertelrud]

Interestingly, from the command line using swift-DEVELOPMENT-SNAPSHOT-2021-02-24-a-ubuntu16.04 it actually works. I wonder if the module wrapping has anything to do with this.

[abertelrud]

Looks like this happens when linking the test suites themselves as executables. That's the case in which we don't want to rename back to main, since the test stub's main is the one symbol with that name that we want to leave.

[abertelrud]

@swift-ci please smoke test

[abertelrud]

@swift-ci please smoke test

[abertelrud]

@swift-ci please smoke test

[abertelrud]

@swift-ci please smoke test

[abertelrud]

@swift-ci please smoke test

[abertelrud]

The self-hosed macOS platform test is failing because:

swift --version
+ swift --version
Apple Swift version 5.3.2 (swiftlang-1200.0.45 clang-1200.0.32.28)
Target: x86_64-apple-darwin19.6.0

This should be addressed by tying this to the package tools version.

[abertelrud]

This should be addressed by tying this to the package tools version.

Actually, that won't help — we're testing the mainline SwiftPM (with a 999.0 tools version) but using a 5.3 Swift compiler version. So we would either need detect that the compiler we're using doesn't support it, or update the self-hosted macOS tools to one that has the mainline compiler. This won't be so interesting in practice since SwiftPM ships as part of the toolchain, so it will always be paired with a recent enough compiler. So this is mostly a testing problem.

[abertelrud]

@swift-ci please smoke test

[tomerd]

looking good! can't wait to start using this

[abertelrud]

@swift-ci please smoke test

[abertelrud]

@swift-ci please smoke test

[abertelrud]

Hi @compnerd, do you know if there are any Windows linker options that could be used here for renaming <module>_main back to main, or would we need to resort to generating a source file?

[compnerd]

I don't think that there is a good way to do a rename on a symbol on Windows. I cannot think of any option in link that would let you do that. You would need to generate a new source file to accomplish it, either way a thunk or some less than ideal horrible things to do the aliasing optionally as a replacement for main.

[abertelrud]

I don't think that there is a good way to do a rename on a symbol on Windows. I cannot think of any option in link that would let you do that. You would need to generate a new source file to accomplish it, either way a thunk or some less than ideal horrible things to do the aliasing optionally as a replacement for main.

Thanks, @compnerd. Would /ALTERNATENAME:_main=_Foo_main work? That would be for LINK.EXE — if lld is used (I don't remember whether it is and don't have access to Windows), then --defsym should work.

Otherwise the generated source file would always be a last resort.

[compnerd]

The /alternatename is an undocumented flag that could work (and is what I meant by the horrible things). I don't know if that is permitted on the command line on all linkers (i.e. we need to test with both link and lld). You would need to embed that into an object file and that is why the generated source file is what came to mind. The behaviour of it is that if one symbol is not provided, this will be used as a fallback. Then the problem becomes that if you have multiple references, it could be a problem.

if lld is used (I don't remember whether it is and don't have access to Windows), then --defsym should work.

lld is meant to be 100% identical to link in invocation (and is how we use it in Swift as well). The --defsym= option might work under the MinGW mode, but that has semantic differences and would break the Windows targets in subtle ways.

For Windows, assume that the only valid way to test is link - lld should be a drop in replacement for link.

[abertelrud]

@swift-ci please smoke test

[abertelrud]

The /alternatename is an undocumented flag that could work (and is what I meant by the horrible things). I don't know if that is permitted on the command line on all linkers (i.e. we need to test with both link and lld). You would need to embed that into an object file and that is why the generated source file is what came to mind. The behaviour of it is that if one symbol is not provided, this will be used as a fallback. Then the problem becomes that if you have multiple references, it could be a problem.

if lld is used (I don't remember whether it is and don't have access to Windows), then --defsym should work.

lld is meant to be 100% identical to link in invocation (and is how we use it in Swift as well). The --defsym= option might work under the MinGW mode, but that has semantic differences and would break the Windows targets in subtle ways.

For Windows, assume that the only valid way to test is link - lld should be a drop in replacement for link.

Thanks, @compnerd. For now I have updated the PR to only do this renaming on Darwin and Linux, but I will put up a supplemental PR to generate a source file to use as fallback. I tried an attempt at doing so in this PR, but the approach suggested in the compiler PR using a Swift source file didn't work. I will add a facility to compile and link C stub source files when building executables, which might be handy in any case, and then we can use that to compile and link in this:

extern int _foo_main(int, char **);
int main(int argc, char **argv) {
   return _foo_main(argc, argv);
}

when producing the executable, which will work on any platform. The stub was the original plan and the linker flags were only an optimization to avoid this extra code generation, but the stub will work for any platform without undocumented features or magic tricks.

[dabrahams]

@abertelrud does this stuff really not work on any platform but macOS?