custom-native-toolchains aren't applied to swiftpm

[etcwilde]

I'm adding custom-native-toolchains[https://github.com//pull/58540] to make it easier to cross-compile a swift stdlib using an existing swift toolchain. While working on the patch, I got into a kerfuffle with swiftpm, llbuild, and the swift-driver on the selection of the tools to use under the "normal" macOS build flow.

While building swiftpm, the swiftpm build product selects a swiftc using self.native_toolchain_path(host_target).
https://github.com/apple/swift/blob/3ee8034c7a9f215c54e122859e06ce2b4f3ea64b/utils/swift_build_support/swift_build_support/products/swiftpm.py#L46-L50

native_toolchain_path looks for native_swift_tools_path in the arguments, and if it's there, uses that path. Otherwise it falls back on the path of the newly built toolchain.
https://github.com/apple/swift/blob/3ee8034c7a9f215c54e122859e06ce2b4f3ea64b/utils/swift_build_support/swift_build_support/products/product.py#L209-L213

Since swiftpm and the swift driver are host tools built to run on the host, they should be using the toolchain used to build the compiler. In the case of a full Canadian cross compile, the newly built compiler will likely not even run on the host, so using that compiler is entirely out of the question.

Unfortunately, there is a tangle somewhere with llbuild that I haven't been able to quite figure out at the moment.
If we change native_toolchain_path to return the path to where swiftc is located, and better, make swiftpm use the swiftc from the toolchain, we end up crossing the streams somewhere. The llbuild swift module is still being emitted with the newly built compiler instead of the original toolchain compiler. On the macOS builders, this results in an error complaining about using swift 5.5 to read a swift 5.7 swift module file.

This bug is mostly to remind me to come back to this and figure out what is going on.

[finagolfin]

I'm unclear what building SPM has to do with "cross-compile a swift stdlib using an existing swift toolchain," the two should be unrelated.

Since swiftpm and the swift driver are host tools built to run on the host, they should be using the toolchain used to build the compiler.

That is true for the early swift-driver, but not for the llbuild, SPM, and swift-driver that finally are packaged with the freshly-built toolchain, for two reasons:

1. That freshly built toolchain is made to be shipped together to another computer, so swift-driver and SPM are linked against the freshly built Swift runtime in usr/lib/swift/linux/libswiftCore.so.
2. There may be version mismatches between the prebuilt compiler and SPM/swift-driver, which are normally only built by the matching freshly built Swift compiler version or tag.

When cross-compiling SPM for Android, I get around this by passing in a prebuilt Swift compiler with the exact same version and instructing SPM to skip the SPM bootstrap and only build with this prebuilt Swift compiler along with a cross-compilation SPM configuration file for Android that I generated.

[etcwilde]

Ah, okay. I was unclear on which tools are built with which libraries and which tools. I had assumed based on the name that the native_tools_path should point to the native tools for the builder, not the tools for the host. (It would be safe to point at the freshly built compiler iff the builder and host are the same.)

Is there a document somewhere that describes what parts of the build are built to run on the builder, the host, and the target?

[finagolfin]

I had assumed based on the name that the native_tools_path should point to the native tools for the builder, not the tools for the host.

I'm not sure what you're referring to as the "builder," as you then refer separately to the target below. The Swift compiler in that path is executed, so it should run on the host building the toolchain.

Is there a document somewhere that describes what parts of the build are built to run on the builder, the host, and the target?

I don't think so, as all this just evolved over time. I'm planning to write up some doc on cross-compiling SDKs or the toolchain, but I wasn't going to include that.

[etcwilde]

As per Wikipedia/GCC/LLVM/Swift docs on cross-compiling compilers:

Builder:

The system here the package is being configured and compiled.

Host:

The system where the package tools runs (the same as the builder by default, but not necessarily the same so you can't assume they'll work on the builder)

Target:

The system for which the tools in the package compile for and the runtimes/stdlibs are compiled to run on

The package being the collection of tools including the compiler, linker, etc...

[finagolfin]

OK, in that case you were right when you "assumed based on the name that the native_tools_path should point to the native tools for the builder," but when the host is the same as the builder, it simply uses the freshly-built install toolchain path, unless manually overriden by passing in a different --native-swift-tools-path.

Of course, if you're not using the freshly built toolchain but a prebuilt one, you still have to watch out for version mismatch, depending on if the Swift component you're building is meant to run on the builder or the host.

[etcwilde]

Okay, I'm willing to close this for now. Given that swiftpm is treated as a runtime target library instead of a host tool, using the just-built compiler is the correct behavior.

[finagolfin]

It is a host tool, but since its version matches the just-built compiler, it is built with that.

You'll notice the same with swift-driver: an early swift-driver is built with any prebuilt Swift compiler, but the final swift-driver that's installed is built with the freshly-built compiler.