Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update SE-0387: Cross-Compilation Destination Bundles #1942

Merged
merged 22 commits into from
Apr 20, 2023
Merged

Update SE-0387: Cross-Compilation Destination Bundles #1942

Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
28ecd8f
Expand multiple sections, update naming of JSON props
MaxDesiatov Feb 6, 2023
e96b917
Mention package registries in the Future Directions section
MaxDesiatov Feb 6, 2023
9b56079
Specify `json5` for syntax highlighting everywhere
MaxDesiatov Feb 6, 2023
1f25bc0
Address PR feedback
MaxDesiatov Feb 7, 2023
0b52595
Address PR feedback
MaxDesiatov Feb 7, 2023
3d2a1c2
Address PR feedback
MaxDesiatov Feb 9, 2023
a9a77c1
Make `sdkRootPath` required
MaxDesiatov Feb 9, 2023
f1851cf
Address PR feedback
MaxDesiatov Feb 10, 2023
be0b74b
Disambiguate `swift build --destination` invocation example
MaxDesiatov Feb 10, 2023
cf2deb6
Add `swiftStaticResourcesPath` field to `destination.json`
MaxDesiatov Feb 10, 2023
5f39655
Clean up formatting
MaxDesiatov Feb 10, 2023
e8cd545
Fix typo
MaxDesiatov Feb 10, 2023
5d8bec2
Expand "Alternatives Considered"
MaxDesiatov Feb 21, 2023
d1f98e7
Refine wording
MaxDesiatov Feb 21, 2023
c7f5c7c
Add a link to implementation PR
MaxDesiatov Feb 28, 2023
58b2c5a
Fix toolset file reference in CLI invocation
MaxDesiatov Feb 28, 2023
515aac3
Add table of contents
MaxDesiatov Mar 2, 2023
6ce9bc1
Clean up table of contents
MaxDesiatov Mar 2, 2023
139e51f
Change `swift destination delete` to `swift destination remove` for c…
MaxDesiatov Mar 2, 2023
775b427
Update `swift destination configure` subcommand
MaxDesiatov Mar 3, 2023
db299df
Clarify command-line argument for `destination configure` subcommand
MaxDesiatov Mar 7, 2023
2d34fda
Fix `Authors` field formatting
MaxDesiatov Apr 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Mention package registries in the Future Directions section
  • Loading branch information
@MaxDesiatov
MaxDesiatov committed Feb 6, 2023
commit e96b917666b4b17050a7db55d07c9da0981bc31f
30 changes: 18 additions & 12 deletions proposals/0387-cross-compilation-destinations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ project is a daunting step that shouldn’t be required.

The solution described below is general enough to scale for any build-time/run-time triple combination.

## Proposed solution
## Proposed Solution

Since CC destination is a collection of binaries arranged in a certain directory hierarchy, it makes sense to distribute
it as an archive. We'd like to build on top of
Expand All @@ -71,7 +71,7 @@ are invoked by `swift build` instead of tools from the top-level toolchain.
The proposal is intentionally limited in scope to build-time experience and specifies only configuration metadata, basic
directory layout for proposed artifact bundles, and some CLI helpers to operate on those.

## Detailed design
## Detailed Design

### CC Destination Artifact Bundles

Expand Down Expand Up @@ -103,6 +103,7 @@ macOS would look like this:
```
swift-5.8_ubuntu.artifactbundle
├ info.json
├ toolset.json
├ ubuntu_jammy
│ ├ destination.json
│ ├ toolset.json
Expand All @@ -126,11 +127,9 @@ Here each artifact directory is dedicated to a specific CC destination, while fi
in `arm64-apple-darwin` and `x86_64-apple-darwin` subdirectories.
MaxDesiatov marked this conversation as resolved.
Show resolved Hide resolved

`info.json` bundle manifests at the root of artifact bundles should specify `"type": "crossCompilationDestination"` for
corresponding artifacts. Artifact identifiers in this manifest file uniquely identify a CC destination.
`supportedTriples` property in `info.json` should contain build-time triples that a given destination supports. An
optional `toolset` property may be supplied that specifies a toolset for the whole bundle, see [the corresponding
section below](#toolsetjson-files) for more details. The rest of the properties of bundle manifests introduced in
SE-0305 are preserved.
corresponding artifacts. Artifact identifiers in this manifest file uniquely identify a CC destination, and
`supportedTriples` property in `info.json` should contain build-time triples that a given destination supports. The rest
of the properties of bundle manifests introduced in SE-0305 are preserved.

Here's how `info.json` file could look like for `swift-5.8_ubuntu.artifactbundle` introduced in the example
above:
Expand All @@ -141,6 +140,7 @@ above:
"swift-5.8_ubuntu22.04" : {
"type" : "crossCompilationDestination",
"version" : "0.0.1",
"toolset": "toolset.json",
MaxDesiatov marked this conversation as resolved.
Show resolved Hide resolved
"variants" : [
{
"path" : "ubuntu_jammy",
Expand Down Expand Up @@ -169,7 +169,7 @@ above:
}
```

### `toolset.json` files
### `toolset.json` Files

We find that properties dedicated to tools configuration are useful outside of the cross-compilation context. Due to
that, separate toolset configuration files are introduced:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that you're removing the host variant directories, have you thought about how to have these bundles use different tools based on the host, particularly on linux? Would be good to specify that, whether you plan to have different toolsets based on the build-time triple or place them in certain directories by convention.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed host variant directories to follow what the artifact bundles proposal does. It mentions them, but ended up not using them in the example code, and I'd like to follow it as closely as possible in its implementation and bundle layout examples.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I took a look at that SE-0305 proposal, that I was not familiar with. It specifically mentions listing out paths to different tools based on the build-time triple needed, which is missing from this proposal. That creates an ambiguity after this pull, where you don't know which build-time triple each of these toolsets is built for, that wasn't there with the host variant directories before, with each presumably containing tools that run on that build host.

Copy link
Contributor Author

@MaxDesiatov MaxDesiatov Feb 9, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it does that, but then check out example code for SwiftPM command plugins that utilize artifact bundles. Host variant directories are not present there and looks like it's convention-based after all.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, not command plugins, but extensible build tools. I'm referring to this one https://github.com/abertelrud/swiftpm-buildtool-plugin-examples/tree/main/Example%201%20-%20SwiftGen/Binaries/SwiftGenBinary.artifactbundle

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO that's over-engineering toolsets for what they weren't designed for (a SwiftPM's answer to CMake toolchains, and those don't support multiple triples in a single toolchain file either). If you need different tools for a different build-time or run-time triple and Universal binaries aren't available, just create a separate toolset.

and you could use those toolsets without bundles too.

I don't think there's anything that prevents using toolsets without bundles in their current state.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you need different tools for a different build-time or run-time triple and Universal binaries aren't available, just create a separate toolset.

As you said above, that won't be enough for bundles, a whole new destination is needed.

I don't think there's anything that prevents using toolsets without bundles in their current state.

I said "those toolsets," meaning a toolset config that works on multiple types of hosts without bundles, whereas this approach you're suggesting will require separate buildtime-specific toolset configs.

I think this is worth consolidating in the toolset spec now, as the location of the build tools for different build-time triples is a single line that shouldn't require a whole new destination that is identical except for that, but up to you.

Copy link
Contributor Author

@MaxDesiatov MaxDesiatov Feb 10, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I think I see what you mean. Would a toolset at the level of artifacts reusable across multiple variants specified in info.json help this in case? I initially had it in one of the previous commits, but felt that it reached too many levels to keep in mind when maintaining toolsets for such destinations.

Copy link

@finagolfin finagolfin Feb 10, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unsure what you're proposing, but my suggestion is to add an optional dictionary to the toolset spec, that maps each of the supportedTriples to their own subdirectory of toolsetRootPath. That way SPM knows where all the build tools are and a single destination installed on linux/Windows can support multiple build-time triples, just like on Darwin.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you thought about how you want to handle this?

Expand Down Expand Up @@ -280,7 +280,7 @@ no absolute paths and no escaping symlinks are allowed. Users are still able to
outside of artifact bundles to specify additional developer tools for which no relative "non-escaping" path can be
provided within the bundle.

### `destination.json` files
### `destination.json` Files

Note the presence of `destination.json` files in each `<destination artifact>` subdirectory. These files should contain
a JSON dictionary with an evolved version of the schema of [existing `destination.json` files that SwiftPM already
Expand Down Expand Up @@ -415,7 +415,7 @@ ready, the generator copies files from the image to a corresponding `.artifactbu
The proposed `--checksum` flag provides basic means of verifying destination bundle's validity. As a future direction,
we'd like to consider sandboxing and codesigning toolchains running on macOS.

## Impact on existing packages
## Impact on Existing Packages

This is an additive change with no impact on existing packages.

Expand Down Expand Up @@ -507,15 +507,15 @@ After an application is built with a CC destination, there are other development
introduce new types of plugins invoked by `swift run` and `swift test` for purposes of remote running, debugging, and
testing. For Linux run-time triples, these plugins could delegate to Docker for running produced executables.

### `swift destination select` subcommand
### `swift destination select` Subcommand

While `swift destination select` subcommand or a similar one make sense for selecting a CC destination instead of
passing `--destination` to `swift build` every time, users will expect `swift run` and `swift test` to also work for any
destination previously passed to `swift destination select`. That’s out of scope for this proposal on its own and
depends on making plugins (from the previous subsection) or some other remote running and testing implementation to
fully work.

### SwiftPM and SourceKit-LSP improvements
### SwiftPM and SourceKit-LSP Improvements

It is a known issue that SwiftPM can’t run multiple concurrent builds for different run-time triples. This may cause
issues when SourceKit-LSP is building a project for indexing purposes (for a host platform by default), while a user may
Expand All @@ -535,3 +535,9 @@ language](https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-c
cross-compilation destination binaries are produced on the fly when needed. We don't consider this option to be mutually
exclusive with solutions proposed in this document, and so it could be explored in the future for Swift as well.
However, this requires reducing the number of dependencies that Swift runtime and core libraries have.

### Destination Bundles and Package Registries

Since `info.json` manifest files contained within bundles contain versions, it would make sense to host destination
bundles at package registries. Althouhg, it remains to be seen whether it makes sense for an arbitrary SwiftPM package
MaxDesiatov marked this conversation as resolved.
Show resolved Hide resolved
to specify a destination bundle within its list of dependencies.