[SR-29] Support custom directory layouts

[krzyzanowskim]

Previous ID	SR-29
Radar	rdar://problem/28039974
Original Reporter	@krzyzanowskim
Type	New Feature
Status	Resolved
Resolution	Done

Attachment: Download

Additional Detail from JIRA

Votes	3
Component/s	Package Manager
Labels	New Feature
Assignee	None
Priority	Medium

md5: e62834d3875d27814a7aba76facdedb5

relates to:

• SR-3405 the package has an unsupported layout

Issue Description:

The Swift Package Manager relies on folder names very much. I found it impossible to create Package with folders layout not blessed by Swift Package Manager. This is issue for current project where Xcode has it's own layout of directories - incompatible with the approach of Swift Package Manager.

I tried create Package and failed. This is what I have build: https://github.com/krzyzanowskim/CryptoSwift/tree/develop/SwiftPackageManager it builds but I can't use it anywhere because it complain about folders.

and yes, I hate it 😉

[lilyball]

Xcode does not care about the folder structure on disk. Many (probably most) Xcode projects put all source files at the top level (or inside a single top-level folder) even when they have a well-defined file groupings in the project. You should be able to take an existing project and change the folder structure to match what Swift Package Manager expects without causing a problem for Xcode (although it can be a bit awkward to update your project when moving a lot of files around on disk).

[krzyzanowskim]

Depends on project change may be quite big. Bear in mind that SPM compile subfolders as separate modules, it's not very common to have Sources organised in Submodules this way - this affect all "imports" and is incompatible with Xcode build system - unless developer create separate targets. Of course one can say that SPM is for Linux only, it's just not highlighted for other developers.

[mxcl]

We intend to allow you to specify custom targets, so this ticket is now the ticket that follows that work.

The convention approach is meant to speed and ease development, but not prohibit other ways of working.

[ddunbar]

There are several things going on here.

The first thing is that the Package.swift has to be at the top-level in order for it to be recognized as a package when cloning as a dependency (we don't look anywhere else for the package manifest file). I tweaked your existing solution to do this here:
https://github.com/ddunbar/CryptoSwift
(on develop branch).

The second thing is that you do currently have to tag a version for it to be found by downstream packages. I tagged "0.1.2" although obviously this isn't the right thing to do from a develop branch. We are still figuring out how to support development versioning from dependencies...

The last thing is that even if you do all this, and then write a client like this:

import PackageDescription

let package = Package(
name: "CryptoSwiftTest",
dependencies: [
.Package(url: "https://github.com/ddunbar/CryptoSwift.git", majorVersion: 0)
]
)

it doesn't work because we don't currently pull down all of the tags, and the 0.1.2 tag won't be pulled by default because it is on an ancillary branch and not reachable from master.

You can probably get things working for experimental purposes by creating a separate repo that had the tag reachable from master, but we will work on resolving each of these issues...

[mxcl]

I have made some fixes, though a few more are required:

1. Ensure we fetch all tags: this ensures we fetch `0.1.2` even though it is not connected to the cloned branch
2. It is valid for a Package.swift to not exist at HEAD: this ensures that the fetch process completes even if HEAD does not have a Package.swift provided the tag that is picked does.

[mxcl]

Success (with pending patch):

Works with Sources subdirectory though it will also work without that now. However it tries to build the tests and fails. Which is a separate issue. But we should look into some solution, eg: allow a custom root sources to be specified, or, to allow exclusions, or to consider directories with `Tests` suffixes to be test directories.

[mxcl]

K, against HEAD the following Package.swift works for me:

import PackageDescription

let package = Package(
   dependencies: [
       .Package(url: "https://github.com/ddunbar/CryptoSwift.git", majorVersion: 0)
   ]
)

I would like to propose the following:

import PackageDescription

let package = Package(
   sourceRoot: "CryptoSwift",
   dependencies: [
       .Package(url: "https://github.com/ddunbar/CryptoSwift.git", majorVersion: 0)
   ]
)

As a first step towards allowing varied layouts.

[ddunbar]

To clarify, I guess the proposal is that sourceRoot defines the root structure under which everything is presumed to follow the package manager conventions?

The one thing I am a little worried about with this approach (as opposed to, say, an "exclusions" mechanism), is that if someones project almost fits into the layout, but not quite, it will be harder for them to adapt. But perhaps that should be a handled in another manner that fewer projects need.

[mxcl]

Yes I think this would be the first of many additions that allow tweaks. This one IMO allows a lot more projects to easily become a package without too much difficulty for them.

And yes you are correct, it is essentially saying the `Sources` directory is called `Foo`.

Perhaps you are right and it would be better to not have this and instead just add an exclusions mechanism, something like:

let package = Package(
   dependencies: [
       .Package(url: "https://github.com/ddunbar/CryptoSwift.git", majorVersion: 0)
   ]
)

package.targets.remove(name: "foo")

[ddunbar]

I think this case is one that should find into the "structured Package() initializer" declaration subset and not require extra statements. I like "sourceRoot:" better than that, but I am mixed on that versus, say:

let package = Package(
   dependencies: [
       .Package(url: "https://github.com/ddunbar/CryptoSwift.git", majorVersion: 0)
   ],
   exclude: ["Examples"]
)

I'd like it if we had a good way for evaluating this choice based on the layout of actual projects...

[krzyzanowskim]

+1 for Daniel syntax. Speaking of "exclude", I'd love to disable "generate module out of every subdirectory". Is it possible?

[swift-ci]

Comment by Kostiantyn Koval (JIRA)

Modules are generated only for subdirectories in Source. Every inner subdirectory in is just a directory.

Hello/Sources/A/A.swift
Hello/Sources/B/C/C.swift
Hello/Sources/B/D/D.swift
// 2 Targets, A.a, B.a

You would be able to exclude some with exclude.

 exclude: ["Examples"]

How would you like it to behave? Would you like to say both A and B folders are one target ?

[krzyzanowskim]

I don't know, maybe "nonTargets", or simply drop assumption that what inside is target and move to explicit definition of targets and modules

[mxcl]

The convention-based approach is designed to accelerate development for people who want to use it, but we understand not everyone enjoys such approaches and would like to make it possible to manually specify targets as you wish.

[swift-ci]

Comment by Kostiantyn Koval (JIRA)

I did sone work on this issue and added exclude directories support
#57