Debian packaging support

[melizasw]

@buttaface suggested that I start a new issue here to discuss problems that arise while I am packaging Swift for Debian. I have found the Fedora packaging that @tachoknight has done to be very helpful in this effort. A related work that I just learned of from @etcwilde is work on packaging for Ubuntu in the swift-installer-scripts repo.

The challenges are mainly getting Swift to build into a .deb package using the debuild tool and getting the packing into compliance with Debian policy. It has a very powerful linting tool that checks for many common errors and issues and most Debian Developers won't sign off on a package that isn't lint clean or reasonably close to it.

Here is the biggest issue that I know of right now. The file ConvertUTF.cpp uses a non-free license and is located in three places:

1. indexstore-db/lib/LLVMSupport/Support/ConvertUTF.cpp
2. llbuild/lib/llvm/Support/ConvertUTF.cpp
3. llvm-project/llvm/lib/Support/ConvertUTF.cpp

Debian suggests replacing these files with libicu, which is already a part of Swift. But there are other alternatives out there as well such as utfcpp.
I am not a software engineer by trade so it is unlikely that I could make this replacement on my own without making a mess of things.

Another linting error that I can probably resolve post-bulid on my own is that many libraries have executable permissions. For example:
E: swiftlang: shared-library-is-executable 0755 [usr/libexec/swift/lib/clang/10.0.0/lib/linux/libclang_rt.dyndd-x86_64.so]

And another example of how challenging this packaging can be, I can't strip the binaries or REPL won't work and when I disable the binary stripping the lint tool then complains that the binaries have not been stripped. sigh

[finagolfin]

The file ConvertUTF.cpp uses a non-free license and is located in three places

That file is under the same University of Illinois Open Source License as a lot of LLVM code. In addition, it requires that disclaimer from Unicode, Inc. in the source, not sure why a simple disclaimer is such a problem.

Since that file is compiled as part of the LLVM support library libLLVMSupport.a, which is presumably already in some Debian LLVM package, you can probably just copy what they do for that existing Debian package here.

I can't strip the binaries or REPL won't work and when I disable the binary stripping the lint tool then complains that the binaries have not been stripped

Surely Debian allows exceptions for developer tools like this that need those symbols to work?

[melizasw]

The file ConvertUTF.cpp uses a non-free license and is located in three places

That file is under the same University of Illinois Open Source License as a lot of LLVM code. In addition, it requires that disclaimer from Unicode, Inc. in the source, not sure why a simple disclaimer is such a problem.

Since that file is compiled as part of the LLVM support library libLLVMSupport.a, which is presumably already in some Debian LLVM package, you can probably just copy what they do for that existing Debian package here.

That was a great idea. The llvm-toolchain-14 package has 840 lint errors including this one:
llvm-toolchain-14 | 1:14.0.6-2 |   | error | license-problem-convert-utf-code | [llvm/lib/Support/ConvertUTF.cpp]

I am able to see what they put in their copyright file and will do the same thing. The lint error will just have to be overridden. I see a lot of familiar lint errors in that llvm package so maybe this won't be as big of an issue as I had thought.

I can't strip the binaries or REPL won't work and when I disable the binary stripping the lint tool then complains that the binaries have not been stripped

Surely Debian allows exceptions for developer tools like this that need those symbols to work?

I'm sure they do. I was just complaining how this turns into a three step process of disabling the stripping, disabling the lint errors that result, then probably having to explain to a Debian Developer why stripping was disabled.

Once I have the low hanging lint errors cleaned up I'll start working on getting the remaining errors reviewed by a Debian Developer to see if they have any concerns that I need to address.

[melizasw]

Here is something else that would help me. Anything that lintian flags as an error is considered to be either a bug in the lint tool or a Debian policy violation. That said, it seems like there is some understanding that it isn't always that simple and overrides of rules are allowed.

For the following lintian errors it would be very helpful to know if the error is something I should be able to fix, or is expected. And if expected, what is the reason why. Any explanation that I can add to the comments of the lintian overrides will help to get approval to add the package to Debian. Most errors have multiple failures, but only the first is shown for brevity.

E: swift source: binary-with-bad-dynamic-table [llvm-project/llvm/test/tools/llvm-xray/X86/Inputs/elf64-objcopied-instrmap.bin]
N: 
N:   This appears to be an ELF file. According to readelf, the program headers
N:   suggests it should have a dynamic section, but readelf cannot find it.
N:   
N:   If it is meant to be external debugging symbols for another file, it
N:   should be installed under /usr/lib/debug. Otherwise, this could be a
N:   corrupt ELF file.

E: swiftlang: custom-library-search-path RUNPATH swift_5.6.2/build/buildbot_linux/llvm-linux-x86_64/lib [usr/libexec/swift/bin/lldb-argdumper]
N: 
N:   The binary or shared library sets RPATH or RUNPATH. This overrides the
N:   normal library search path, possibly interfering with local policy and
N:   causing problems for multilib, among other issues.
N:   
N:   The only time a binary or shared library in a Debian package should set
N:   RPATH or RUNPATH is if it is linked to private shared libraries in the
N:   same package. In that case, place those private shared libraries in
N:   /usr/lib/*package*. Libraries used by binaries in other packages should be
N:   placed in /lib or /usr/lib as appropriate, with a proper SONAME, in which
N:   case RPATH/RUNPATH is unnecessary.
N:   
N:   To fix this problem, look for link lines like: gcc test.o -o test
N:   -Wl,--rpath,/usr/local/lib or gcc test.o -o test -R/usr/local/lib and
N:   remove the -Wl,--rpath or -R argument. You can also use the chrpath
N:   utility to remove the RPATH.
N: 
N:   Please refer to https://wiki.debian.org/RpathIssue for details.

This should be pretty easy for me to fix if this is just an artifact of packaging and not a responsibility of the swift build scripts.

E: swift source: missing-notice-file-for-apache-license [swift-crypto/NOTICE.txt]
N: 
N:   The package appears to be licensed under the Apache 2.0 license and a
N:   NOTICE file (or similar) exists in the source tree. However, no files
N:   called NOTICE or NOTICE.txt are installed in any of the binary packages.
N:   
N:   The Apache 2.0 license requires distributing of such files:
N:   
N:    (d) If the Work includes a "NOTICE" text file as part of its
N:        distribution, then any Derivative Works that You distribute must
N:        include a readable copy of the attribution notices contained
N:        within such NOTICE file [..]
N:   
N:   Please include the file in your package, for example by adding
N:   path/to/NOTICE to a debian/package.docs file.
N: 
N:   Please refer to usr/share/common-licenses/Apache-2.0 for details.

E: swiftlang: no-code-sections [usr/libexec/swift/lib/swift_static/linux/libicudataswift.a]
N: 
N:   
N:   The named members of the static library have no usable code sections.
N:   
N:   It happens when shared objects are built with -flto=auto but without
N:   -ffat-lto-objects. dh_strip strips the LTO sections but may leave the
N:   static library without any usable code.
N: 
N:   Please refer to Bug#977596 for details.

These don't need to be executable do they? I'm not even sure just how they are getting +x.

E: swiftlang: shared-library-is-executable 0755 [usr/libexec/swift/lib/clang/13.0.0/lib/linux/libclang_rt.asan-x86_64.so]
N: 
N:   Shared libraries should be mode 0644.
N: 
N:   Please refer to Run-time shared libraries (Section 8.1) in the Debian
N:   Policy Manual for details.

There are many of this kind. I'm not sure what the issue is as they don't go into the package. Maybe there is a way to clean them up just prior to running lintian?

E: swift source: source-is-missing [cmake/Tests/CMakeTests/ELF/elf32lsb.bin]
N: 
N:   The source of the following file is missing. Lintian checked a few
N:   possible paths to find the source, and did not find it.
N:   
N:   Please repack your package to include the source or add it to
N:   "debian/missing-sources" directory.
N:   
N:   Please note, that very-long-line-length-in-source-file tagged files are
N:   likely tagged source-is-missing. It is a feature not a bug.

This is just a warning, but it makes me wonder if I built Swift incorrectly? Maybe it doesn't matter as this is pertaining to the debug symbol package (swiftlang-dbgsym_5.6.2-1_amd64.deb) and I don't think that would be getting added.

W: swiftlang-dbgsym: debug-file-with-no-debug-symbols [usr/lib/debug/.build-id/00/bb2a059d939872ab355e8fa581567eac042f7d.debug]
N: 
N:   The binary is installed as a detached "debug symbols" ELF file, but it
N:   does not appear to have debug information associated with it.
N:   
N:   A common cause is not passing -g to GCC when compiling.
N:   
N:   Implementation detail: Lintian checks for the ".debug_line" and the
N:   ".debug_str" sections. If either of these are present, the binary is
N:   assumed to contain debug information.
N: 
N:   Please refer to Bug#668437 for details.

If anyone can help make sense of any of these lintian issues that would be most helpful.

[finagolfin]

• The test files, elf64-objcopied-instrmap.bin and elf32lsb.bin, I'm not sure why your linter is even looking at those: you're not running the tests and those files are not installed at the end, are they? The first one is from LLVM, so I'd look at how the existing package handles that.
• The RUNPATH issue is a definite bug on our end, specifically with lldb to have an absolute path like that. Please look into where that's getting added, and I bet the LLVM devs would accept a patch to remove that.
• The missing license file may be an oversight on our end, unsure as pretty much all Swift repos use that same Apache license, not sure why your linter is pointing just that one out.
• The ICU issue needs to be investigated, could be legit. You could also try removing the Swift build of libicu and just use the Debian system package, which should work too but willmay require you to bump the Swift package anytime the system libicu is updated.
• Those libcompiler-rt shared libraries don't need executable permissions, probably a mistake on our end.
• I have no idea about that debug file with no symbols, haven't used that.

[melizasw]

• The test files, elf64-objcopied-instrmap.bin and elf32lsb.bin, I'm not sure why your linter is even looking at those: you're not running the tests and those files are not installed at the end, are they? The first one is from LLVM, so I'd look at how the existing package handles that.

I looked up Debian's llvm-toolkit-14 package and wouldn't you know it, they have these same lintian errors and many of the others that I am seeing. If they can have them in llvm then swift should be allowed them as well. The lint tool looks at both the completed package as well as the source, so it seems to pick up all sorts of files that shouldn't matter.

* The `RUNPATH` issue is a definite bug on our end, specifically with lldb to have an absolute path like that. Please look into where that's getting added, and I bet the LLVM devs would accept a patch to remove that.

I'm going to look into this some more, but maybe it isn't an issue. Based on the Debian wiki this forces the use of the custom Swift libraries, which I presume is better than picking up the system's libraries for llvm and others. Again, the Debian llvm package has these errors as well so I will contact the maintainer to find out what I can.

* The missing license file may be an oversight on our end, unsure as pretty much all Swift repos use that same Apache license, not sure why your linter is pointing just that one out.

For now I'm going to let the Debian build scripts copy over the file and see if that resolves things.

* The ICU issue needs to be investigated, could be legit. You could also try removing the Swift build of libicu and just use the Debian system package, which should work too but ~will~may require you to bump the Swift package anytime the system libicu is updated.

Well, wouldn't you know it. Debian's libicu-dev package has the exact same error on libicudata.a. So I guess nothing to worry about there.

* Those libcompiler-rt shared libraries don't need executable permissions, probably a mistake on our end.here

I was able to get those fixed by manually running chmod at the appropriate place in the Debian packaging flow. There is a tool that is supposed to fix this automatically for me so maybe it has a bug. I will have to look into that. At no point in the build process do I see the swift tools setting these files to anything but 0644 and they don't become 0755 until they are copied out into the final directory structure so I'm not sure yet who is doing it.

* I have no idea about that debug file with no symbols, haven't used that.

Yah, probably not an issue.

I've just realized that the person that originally wanted to package Swift for Debian back in 2015 was and is a Debian Developer with the LLVM packaging team. If I can pique his interest to sponsor/mentor this package then he already has half of the answers and might even jump at the opportunity to get involved with it again. Crossing my fingers!

[melizasw]

I'm at the point of trying to get a Debian Developer to sponsor my package and clean up any policy violations. The violation that was discovered first is that there is an existing Debian packaged named "python3-swiftclient" and they supply a binary named "swift". Since their binary does something completely different I am not allowed to have a binary named "swift" no matter how unlikely it is for someone to want to use the swift that is designed for server object storage and also compile Swift programs on that same server. I could use the "conflict" mechanism to prevent my package and theirs from being installed at the same time, but policy does not allow that unless the binaries provide the same functionality.

The recommended action is to contact the debian-devel mailing list to start a discussion of who has to change their name, or if both have to change their name. At the same time I could ask for an exception to the "conflict" mechanism rule.

I guess my main question here is that since "swift" and "swiftc" are both just symbolic links to "swift-frontend" could I drop "swift" from my package and thus require Debian users to always call swiftc instead? Would that break anything other than reams of documentation?

[finagolfin]

since "swift" and "swiftc" are both just symbolic links to "swift-frontend" could I drop "swift" from my package and thus require Debian users to always call swiftc instead?

swift has two purposes: it invokes the REPL when run alone and it interprets a Swift file when given one. As such, you could always drop it as long as you don't care about those two uses, or look into renaming it instead.

Would that break anything other than reams of documentation?

No, I doubt there's even that much doc on it.

[melizasw]

I don't know yet if this will be allowed by the Debian policy, but what I have done is create a set of scripts such that during the installation process if /usr/bin/swift does not exist the user is presented with a prompt asking if they wish to add it or not. If an actual binary file is located at /usr/bin/swift instead of a symbolic link then a message is printed out warning that "swift" already exists and it cannot be used to call the Swift REPL or compiler/interpreter.

[melizasw]

I've been copying /usr/bin/ls to /usr/bin/swift to test the Debian package install in the case where the "swift" package is installed. If I forget to remove this copy before building Swift then it tries to build newswiftdriver with /usr/bin/swift and fails with a less than helpful message. I'm not sure what all I can do on my end to address this (and may not need to do anything), but I'm wondering if Swift can do a sanity check on /usr/bin/swift before trying to build with it. At a minimum it could verify that it is indeed Swift and is past some minimum version since we know Swift 5.5.1 won't work.

[finagolfin]

I'm wondering if Swift can do a sanity check on /usr/bin/swift before trying to build with it. At a minimum it could verify that it is indeed Swift and is past some minimum version since we know Swift 5.5.1 won't work.

Are you sure the compiler build even uses swift and not swiftc? If so, feel free to submit a pull to do that sanity check for the swift toolchain build.

[melizasw]

Are you sure the compiler build even uses swift and not swiftc? If so, feel free to submit a pull to do that sanity check for the swift toolchain build.

I tried to reproduce this error on a Swift 5.5.1 build, but was unable to. Here is the log file from trying to build Swift 5.6.2 when Swift is not installed but /usr/bin/swift is some other binary than Swift (copied from /usr/bin/ls in this case):

swift/utils/build-script --preset=buildbot_linux,no_test install_destdir=/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2 installable_package=/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swiftlang_5.6.2-1-debian.tar.gz
[swift/utils/build-script] NOTE: using preset "buildbot_linux,no_test", which expands to 

swift/utils/build-script --assertions --swift-enable-ast-verifier=0 --no-swift-stdlib-assertions '--swift-install-components=autolink-driver;compiler;clang-resource-dir-symlink;stdlib;swift-remote-mirror;sdk-overlay;parser-lib;toolchain-tools;license;sourcekit-inproc' '--llvm-install-components=llvm-cov;llvm-profdata;IndexStore;clang;clang-resource-headers;compiler-rt;clangd;lld;LTO' --llbuild --swiftpm --swift-driver --xctest --libicu --libcxx --swiftdocc --build-ninja --install-llvm --install-swift --install-lldb --install-llbuild --install-swiftpm --install-swift-driver --install-xctest --install-libicu --install-prefix=/usr --install-libcxx --install-sourcekit-lsp --install-swiftdocc --build-swift-static-stdlib --build-swift-static-sdk-overlay --build-swift-stdlib-unittest-extra --test-installable-package --toolchain-benchmarks --install-destdir=/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2 --installable-package=/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swiftlang_5.6.2-1-debian.tar.gz --relocate-xdg-cache-home-under-build-subdir --build-subdir=buildbot_linux --lldb --release --test --validation-test --long-test --stress-test --test-optimized --foundation --libdispatch --indexstore-db --sourcekit-lsp --swiftdocc '--lit-args=-v --time-tests' --lldb-test-swift-only --install-foundation --install-libdispatch --reconfigure --skip-test-cmark --skip-test-lldb --skip-test-swift --skip-test-llbuild --skip-test-swiftpm --skip-test-swift-driver --skip-test-xctest --skip-test-foundation --skip-test-libdispatch --skip-test-playgroundsupport --skip-test-libicu --skip-test-indexstore-db --skip-test-sourcekit-lsp --skip-test-swiftdocc

[swift/utils/build-script] NOTE: Using toolchain default
+ cmake --version
+ mkdir -p /home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/build/buildbot_linux
+ rm -rf /home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/build/buildbot_linux/ninja-build
+ cp -r /home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/ninja /home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/build/buildbot_linux/ninja-build
+ pushd /home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/build/buildbot_linux/ninja-build
+ env CXX=/usr/bin/clang++ /usr/bin/python3 configure.py --bootstrap
[1/29][  3%][0.045s] INLINE build/browse_py.h
[2/29][  6%][0.057s] CXX build/debug_flags.o
[3/29][ 10%][0.635s] CXX build/browse.o
[4/29][ 13%][0.787s] CXX build/depfile_parser.o
[5/29][ 17%][0.971s] CXX build/disk_interface.o
[6/29][ 20%][1.180s] CXX build/clparser.o
[7/29][ 24%][1.208s] CXX build/edit_distance.o
[8/29][ 27%][1.449s] CXX build/dyndep.o
[9/29][ 31%][1.652s] CXX build/deps_log.o
[10/29][ 34%][2.033s] CXX build/clean.o
[11/29][ 37%][2.073s] CXX build/line_printer.o
[12/29][ 41%][2.137s] CXX build/dyndep_parser.o
[13/29][ 44%][2.163s] CXX build/build_log.o
[14/29][ 48%][2.252s] CXX build/lexer.o
[15/29][ 51%][2.430s] CXX build/graphviz.o
[16/29][ 55%][2.691s] CXX build/metrics.o
[17/29][ 58%][2.967s] CXX build/version.o
[18/29][ 62%][3.019s] CXX build/eval_env.o
[19/29][ 65%][3.108s] CXX build/string_piece_util.o
[20/29][ 68%][3.210s] CXX build/parser.o
[21/29][ 72%][3.246s] CXX build/util.o
[22/29][ 75%][3.353s] CXX build/graph.o
[23/29][ 79%][3.606s] CXX build/subprocess-posix.o
[24/29][ 82%][3.873s] CXX build/manifest_parser.o
[25/29][ 86%][3.964s] CXX build/build.o
[26/29][ 89%][4.120s] CXX build/state.o
[27/29][ 93%][4.192s] AR build/libninja.a
[28/29][ 96%][4.692s] CXX build/ninja.o
[29/29][100%][4.813s] LINK ninja
bootstrapping ninja...
warning: A compatible version of re2c (>= 0.11.3) was not found; changes to src/*.in.cc will not affect your build.
wrote build.ninja.
bootstrap complete.  rebuilding...
+ popd
--- Building earlyswiftdriver ---
Traceback (most recent call last):
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/build-script", line 722, in <module>
    sys.exit(main())
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/build-script", line 717, in main
    return main_normal()
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/build-script", line 673, in main_normal
    invocation.execute()
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/swift_build_support/swift_build_support/build_script_invocation.py", line 669, in execute
    self._execute(pipeline, all_host_names)
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/swift_build_support/swift_build_support/build_script_invocation.py", line 727, in _execute
    self.execute_product_build_steps(product_class, host_target)
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/swift_build_support/swift_build_support/build_script_invocation.py", line 783, in execute_product_build_steps
    product.build(host_target)
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/swift_build_support/swift_build_support/products/earlyswiftdriver.py", line 69, in build
    run_build_script_helper('build', host_target, self, self.args)
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/swift_build_support/swift_build_support/products/earlyswiftdriver.py", line 107, in run_build_script_helper
    swiftc_path = os.path.abspath(product.toolchain.swiftc)
  File "/usr/lib/python3.10/posixpath.py", line 378, in abspath
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType
ERROR: command terminated with a non-zero exit status 1, aborting

I'm not likely to be doing any debug or pull requests on this any time soon as I'm focused on packaging.

[tachoknight]

I had a similar issue with Swift being installed while packaging, but that was a mistake on my part (testing one version while building another) and the Fedora builders[1] won't have Swift installed by default.

[1] Dunno how Debian works, but all packages in the Fedora/RHEL environment are built from source on Fedora-controlled hardware, essentially in a special spun-up container; if I don't include it in the %BuildRequires section, it won't be in the container (which was how I discovered building Swift requires rsync 🙃).

[melizasw]

Yes, it is the same with Debian. I should be using chroot and a clean build environment each time, but I'm not there yet. There is pretty much no chance that the official Debian package would ever get built with that other "swift" package installed. I have rsync in my build dependencies as well, not because I discovered that on my own, but from benefiting from your work. :-)

I've still got some things to sort out, but I'm going to ask if what I have now is good enough to upload to Debian as a starting point.

[tachoknight]

out of curiosity, and because I don't have a Debian box handy, what is the other swift program? In Fedora's case there is another swift, but it's part of the OpenStack Object Storage API Python library, so the chance of a conflict with Swift® is effectively nil.

[melizasw]

On Debian/Ubuntu/Pop_OS there is another package "swift" but it doesn't actually provide the binary /usr/bin/swift, that is provided by the related python-swiftclient and they are indeed part of the OpenStack Object Storage API Python library.

$ sudo nala install python3-swiftclient
$ which swift
/usr/bin/swift
$ swift --version
python-swiftclient 3.13.1

I'd prefer to have swiftlang refuse to install if python3-swiftclient is installed, but Debian policy prohibits two packages from providing the same binary unless they are are the same program having different implementations. I could ask for an exception to this policy, but I think prompting to make the link works too.

[tachoknight]

Not sure if I got the reviewer on a good day, but I basically said that there is practically zero chance of conflict, given the usage domains were so wildly different; a system using OpenStack Storage is unlikely to also be a developer's machine wanting to use a cool programming language.

[finagolfin]

Here is the log file from trying to build Swift 5.6.2 when Swift is not installed but /usr/bin/swift is some other binary than Swift (copied from /usr/bin/ls in this case)

It looks like it's reporting correctly that there's no swiftc, though the error is a bit weird. It has nothing to do with the fake swift binary you added.

[melizasw]

It looks like it's reporting correctly that there's no swiftc, though the error is a bit weird. It has nothing to do with the fake swift binary you added.

And yet I can repeatedly turn this error on and off by copying a binary to /usr/bin/swift or deleting it.

[melizasw]

Not sure if I got the reviewer on a good day, but I basically said that there is practically zero chance of conflict, given the usage domains were so wildly different; a system using OpenStack Storage is unlikely to also be a developer's machine wanting to use a cool programming language.

Good to know. I've asked my package sponsor how best to handle this and mentioned this as being what Fedora has done. I think the only issue would be not having separate runtime libraries and needing to install swiftlang to get them. Have you done any work on exploring separating out the runtime libraries into their own package?

[finagolfin]

And yet I can repeatedly turn this error on and off by copying a binary to /usr/bin/swift or deleting it.

Ah, you're right, found it. The current build checks for both swift and swiftc and tries to build the early swift driver if it finds just the former, as it assumes both will be installed together. Obviously, that assumption will break if your other OpenStack package is installed instead. I wouldn't worry about this, as very few will likely build the Swift toolchain, and almost nobody will do it with that other Swift package installed.

[melizasw]

@tachoknight my Debian sponsor has asked me to move from the Swift provided clang, llvm, cmake, and icu packages to those that come with Debian. I know that lldb is customized by Swift as mentioned in your blog post, but what of these others? I'm assuming that icu and cmake would be fine as long as Swift doesn't depend on a specific version's features.

Do you know and/or do you have a blog post about that sort of issues that I can point him to? He is one of the Debian LLVM package maintainers so he is very experienced in this area.

Of course anyone else that knows is welcome to chime in too.

[tachoknight]

This was my experience as well. The Swift-provided version of LLVM/LLDB is custom for Swift; the standard LLVM/LLDB you might install on any Debian or Fedora box knows nothing about Swift and it simply won't work. The solution I use in Fedora is to hide everything in /usr/libexec/swift and just symlink swift and swiftc to /usr/bin; the custom lldb executable for Swift is not on the path so there's no collision with an existing LLVM/LLDB installation.

Two things I can remember too: When I was originally getting it working with Fedora, I discovered that a lot of of the code assumes executables are in the same directory, or if dynamically loading a library, it will explicitly say something like ../lib/libFoo.so or whatever. The Swift toolchain is a complete package unto itself and thus it really needs to stay together; many early bugs with Swift on Fedora was simply due to constantly missing a spot where I had tried to redirect the loader to a more general location (e.g. Fedora wants all its 64-bit libs in /usr/lib64) and it was really frustrating. Thus when I was able to just drop everything into /usr/libexec/swift, it made life so much easier, and so many patches (and related bugs) just magically went away.
The other thing that has come up, and I want to say it's somewhere on the Swift forums, that there was some discussion about moving Swift upstream to be part of the LLVM project. I have no idea what, if any, plans there are to do that and there's no point in thinking about it (this is to head off the possibility of someone asking "Why don't they just move everything to the LLVM project?").

[melizasw]

Thanks @tachoknight, I have forwarded your reply to my sponsor in the hopes that we will agree to keeping Swift together until upstream makes it possible for us to more easily split it apart.

[finagolfin]

As for CMake and ICU, the Swift build just uses the upstream source for each of those, so no problem with not building those from source and using the prebuilt Debian packages instead, provided they're not really old.

The only issue is that if you use the system ICU, you may have to bump the Foundation build everytime the system ICU is updated, to make sure it's linked against the latest version, whereas you don't need to do that if you build another libicu for Swift and stash it separately with the Swift-forked lldb and so on.

[finagolfin]

there was some discussion about moving Swift upstream to be part of the LLVM project.

Not Swift, I believe they just want to slowly upstream all Swift changes from their forked LLVM/clang/lldb to the upstream LLVM repo and get rid of those LLVM forks one day, but I'm guessing that's probably years away.

[tachoknight]

Ah, that's right, I was remembering it incorrectly.

[melizasw]

Thank you all for the information and help. My sponsor would like me to move to as many Debian packages as possible so here is our plan:

1. Switch to Debian cmake and ninja (Debian ftpmasters will reject this package otherwise)
2. Release swiftlang to Debian Experimental
3. Switch to Debian icu, llvm, and clang
4. Release to Debian Testing (from here it migrates to unstable after 5 days then to stable releases and derivative distributions)

[tachoknight]

Out of curiosity, what does it mean to switch to the Debian llvm and clang?

[melizasw]

Out of curiosity, what does it mean to switch to the Debian llvm and clang?

I'm hoping that will be open for negotiation. The desire is to not be packaging yet another fork of llvm/lldb/clang and having to deal with its bugs, but how do I make use of a tool chain that doesn't support my language? Start submitting all of Swift's changes as pull requests to Debian's LLVM packages? At that point it seems like you're still having to maintain a fork, but now those bugs impact far more Debian users than just the swiflang users. In any case, I must at least give it the old college try.

[stevapple]

The CYaml target from Yams is identically libyaml, so it's true that every executable target depending on Yams will embed a copy of libyaml. I have tried to build Yams against libyaml-dev (with SwiftPM) and it just works.

I read through the Debian policies and, although it's technically possible to override the tag, it seems this is still likely being rejected because libyaml is already provided by Debian.

Since Yams is owned and maintained by @jpsim, I think it's eventually up to him if CYaml can be replaced by system-wide libyaml. Not too many technical blockers here, but it adds to build-system complexity.

[melizasw]

@stevapple Thanks for explaining it to me. It sounds like I need to modify Yams to pull in the system libyaml instead of building its own CYaml library which is a duplication of libyaml. I haven't got it working yet, but that is what I am trying to do now.

[melizasw]

It took me a while to get both the Swift code and Cmake files changed to pull in the system's libyaml, but it is now working. The swiftlang package is in the NEW queue for Debian Experimental. From here it is a matter of waiting days or weeks until it gets reviewed and either accepted or rejected. And then after that it still needs to get into Unstable before it goes anywhere useful. This is a major milestone and my thanks to everyone that has helped.

[tachoknight]

Congrats @melizasw! Hope it all goes well, will be very cool to apt-get install swiftlang! Out of curiosity, what architectures are you building it for, x864_64 as well as aarch64?

[stevapple]

@melizasw Some questions around libswiftlang:

• Is it required to be extracted into a separate package? I believe so far the community still don’t agree on which part of Swift is runtime, toolchain or SDK except for the established pattern on macOS — and the macOS case is so special that it may not work on other platforms. I wish to delay separate packaging until the issue is fully discussed.
• Is the name libswiftlang the only option in respect of Debian conventions? If feels a little weird to me though.
• I see ./usr/libexec/swift/lib/swift/linux/libXCTest.so in libswiftlang and I would say, on both macOS and Windows we’re not treating XCTest as a runtime library. It is part of the development suite which shouldn’t be leaked into the system.
• If we want to make libswiftlang a usable thing, is ./usr/lib a more suitable place than ./usr/libexec/swift/lib?

---

Update: (some more questions)

• swiftlang: Why is swift not linked into /usr/bin?
• swiftlang-dev: I may come a bit late but is it really necessary to separate this?
• swiftlang-doc: Will it be better if we put man documents here?

[melizasw]

Congrats @melizasw! Hope it all goes well, will be very cool to apt-get install swiftlang! Out of curiosity, what architectures are you building it for, x864_64 as well as aarch64?

Uhhh. Yes? I admit I'm not very familiar with the packaging process yet. But in the past new applications would get built for all architectures for which all dependencies could be met. The swiftlang package is marked as having an architecture of "any" so I think it will be the same way.

[melizasw]

@melizasw Some questions around libswiftlang:

* Is it required to be extracted into a separate package? I believe so far the community still don’t agree on _which part of Swift is runtime, toolchain or SDK_ except for the established pattern on macOS — and the macOS case is so special that it may not work on other platforms. I wish to delay separate packaging until the issue is fully discussed.

It is not required, I did it to provide runtime libraries in a much smaller and lighter package than the full swiftlang package. I do not wish to delay and am willing to move libraries between packages when a formal decision has been made. Worst case is that someone installs libswiftlang only and a Swift binary won't run as it is missing a library that was put into the swiftlang package. In the interim I don't think there is any harm in having internal libraries exposed in libswiftlang unless people think that is an endorsement to link to them for their own binaries.

• Is the name libswiftlang the only option in respect of Debian conventions? If feels a little weird to me though.

No, it could have been swift_lang as it is in Fedora or swiftc as was done with rust. The package name swift is already taken. So I opted to use swiftlang which is the name that was used in the ITP bug that was created in 2015.

• I see ./usr/libexec/swift/lib/swift/linux/libXCTest.so in libswiftlang and I would say, on both macOS and Windows we’re not treating XCTest as a runtime library. It is part of the development suite which shouldn’t be leaked into the system.

I will remove this from the libswiftlang package and put it in the swiftlang package.

• If we want to make libswiftlang a usable thing, is ./usr/lib a more suitable place than ./usr/libexec/swift/lib?

That would be preferred and I will give it a try, but I'm not sure how well the rest of Swift will handle having its libraries moved out of adjacent directories. I've found libswiftlang usable as-is by building a simple binary on Debian and running it on Pop_OS where only libswiftlang was installed.

• swiftlang: Why is swift not linked into /usr/bin?

Debian policy prohibits me from supplying /usr/bin/swift as there already exists one as provided by python3-swiftclient. Maybe we can talk them into picking a new name and freeing up /usr/bin/swift, but until then I cannot provide it. What I have done instead is provide a prompt during the installation of swiftlang to create the link to /usr/bin/swift if it does not already exist and the user selects to create it.

• swiftlang-dev: I may come a bit late but is it really necessary to separate this?

Only one package is necessary, which is the approach taken by @tachoknight for Fedora. My Debian sponsor requested a -dev and -doc package so he gets them

• swiftlang-doc: Will it be better if we put man documents here?

I don't believe so, no. Debian policy requires a man page for each binary in /usr/bin so those man pages have to go with their binaries in swiftlang. They also live in the debian directory of the package so any man pages provided by upstream have to be moved as part of the packaging process.

[jpsim]

Since Yams is owned and maintained by @jpsim, I think it's eventually up to him if CYaml can be replaced by system-wide libyaml. Not too many technical blockers here, but it adds to build-system complexity.

@melizasw @stevapple I'm open to this, just file a PR and we can discuss the specifics.

[stevapple]

No, it could have been swift_lang as it is in Fedora or swiftc as was done with rust. The package name swift is already taken. So I opted to use swiftlang which is the name that was used in the ITP bug that was created in 2015.

Ah, I was explicitly asking for libswiftlang, but now I get the naming at some point.

That would be preferred and I will give it a try, but I'm not sure how well the rest of Swift will handle having its libraries moved out of adjacent directories. I've found libswiftlang usable as-is by building a simple binary on Debian and running it on Pop_OS where only libswiftlang was installed.

My concern is that placing the libraries in /usr/libexec/swift/lib diverges from the current installation (and also the slim Docker image). And since SE-0342 will be a thing (hopefully soon), there should be less need for a runtime package.

Debian policy prohibits me from supplying /usr/bin/swift as there already exists one as provided by python3-swiftclient. Maybe we can talk them into picking a new name and freeing up /usr/bin/swift, but until then I cannot provide it. What I have done instead is provide a prompt during the installation of swiftlang to create the link to /usr/bin/swift if it does not already exist and the user selects to create it.

I see. Asking them to rename seems not so realistic, and your workaround makes sense👍 I wonder if this is a valid use case for update-alternative...

Only one package is necessary, which is the approach taken by @tachoknight for Fedora. My Debian sponsor requested a -dev and -doc package so he gets them

swiftlang-dev arouses new questions to me now:

• Naming. I think libswiftlang-dev should be the correct name, to correctly indicate that the package is used for developing apps that use libswiftlang, not swiftlang (eg. some imaginary Swift IDE).
• Contents. The C++ headers are one of the undecided pieces (on Windows we package it along with SDK, while on macOS both toolchain and SDK hold it). Also, Swift modules should inevitably be a part of development files IMO.

---

Some more thoughts:

• It would be nice have new packages like libswiftlang-dev-arm64-cross to easily enable cross compilation. Actually adopting this feature will reflect some short-sighted designs in toolchain and layout.
• I wonder if it's necessary to extract a swiftlang-common package, and keep swiftlang as simplified as what the user can directly see (and use).
• It's interesting to see what could be in swiftlang-dev — the package that contains development files for the Swift compiler itself.
• Are we going to extract sourcekit-lsp and docc (and even SwiftPM) into separate packages? Depending on use cases, not every Swift user is using these tools. And all these components are pluggable, so it should be fairly easy.🤔

[melizasw]

My concern is that placing the libraries in /usr/libexec/swift/lib diverges from the current installation (and also the slim Docker image). And since SE-0342 will be a thing (hopefully soon), there should be less need for a runtime package.

That was an interesting read. One of the premises of SE-0342 is that Linux does not ship with Swift runtime libraries. This will no longer be true of Debian and its derivatives at some point. It is somewhat circular to say that libswiftlang is not needed due to SE-0342 and SE-0342 is needed (in part) because there are no runtime libraries on Linux. Obviously there is much more to consider with static vs dynamic linking, but giving the option of a runtime library package for now seems like a good idea to me. I'll look into changing that lib path though.

I see. Asking them to rename seems not so realistic, and your workaround makes sense👍 I wonder if this is a valid use case for update-alternative...

I don't think it would be allowed since the requirement for update-alternative is that they provide the same kind of functionality. I initially tried to flag swiftlang and being a conflict with package python3-swiftclient, but that isn't allowed unless they are identically functional but having different implementations. I'm not even sure asking the user to create a dynamic link during install will be permitted, but it makes sense to me and we decided to give it a try.

As for the rest of your comments, I appreciate you taking the time to leave them and I will spend more time later looking into them and thinking about what changes could be made to make the packaging better. But for now, lunch break is over.

[melizasw]

Another significant milestone was reached today. My Swift 5.6.3 package was approved by the Debian FTP team and is now in experimental. That is still not accessible for most people to install, but it means that the FTP team approved of the various packaging methodologies and workarounds that were developed with your help. From here on out there is no manual review by the FTP team so when approved by my sponsor, it can move out of experimental and start making its way out into the world.

You can see some info about the package here: https://tracker.debian.org/pkg/swiftlang
On the left side under "binaries" you can click to see more info about each of the four packages and which files that they contain.

For now it is only building on "amd64" and failing or not installable due to missing build dependencies on all other architectures. It will take some time for me to work through the various build logs and see what is wrong. https://buildd.debian.org/status/package.php?p=swiftlang&suite=experimental

Next steps of course include updating the package to 5.9.1. I've not been following along so I don't know yet if Swift 5.9.x requires swift to build or not. Hopefully I can get this out in time for it to get pulled into Ubuntu 24.04 LTS.

[stevapple]

My Swift 5.6.3 package was approved by the Debian FTP team and is now in experimental.

🎉🎉🎉

I've not been following along so I don't know yet if Swift 5.9.x requires swift to build or not.

It does if you’re going to enable macro/regex literal/… and other related features.

Since these are becoming mandatory dating back to Swift 5.7, things are actually going to be tricky if the Debian team don’t allow some “temporary/modified version” to be used for building the release package. Because Swift in Swift supports only the last minor release, you’ll at least have to:

1. Use the existing Swift 5.6.3 toolchain to build Swift 5.7.3;
2. Use the Swift 5.7.3 toolchain to build Swift 5.8.1;
3. Use the Swift 5.8.1 toolchain to build Swift 5.9.1.

Hopefully I can get this out in time for it to get pulled into Ubuntu 24.04 LTS.

You can always ask for assistance if you wish:)

[melizasw]

Because Swift in Swift supports only the last minor release, you’ll at least have to:

1. Use the existing Swift 5.6.3 toolchain to build Swift 5.7.3;

2. Use the Swift 5.7.3 toolchain to build Swift 5.8.1;

3. Use the Swift 5.8.1 toolchain to build Swift 5.9.1.

Thanks for the roadmap. Once out of experimental and into unstable I can start working my way up the versions. My sponsor says that we can leave experimental whenever I feel the package is ready, and I think that will be after I fix the non-amd64 builds that should be working, especially arm64.

You can always ask for assistance if you wish:)

Ok. I've been working on the failures to build on other architectures. Some fail because Swift does not support them:

File "/<<PKGBUILDDIR>>/swift/utils/swift_build_support/swift_build_support/targets.py", line 348, in host_target
    raise NotImplementedError('System "%s" with architecture "%s" is not '
NotImplementedError: System "Linux" with architecture "armv8l" is not supported

And the others fail because -fstack-clash-protection is being called on an architecture in which Clang 13.0 does not support that option:

-- Build files have been written to: /<<PKGBUILDDIR>>/build/buildbot_linux/libdispatch-linux-aarch64
+ popd
/<<PKGBUILDDIR>>
<string>:1: DeprecationWarning: 'pipes' is deprecated and slated for removal in Python 3.13
+ /usr/bin/cmake --build /<<PKGBUILDDIR>>/build/buildbot_linux/libdispatch-linux-aarch64 -- -j4 all
[1/82][  1%][0.084s] Building C object src/CMakeFiles/dispatch.dir/apply.c.o
FAILED: src/CMakeFiles/dispatch.dir/apply.c.o 
/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-aarch64/bin/clang -DDISPATCH_USE_DTRACE=0 -DHAVE_CONFIG_H -D_GNU_SOURCE=1 -Ddispatch_EXPORTS -I/<<PKGBUILDDIR>>/build/buildbot_linux/libdispatch-linux-aarch64 -I/<<PKGBUILDDIR>>/swift-corelibs-libdispatch -I/<<PKGBUILDDIR>>/swift-corelibs-libdispatch/src -I/<<PKGBUILDDIR>>/build/buildbot_linux/libdispatch-linux-aarch64/src -I/<<PKGBUILDDIR>>/swift-corelibs-libdispatch/private -I/<<PKGBUILDDIR>>/swift-corelibs-libdispatch/src/BlocksRuntime -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -mbranch-protection=standard -O3 -DNDEBUG -std=gnu11 -fPIC -fvisibility=hidden -Werror -Wall -Wextra -Warray-bounds-pointer-arithmetic -Wassign-enum -Watomic-properties -Wcomma -Wconditional-uninitialized -Wconversion -Wcovered-switch-default -Wdate-time -Wdeprecated -Wdocumentation -Wdouble-promotion -Wduplicate-enum -Wexpansion-to-defined -Wfloat-equal -Widiomatic-parentheses -Winfinite-recursion -Wmissing-prototypes -Wnewline-eof -Wnullable-to-nonnull-conversion -Wobjc-interface-ivars -Wover-aligned -Wpacked -Wpointer-arith -Wselector -Wshadow -Wshorten-64-to-32 -Wsign-conversion -Wstatic-in-inline -Wsuper-class-method-mismatch -Wswitch -Wunguarded-availability -Wunreachable-code -Wunused -Wno-unknown-warning-option -Wno-trigraphs -Wno-four-char-constants -Wno-disabled-macro-expansion -Wno-pedantic -Wno-bad-function-cast -Wno-c++-compat -Wno-c++98-compat -Wno-c++98-compat-pedantic -Wno-cast-align -Wno-cast-qual -Wno-documentation-unknown-command -Wno-format-nonliteral -Wno-missing-variable-declarations -Wno-old-style-cast -Wno-padded -Wno-reserved-id-macro -Wno-shift-sign-overflow -Wno-undef -Wno-unreachable-code-aggressive -Wno-unused-macros -Wno-used-but-marked-unused -Wno-void-pointer-to-int-cast -Wno-vla -Wno-error=assign-enum -fno-exceptions -fblocks -MD -MT src/CMakeFiles/dispatch.dir/apply.c.o -MF src/CMakeFiles/dispatch.dir/apply.c.o.d -o src/CMakeFiles/dispatch.dir/apply.c.o -c /<<PKGBUILDDIR>>/swift-corelibs-libdispatch/src/apply.c
clang-13: error: argument unused during compilation: '-fstack-clash-protection' [-Werror,-Wunused-command-line-argument]

It seems to be happening when building libdispatch. The compile is being called with -fstack-clash-protection for some reason. I am still working to figure out where this is coming from so that I can address it. Any help on this would be appreciated.

[stevapple]

Some fail because Swift does not support them:

File "/<<PKGBUILDDIR>>/swift/utils/swift_build_support/swift_build_support/targets.py", line 348, in host_target
    raise NotImplementedError('System "%s" with architecture "%s" is not '
NotImplementedError: System "Linux" with architecture "armv8l" is not supported

For this specific case, ARMv8L is not likely to be supported, because Swift doesn't support 32-bit host. I believe it's supported as a target (i.e. for libswiftlang).

[melizasw]

Swift 5.6.3 list the following valid hosts for Linux:

    Linux = Platform("linux", archs=[
        "x86_64",
        "i686",
        "armv6",
        "armv7",
        "aarch64",
        "powerpc64",
        "powerpc64le",
        "s390x"])

This would seem to mean that i686, armv6, and armv7 are valid 32-bit hosts. However, the only hosts from this list that Debian supports are x86_64, i686, aarch64, powerpc64, and s390x (powerpc and riscv64 as well). So I think I can just restrict my package to building on these 5 architectures.

x86_64 builds
i686 fails because it needs libatomic
aarch64 fails because its build is using the unsupported -fstack-clash protection
powerpc64 is also missing libatomic as well as whatever provides __gnu_f2h_ieee, __gnu_h2f_ieee, and __truncdfhf2
s390x is failing because llvm-project/lld/ELF/Target.cpp doesn't see it as a valid architecture

I'm working to fix these, but any help would be appreciated.

Also, I'm curious how to add support for powerpc and riscv64 since they were added in 5.7.x or newer and thus there is no swift compiler on those platforms to use to compile.

[stevapple]

Swift 5.6.3 list the following valid hosts for Linux:

    Linux = Platform("linux", archs=[
        "x86_64",
        "i686",
        "armv6",
        "armv7",
        "aarch64",
        "powerpc64",
        "powerpc64le",
        "s390x"])

This would seem to mean that i686, armv6, and armv7 are valid 32-bit hosts.

IIRC 32-bit host support was dropped long before (or hadn’t ever existed). Latest PRs that add new architectures show they’re adding support for targets that are meant to be cross-compiled against. 32-bit ARMv8 host triples should be handled similarly, while 64-bit ones can be mapped into aarch64.

s390x is failing because llvm-project/lld/ELF/Target.cpp doesn't see it as a valid architecture

Maybe the port is not maintained any more and should be removed?

Also, I'm curious how to add support for powerpc and riscv64 since they were added in 5.7.x or newer and thus there is no swift compiler on those platforms to use to compile.

powerpc is target-only IIRC. riscv64 and other newly added architectures should be cross-compiled first. Not sure if the Debian packaging team will allow though. BTW this port doesn’t seem to be complete, so you can skip for now.

[stevapple]

Since these are becoming mandatory dating back to Swift 5.7, things are actually going to be tricky if the Debian team don’t allow some “temporary/modified version” to be used for building the release package.

@melizasw In fact, you can tweak the process to first build a toolchain without Swift in Swift support, and then build another one with it. This still works as for Swift 5.9, but will be removed at some time.

[melizasw]

Ok, I think the problem is that I was approaching the Swift build process with the notion that it supported a wider range of architectures than it does. I see now that tachoknight's packaged for Fedora only supports x86_64 and aarch64. For an optional development tool it seems reasonable to have a smaller set of supported architectures to start with.

Debian does indeed support the addition of new architectures and I was able to find some documentation on bootstrapping and cross-building which may be needed for riscv64 support. But for now I think I'm going to just focus on x86_64 and aarch64 and maybe s390x (Debian's LLVM toolchain builds for s390x).

Thanks for the help.

[tachoknight]

Ok, I think the problem is that I was approaching the Swift build process with the notion that it supported a wider range of architectures than it does. I see now that tachoknight's packaged for Fedora only supports x86_64 and aarch64. For an optional development tool it seems reasonable to have a smaller set of supported architectures to start with.

Fedora supports multiple architectures as well and for a very brief time I had everything working on the x390 platform, but then a newer version of clang would crash with a stacktrace when building Swift. I filed a bug with the LLVM team that has gotten no traction at all, so I abandoned it.

FYI I would like to know your experience building 5.9 on Debian/aarch64 because currently Fedora doesn't build successfully on that platform (builds and works fine on x86_64).

[melizasw]

Fedora supports multiple architectures as well and for a very brief time I had everything working on the x390 platform, but then a newer version of clang would crash with a stacktrace when building Swift. I filed a bug with the LLVM team that has gotten no traction at all, so I abandoned it.

Yep, I ran into the same problem and added to your bug report from 3+ years ago. There was some activity on the bug 2 months ago, but no fix yet. I guess no s390x for now.

FYI I would like to know your experience building 5.9 on Debian/aarch64 because currently Fedora doesn't build successfully on that platform (builds and works fine on x86_64).

It'll take a while before I get there, but I will let you know.

On a RPI 4, it takes around 24 hours. 🙃

That beats the 2.5 days it took to build through QEMU! I ordered an RPI 5 that will be here in a few days to help out with that problem. I've also got a LicheePi 4A arriving later today to help me with building for riscv64 which I have been making some progress on by porting patches from @futurejones to 5.6.3.

[melizasw]

Great news - Swift is now on Debian unstable (sid) and in a few days should go out to testing (trixie) on amd64 and arm64.
https://tracker.debian.org/pkg/swiftlang

Unfortunately when Ubuntu pulled it in and attempted to build on Noble it failed.
https://launchpad.net/ubuntu/+source/swiftlang

The issue appears to me to be related to the gold linker

[4402/4423][ 99%][4529.955s] Building CXX object utils/unittest/CMakeFiles/gtest.dir/googletest/src/gtest-all.cc.o
[4403/4423][ 99%][4529.991s] Linking CXX static library lib/libgtest.a
[4404/4423][ 99%][4530.020s] Linking CXX static library lib/libLLVMTestingSupport.a
[4405/4423][ 99%][4530.061s] Creating directories for 'compiler-rt'
[4406/4423][ 99%][4530.071s] No download step for 'compiler-rt'
[4407/4423][ 99%][4530.082s] No update step for 'compiler-rt'
[4408/4423][ 99%][4530.084s] Building CXX object utils/benchmark/src/CMakeFiles/benchmark.dir/benchmark_register.cc.o
[4409/4423][ 99%][4530.094s] No patch step for 'compiler-rt'
[4409/4423][ 99%][4530.094s] Performing configure step for 'compiler-rt'
-- The C compiler identification is Clang 13.0.0
-- The CXX compiler identification is Clang 13.0.0
-- The ASM compiler identification is Clang with GNU-like command-line
-- Found assembler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang
-- Check for working C compiler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang - broken
CMake Error at /usr/share/cmake-3.27/Modules/CMakeTestCCompiler.cmake:67 (message):
  The C compiler

    "/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: '/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/CMakeScratch/TryCompile-kQ8HgL'
    
    Run Build Command(s): /usr/bin/ninja -v cmTC_d2d03
    [1/2][ 50%][0.017s] /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang   -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/<<PKGBUILDDIR>>=/usr/src/swiftlang-5.6.3-2 -MD -MT CMakeFiles/cmTC_d2d03.dir/testCCompiler.c.o -MF CMakeFiles/cmTC_d2d03.dir/testCCompiler.c.o.d -o CMakeFiles/cmTC_d2d03.dir/testCCompiler.c.o -c /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/CMakeScratch/TryCompile-kQ8HgL/testCCompiler.c
    clang-13: warning: optimization flag '-ffat-lto-objects' is not supported [-Wignored-optimization-argument]
    [2/2][100%][0.030s] : && /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/<<PKGBUILDDIR>>=/usr/src/swiftlang-5.6.3-2 -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro CMakeFiles/cmTC_d2d03.dir/testCCompiler.c.o -o cmTC_d2d03   && :
    FAILED: cmTC_d2d03 
    : && /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/<<PKGBUILDDIR>>=/usr/src/swiftlang-5.6.3-2 -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro CMakeFiles/cmTC_d2d03.dir/testCCompiler.c.o -o cmTC_d2d03   && :
    /usr/bin/ld: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/bin/../lib/LLVMgold.so: error loading plugin: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/bin/../lib/LLVMgold.so: cannot open shared object file: No such file or directory
    clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
    ninja: build stopped: subcommand failed.
    
    

  

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:10 (project)


-- Configuring incomplete, errors occurred!
[4415/4423][ 99%][4530.818s] Building CXX object utils/benchmark/src/CMakeFiles/benchmark.dir/timers.cc.o
FAILED: tools/clang/runtime/compiler-rt-stamps/compiler-rt-configure /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-stamps/compiler-rt-configure 
cd /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins && /usr/bin/cmake -DCMAKE_C_COMPILER=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang -DCMAKE_CXX_COMPILER=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang++ -DCMAKE_ASM_COMPILER=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/bin/ninja -DLLVM_CONFIG_PATH=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/llvm-config "-DLLVM_LIT_ARGS=-v --time-tests -j 4" -DCOMPILER_RT_OUTPUT_DIR=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./lib/clang/13.0.0 -DCOMPILER_RT_EXEC_OUTPUT_DIR=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin -DCOMPILER_RT_INSTALL_PATH:PATH=lib/clang/13.0.0 -DCOMPILER_RT_INCLUDE_TESTS=ON -DCMAKE_INSTALL_PREFIX=/usr -DLLVM_LIBDIR_SUFFIX= -DLLVM_RUNTIME_OUTPUT_INTDIR=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin -DCMAKE_OSX_DEPLOYMENT_TARGET= -DCMAKE_OSX_SYSROOT:PATH= -DCOMPILER_RT_INTERCEPT_LIBDISPATCH=ON -DCOMPILER_RT_PREFIX=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/projects/compiler-rt -DCOMPILER_RT_SRC_ROOT=/<<PKGBUILDDIR>>/llvm-project/llvm/../compiler-rt -GNinja -S /<<PKGBUILDDIR>>/llvm-project/llvm/../compiler-rt -B /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins && /usr/bin/cmake -E touch /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-stamps/compiler-rt-configure
[4417/4423][ 99%][4531.167s] Building CXX object utils/benchmark/src/CMakeFiles/benchmark.dir/sysinfo.cc.o
ninja: build stopped: subcommand failed.
-- Warning: {} Host toolchain could not locate a compiler to build swift-driver. (Try `--skip-early-swift-driver`)
--- Building cmark ---
Building the standard library for: swift-stdlib-linux-x86_64
Running Swift tests for: check-swift-only_early_swiftdriver-linux-x86_64 check-swift-all-linux-x86_64 check-swift-all-optimize-linux-x86_64
ERROR: command terminated with a non-zero exit status 1, aborting

The same build on Debian produces the following build log:

[4405/4423][ 99%][2053.044s] Building CXX object utils/unittest/CMakeFiles/gtest.dir/googletest/src/gtest-all.cc.o
[4406/4423][ 99%][2053.094s] Linking CXX static library lib/libgtest.a
[4407/4423][ 99%][2053.137s] Linking CXX static library lib/libgtest_main.a
[4408/4423][ 99%][2053.138s] Linking CXX static library lib/libLLVMTestingSupport.a
[4409/4423][ 99%][2053.157s] Creating directories for 'compiler-rt'
[4410/4423][ 99%][2053.175s] No download step for 'compiler-rt'
[4411/4423][ 99%][2053.193s] No update step for 'compiler-rt'
[4412/4423][ 99%][2053.212s] No patch step for 'compiler-rt'
[4412/4423][ 99%][2053.212s] Performing configure step for 'compiler-rt'
-- The C compiler identification is Clang 13.0.0
-- The CXX compiler identification is Clang 13.0.0
-- The ASM compiler identification is Clang with GNU-like command-line
-- Found assembler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for unwind.h
-- Looking for unwind.h - found
-- Looking for rpc/xdr.h
-- Looking for rpc/xdr.h - not found
CMake Warning at cmake/Modules/CompilerRTUtils.cmake:352 (message):
  llvm-config finding testingsupport failed with status 1
Call Stack (most recent call first):
  CMakeLists.txt:81 (load_llvm_config)

I'm not sure why Debian is skipping the check but Ubuntu is not and then failing it. My Debian build doesn't have LLVMgold.so so no surprise that Ubuntu doesn't have it either.

I'm going to try the following to see if they help:

1. Add binutils-dev to the build dependencies
2. patch llvm-project-orig/llvm/cmake/config-ix.cmake as @tachoknight did for his Fedora 5.6.3 build to set LLVM_BINUTILS_INCDIR to /usr/include (Debian LLVM 16 also sets this path for its build).

Any help or advice would be appreciated as at the moment I am not set up to run test builds on Ubuntu and cannot directly test these without first releasing them to Debian then waiting for Ubuntu to pull it in and build.

[melizasw]

More good news. Swift has now migrated to Debian testing and the two fixes mentioned above did indeed get the Ubuntu Noble build past that hang-up. I drug an old system out of retirement and installed Ubuntu on it so that I could work through the build errors on Noble.

Unfortunately now I'm getting another error, something about trying to link a shared library and needed to compile with -fPIC.

[770/863][ 89%][27.919s] Linking CXX shared library swiftlang-5.6.3/build/buildbot_linux/llvm-linux-x86_64/lib/clang/13.0.0/lib/linux/libclang_rt.scudo_standalone-x86_64.so
FAILED: swiftlang-5.6.3/build/buildbot_linux/llvm-linux-x86_64/lib/clang/13.0.0/lib/linux/libclang_rt.scudo_standalone-x86_64.so 

/usr/bin/ld: /tmp/lto-llvm-bde8d8.o: warning: relocation against `__gwp_asan_default_options' in read-only section `.text._ZN8gwp_asan7options11initOptionsEPKcPFvS2_zE'
/usr/bin/ld: /tmp/lto-llvm-bde8d8.o: relocation R_X86_64_PC32 against symbol `stderr@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)

That doesn't seem too hard, I just need to find the right place to do so when I get more time.

[futurejones]

@melizasw I have installed and tried to test your debian packages but they don't work.
I am just trying to run a simple "hello world" using the swift package manager.
The errors are very easy to replicate.

The test environment is a Debian Trixie docker container.

docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -it debian:trixie /bin/bash

apt update

apt install swiftlang

mkdir hello && cd hello && swift package init --type executable && swift run

root@d5e164273bb5:/# mkdir hello && cd hello && swift package init --type executable && swift run
Creating executable package: hello
Creating Package.swift
Creating README.md
Creating .gitignore
Creating Sources/
Creating Sources/hello/main.swift
Creating Tests/
Creating Tests/helloTests/
Creating Tests/helloTests/helloTests.swift
'hello': error: invalidManifestFormat("<module-includes>:1:10: note: in file included from <module-includes>:1:\n#include \"CoreFoundation.h\"\n        
 ^\n/usr/libexec/swift/lib/swift/CoreFoundation/CoreFoundation.h:25:10: error: \'sys/types.h\' file not found\n#include <sys/types.h>\n         
^\n/usr/libexec/swift/lib/swift/CoreFoundation/CFStream.h:20:10: note: while building module \'CDispatch\' imported from /usr/libexec/swift/lib/swift/CoreFoundation/CFStream.h:20:\n#include <dispatch/dispatch.h>\n         
^\n<module-includes>:1:10: note: in file included from <module-includes>:1:\n#include \"dispatch.h\"\n         ^\n/usr/libexec/swift/lib/swift/dispatch/dispatch.h:32:10: note: in file included from 
/usr/libexec/swift/lib/swift/dispatch/dispatch.h:32:\n#include <os/generic_unix_base.h>\n  
^\n/usr/libexec/swift/lib/swift/os/generic_unix_base.h:24:10: 
error: \'sys/param.h\' file not found\n#include <sys/param.h>\n         
^\n<unknown>:0: error: could not build C module \'CoreFoundation\'", diagnosticFile: nil)
root@d5e164273bb5:/hello# 

This error also occurs when using Debian Unstable.

When trying to build using swiftc we get an another error.

root@d5e164273bb5:/hello# swiftc Sources/hello/main.swift 
error: link command failed with exit code 1 (use -v to see invocation)
clang-13: error: invalid linker name in argument '-fuse-ld=gold'

This indicates that binutils is missing from the required dependancies.
But after installing binutils we get another error.

root@d5e164273bb5:/hello# swiftc Sources/hello/main.swift 
error: link command failed with exit code 1 (use -v to see invocation)
/usr/bin/ld.gold: error: cannot open Scrt1.o: No such file or directory
/usr/bin/ld.gold: error: cannot open crti.o: No such file or directory
/usr/bin/ld.gold: error: cannot open crtbeginS.o: No such file or directory
/usr/bin/ld.gold: error: cannot open crtendS.o: No such file or directory
/usr/bin/ld.gold: error: cannot open crtn.o: No such file or directory
/usr/bin/ld.gold: error: cannot find -lgcc
/usr/bin/ld.gold: error: cannot find -lgcc_s
/usr/bin/ld.gold: error: cannot find -lc
/usr/bin/ld.gold: error: cannot find -lgcc
/usr/bin/ld.gold: error: cannot find -lgcc_s
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)

I have filed a bug with Debian to track the issue.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1057117

[melizasw]

@futurejones this is due to missing dependencies on binutils and gcc. If you install these two packages it will get you up and running for the next few days until I can get a new release out.

[futurejones]

@melizasw there appears to be a problem with the relationship between the different swiftlang packages.

The swiftlang package is listing the swiftlang-dev package as a dependency, whereas the swiftlang-dev package only lists swiftlang as recommends`.

The swiftlang-dev package should list both swiftlang and libswiftlang as dependencies.
https://packages.debian.org/trixie/swiftlang-dev

The swiftlang package should not list swiftlang-dev as a dependency.
https://packages.debian.org/trixie/swiftlang

[melizasw]

@futurejones I can certainly do that. Can you help me understand though what use swiftlang is without swiftlang-dev? I was under the impression that swiftlang wasn't of much use without both libswiflang and swiftland-dev.

[etcwilde]

So first, I'm really happy to see this work making progress. I think this will really help with the Swift-on-Linux story.

For some ideas on the separation of the libraries and tools, it might be worth looking at the Windows installer. I don't remember the exact split but it does a good job of separating concerns for developers, CI, and end-consumers of Swift programs. You shouldn't need to download the compiler if you just want to run a pre-compiled program that happens to be written in Swift, and you shouldn't need to download the headers for CoreFoundation if you just want to build a hello-world, for instance. It looks like that separation is mostly covered with the separation of swiftlang and libswiftlang, though perhaps could be split further for the stdlib and corelibs.

I may have missed a few runtime libraries in the listing here, so worth double-checking, but I think it's reasonable breakdown:

Runtimes:

• swift runtimes (both with and without swiftmodules/header files (dev and non-dev versions))
  • swiftCore
  • swiftGlibc
  • swiftRemoteMirror
  • swiftSwiftOnoneSupport
  • swift_Concurrency
  • swift_Differentiation
  • swift_MatchingEngine
  • swift_StringProcessing
  • libdispatch*
• corelibs
  • blocksRuntime
  • Foundation
  • FoundationNetworking
  • FoundationXML

* The challenge with libdispatch is that concurrency depends on it. The C bits of corelibs-libdispatch actually get built twice in the toolchain build, so I wonder if it would make sense to configure the build to emit a static archive so that we don't expose that. Then libdispatch.so could be installed as part of the corelibs package.

We should keep the runtime libraries together installed as one. I don't have a strong opinion on the corelibs libraries being together or apart, though mildly in favor of keeping them together as a swiftlang-corelibs package. Then we would want a dev variant of the swiftlang runtimes and swiftlang corelibs packages containing swiftmodules and headers. I think that keeping the corelibs separate from the runtimes is important, especially as work on swift-Foundation progresses, I'm not entirely sure what the future of corelibs-Foundation looks like.

Looking at swiftlang-dev, I'm concerned about the libcxx headers getting installed? If folks are doing C++ interop, which I don't think Swift 5.6 supports anyway, those headers will need to match the C++ runtime they're building for, which natively would be libstdc++ on the system.

Build Tools: (what I assume is the umbrella swiftlang package)

• swiftc
• swiftpm
• clang

The swift-clang will be an interesting tricky bit. We'll want to provide the special swift-clang that has support for swift and swift-async calling conventions so that folks can use the C and C++ interop features. That of course conflicts with the existing clang package. I think the provides mechanism might work here though? A swift-clang package provides clang perhaps?

My concern with naming clang swift-clang is that various build-scripts may try to use clang while consuming a header emitted by Swift. Those headers may contain functions with swift or swiftasync calling conventions, so if they use the wrong C/C++ compiler, they will get a warning about the unknown attribute, but the program will compile with a misaligned ABI. On a good day, that will crash their program. On a bad day, their program will look like it runs, but has totally random garbage floating around. I'm open to push-back on that.

swiftlang can probably drop the headers and swiftmodules, and instead install those in the runtime-dev package. It's true that the compiler isn't terribly useful without them, but it's more conventional to install the headers associated with the library in the <library>-dev package. See the gcc vs libc6 vs libc6-dev filelists, for example. gcc doesn't come with headers, but libc6-dev comes with the C headers.

Dev Tools:

• lldb
• sourcekit-lsp
• other tools and files useful for development but not strictly necessary for running or building a program

Same sort of situation with the swift-lldb as with clang. The lldb from apple/llvm-project has extra bits to make debugging Swift work, but that will conflict with the existing lldb package. Probably the same resolution since it provides lldb + swift extras.

Anyway, my initial thoughts. Thank you for working on this.