Various problems with building Swift packages

[philipturner]

When building Swift packages, there are multiple problems:

1. The formatting from build output could leak out and make a hard-coded message about Clang modules turn green.
2. There is an error of build.db missing when compiling a Swift package with the --verbose flag, at least with S4TF.
3. Two module.modulemap files that declare the same Clang module can overwrite each other, even if one is part of the documentation of a Swift package and never actually involved in the build process. This happened with the modulemap currently in the Utilities directory of s4tf/s4tf.
4. The headers referenced by a modulemap file are absent when you re-load the package. The -Xcc -I/... flags from when you built the package helped SwiftPM locate headers. Now that LLDB is loading the module with no knowledge of those manual includes, it doesn't know where the headers are.
5. [SwiftPM][Diagnostics] Missing or misplaced \u{1b}[2K sequence in build output apple/swift#58916 is still an issue and I don't have a workaround for it.

For some of these problems, a workaround may not be possible or feasible. Thus, I might have to make documentation warning the user to copy certain header files to the system include path. Or, I might have to warn against adding duplicate modulemap files to a GitHub repo and anything it recursively depends on.

[philipturner]

Problem 1: Solved

No longer present on the May 11, 2022 toolchain. Thus, it probably has the same source as (5). I added a partial patch for older toolchains where the text colorization bug was still present: append \u{1b}[0m to Stdout from spawned processes. This prevents incorrect formatting from leaking into the message about Clang modules.

Problem 2: Solved

The --verbose flag caused the output of swift build --show-bin-path to have multiple lines. This would make Swift-Colab treat the multi-line output as one massive directory path, resulting in wildly invalid URLs in several places. One victim was build.db, which threw an error. This bug was present in the old google/swift-jupyter and all versions of Swift-Colab until v2.2.

Patch: split the output of swift build --show-bin-path into multiple lines and only process the last line.

Problem 3: Temporary Workaround

Reported in apple/swift-package-manager#5565. Workaround: rename the modulemap in S4TF's Utilities folder to something besides module.modulemap.

Problem 4: Temporary Workaround

Proposed patch: scan the SwiftPM flags beginning with -I/, then find the full URL of every header found in the -I'd directories. For clarity, call these resolved headers "primary headers". Scan the primary headers for any #include <...>'s that reference "secondary headers", then resolve the secondary headers' full paths. Recursively repeat with tertiary headers, etc. and avoid circular references.

This solution is too complex to implement right now. Workaround: never include headers into SwiftPM via -Xcc -I/.... Instead, copy them to a system search path as shown below. This restriction should be noted in Swift-Colab's documentation.

%system cp -r "/content/Library/tensorflow-2.4.0/usr/include/tensorflow" "/usr/include/tensorflow"

Problem 5: Solved

Seems to be resolved on recent Swift toolchains. Check apple/swift#58916 for the latest information.

[philipturner]

I am going to declare problem 4 a permanent workaround. Scanning headers could become theoretically impossible if someone uses C macros to generate the include statements.

Problem 3 is not Swift-Colab's fault; it's a SwiftPM bug.

[philipturner]

Actually, there is a potential solution for problem 4 - I'm going to try overhauling the build system and how LLDB loads modules. I could store metadata with each Clang module that asks LLDB to look in those search paths before initializing. There's a one-Jupyter session delay between when the Clang module is generated and when it can be used. This provides time to ensure every Clang module gets its proper search paths stored.

How will I resolve those search paths? The new %install-swiftpm-environment magic could call cd and change the working directory. I will force the package to build in a pre-determined directory for unrelated reasons, but this means cd won't affect the resolved -I paths.

If problem 4 is solved, change the S4TF template notebook to stop copying tensorflow/c/... into a system search path. It can now achieve the same effect with SwiftPM build flags.

Relevant context: #22 (comment)