Introduce initial Swift support in fdbserver

[ktoso]

Hello everyone,

This is a cumulative draft pull request of an effort we had ongoing for a while now that enables the use of Swift to implement (or port) FDB functionality. The pull request specifically extended the cmake build to support Swift "side by side" existing fdb modules (flow, fdbrpc, fdbclient, fdbserver), integrated the Swift concurrency runtime into sim2 and net2 runtimes and proofs that this works by porting masterserver logic to Swift.

Swift

Swift brings to the table its strong typesafety and compiler verified concurrency model along with async-await, tasks and actors (and even distributed actors, though not shown in this PR yet) all built into the language.

Swift Concurrency and Flow

Swift comes with a fully fledged actor model runtime as well as distributed actors (which we're not making use of yet though in this PR).

Of course, it is important to keep the existing runtime characteristics of Flow and the sim2 and net2 eventloops. So we've ensured that all Swift concurrency code executes on the configured Flow evenloop.

Both simulation and production workloops are supported. In other words, creating a Task {}, child task or running Swift actors (which execute on Tasks), including hitting suspension points (await) all execute on foundationdb's "usual" scheduling infrastructure.

You can learn about the basics of Swift concurrency here: https://docs.swift.org/swift-book/documentation/the-swift-programming-language/concurrency/ and we're more than happy to provide additional guidance, docs, or just chat about the capabilities.

Swift / C++ Interoperability

To enable all of this we're using Swift's new C++ interopability feature in order to directly call into existing Flow/C++ code (no shims, or bindings!), and can support interop both ways: Flow/C++ calling Swift, or Swift using existing C++ types and functions. Swift is even able to ref-count existing C++ types (e.g. see MasterData).

You can learn more about the C++ interop here:

• more documentation about interoperability will be available here soon: [cxx-interop] add initial 'Mixing Swift and C++' docu… swiftlang/swift-org-website#299
  • meanwhile the SWIFT_GUIDE.md in this PR has already a bit more documentation and explains patterns we're using
• "forward" interop vision document - https://github.com/zoecarver/swift/blob/docs/interop-roadmap/docs/CppInteroperability/ForwardVision.md
• "reverse" interop vision document - https://github.com/apple/swift-evolution/blob/main/visions/using-swift-from-c%2B%2B.md

Status of the PR

The pull requested code is functional and we've been passing manually executed existing tests for a while now, but it is possible we missed some issues. For build instructions you can refer to the SWIFT_GUIDE.md. For setting up a developer environment including IDE with autocompletion across languages you can refer to the SWIFT_IDE_SETUP.md guide.

We'll continue working on more cleanups (e.g. re-enable some commented out code), improvements in quality of life and some more build adjustments in this pull request for the time being.

Our immediate next steps are that @ammolitor will prepare the necessary the integration testing jobs so we'll be able to verify this code does not regress accidentally. At the same time, we'll want to make sure building with Swift is not causing the current developer workflow any issues, so that's another focus area we'll look at.

Overall, the goal here is to allow for a non-disruptive introduction of the language, and allow development of new features in Swift. Right now there is a FLOW_WITH_SWIFT Knob that can be used to toggle between using Swift or C++/Flow implementations of masterserver (see masterserver.swift and masterserver.actor.cpp), so we can build a binary with "both versions" and gain confidence before turning on Swift implementations for good. The WITH_SWIFT build time option does not yet entirely enable/disable building Swift but the intent would be to do that in order to smooth out adoption.

Continuous Integration

This PR right now is expected to fail, and we're working with @ammolitor to prepare a suitable CI configuration.

Not featured in this PR yet

Swift also has macros and distributed actors which we intend to use to simplify and streamline some of the boilerplate involved in crossing the languages today. Macros specifically we'll be able to use to get rid of the manual wrappers Flow future using code into async functions in Swift. Though these are not included in this initial effort yet.

---

FYI: @mpilman @sfc-gh-jslocum @sfc-gh-ljoswiak @vishesh @ammolitor @jleach4 @xis19

From the Swift team @zoecarver, @hyp (for Swift/C++ topics), @etcwilde (with most build expertise) and myself (mostly concurrency, however feel free to ping me with any questions) are all available to answer questions or debug issues.

[vishesh]

@jzhou77 fixed the formatting in those cmake files.

[jzhou77]

I've looked through C++ changes. Mostly nits and trying to preserve the existing C++ behavior.

[jzhou77]

can you be more specific of "FIXME" actions?

[vishesh]

I'm not sure why it's here. I can check, but I think probably something to do with Swift importing this module creating issue.

[fdb-windows-ci]

Doxense CI Report for Windows 10

• Commit ID: b3e9a72
• Result: ❌ FAILED
• Build Logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux CentOS 7

• Commit ID: b3e9a72
• Duration 0:34:37
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: b3e9a72
• Duration 1:23:27
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: b3e9a72
• Duration 1:26:50
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux CentOS 7

• Commit ID: b3e9a72
• Duration 1:40:33
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)