Add a C++ Interoperability Vision Document

[zoecarver]

The markdown document can be viewed here: https://github.com/zoecarver/swift/blob/docs/interop-roadmap/docs/CppInteroperability/ForwardVision.md

[zoecarver]

Does it make sense to provide some examples of how we import various APIs? Maybe as an appendix at the end?

[compnerd]

Yes, I think that would be helpful to many people.

[mikepinkerton]

It would be very helpful for me, and would help solidify a lot of the vocabulary.

[plotfi]

I agree as well. I far too often have to help folks experimenting with C++-Interop to get swift-ide-test built just for them to see some examples like these actually.

[zoecarver]

different goals and constraints

Maybe expand on these constrains and goals: going from some specific Swift API to a broad language with less strong idioms. This actually gives a lot for flexibility for the interop model.

[compnerd]

I think it may be better to just a) describe what "reverse" interop is, and then link a placeholder document that @hyp can fill out.

[dabrahams]

I disagree that these things are very different. As soon as you can pass function pointers or closures across the language boundary, you effectively have the problems of bidirectional interop. Not to be alarmist, but tackling the two directions independently with the assumption that they are fundamentally different seems like a disaster waiting to happen.

[mikepinkerton]

This also speaks to the implicit/explicit conversion discussion we had today. If I pass (in a swift file) a swift string to a C++ API that expects a std::string, that C++ implementation is going to go off and make assumptions about how it can edit and access the data. So we have to think about how C++ would use Swift data types, not just how Swift would use C++ data types.

[plotfi]

For reverse interop I suggest eventually we link to the reverse interop document. Also, should the document explicitly call out that using Swift types from C++ code is a form of reverse interop and not the responsibility of forward interop?

[rjmccall]

You can be shorter and sweeter about this, something like: ""Reverse" interoperability, which allows Swift to be used from C++, is an interesting problem, but the technical and design problems it faces are largely independent from those of forward interoperability, and it will be addressed in a separate roadmap document."

It is not correct to link two clauses with ", however,"; you can use a semicolon (or a period) before "however", or you use ", but".

I wouldn't predict anything about needing edits to this document to explain how the halves fit together. If we need that, we can just make those edits to whichever document they're appropriate for.

[zoecarver]

These definitions feel verbose after reading the proceeding section. I think we can probably just delete the whole thing. Is there any useful information here?

[Huddie]

I personally appreciated it -- as someone with less understanding of these things

[compnerd]

Maybe we can move them to an appendix? Or these should be hoisted above the preceding sections to explain to the reader the terminology being used in the document.

[zoecarver]

Yeah, let's move them to the top of the document and define a few other things. Specifically for owned types, I want to remove the specifics about special members (that will be covered in detail below).

[beccadax]

The title “definitions” makes it sound like this is a collection of unrelated terminology, but I believe this is really the list of “common C++ API patterns” you were talking about before. If so, you may want to retitle this section.

I also think there’s a sort of hierarchy that you might want to represent more clearly here by reorganizing and referencing back to previous definitions:

• Reference types
• Owned types
  • Trivial types (a specific kind of owned type)
• Projections
  • Iterators (a specific kind of projection)

[dabrahams]

IMO these terms are not universally understood, cannot be assumed, and must be defined up-front.

[mikepinkerton]

I agree we should keep these. I still have a difficult time keeping them straight without some examples (even after reading this a couple times).

[plotfi]

I agree, these definitions should be kept in. It really helped my reading of the document.

[zoecarver]

Swift semantics require that copies can be made all over the place, imported types must fit with these semantics until Swift does not require that types are copied all over.

[ravikandhadai]

Would a type like the following be a owned type: (It owns some storage but not all)

struct S { 
   int &unownedReference;
   int ownedIntegerField; 
}

[ravikandhadai]

Guess it is a trivial type as per the below definition. So owned type is a super set of trivial types right or are we expecting an explicit copy constructor (directly or transitively)? It may be worth clarifying.

[beccadax]

I was surprised not to see “value semantics” in this definition, even in a qualified way. Owned types are often value types, right?

[dabrahams]

It's wrong to say that Swift semantics require copying all over the place. The rampant copying in Swift is almost entirely an implementation artifact, and this fact is easy to demonstrate. Nothing in the Swift documentation gives any assurance of observable differences in address between two objects. The only copies actually required by the language are those needed to make sure that mutations to one var/inout binding don't affect the values of any other bindings.

[dabrahams]

I'm confused again: the previous paragraph reads like it's talking about categories of Swift types, and this one reads like it's categorizing C++ types.

[ravikandhadai]

I was surprised not to see “value semantics” in this definition, even in a qualified way. Owned types are often value types, right?

@beccadax just to clarify, the notion of owned types here is only types that has some storage (because they could potentially expose internal pointers to it through some method). It doesn't necessarily has to have "value semantics" if you are thinking of types that behaves like conceptual values e.g. like String, COW types (assuming their type parameters have value semantics), functional data structures, and the like. E.g. the following would be "owned", and so is a smart pointer because it allocates and stores a ref count and copies that storage in copy constructors.

More semantically, the property that is required is whether a type has "some storage it copies" and can expose inner pointers to it. (It can totally have any others pointer, point to any storage and may have any side-effects.)

// This is an owned type as per the above definition.
struct S { 
   StringRef name;
   int &integerRefence
   vector<int> intVector;
   ...  
}

[zoecarver]

It may be helpful to call out that std::array is an example of a trivial types (whereas std::vector is an example of an owned type).

[Huddie]

+1 -- example would be helpful :)

[zoecarver]

Is it possible to have an owned type that holds a pointer and doesn't have a copy constructor?

[compnerd]

Hmm, if I understand correctly, I don't think that is possible. There would be a copy constructor, even if implicit (and fully inlined).

[zoecarver]

OK, we need some better terminology here. (I will include this in the "definitions" section.) What I meant to ask is: "Is it possible to have an owned type that holds a pointer and has a trivial (implicit, compiler generated) copy constructor?"

[zoecarver]

An oddly specific example, maybe this should instead discuss casting or representing class hierarchies.

[compnerd]

I agree that this is an oddly specific example and deserves some discussion. Am I correct in interpreting this as ABCs?

[zoecarver]

Yeah, exactly. I'm just giving this as an example of how we might evolve to support some specific API pattern. Maybe immortal, immutable owned types would be a better example (where we could allow projections).

[zoecarver]

This is not part of this document, just some things that I was thinking about but didn't know exactly where to put.

[compnerd]

Maybe move some of the suggested removals here, i.e. TODOs

[Huddie]

Yippee!! love it 🚀

One general comment:

Good compiler feedback

I may have missed it but having really good compiler error messages around interop seems like it should be a high priority.

One thing people love about swift is how understandable it is and as mentioned in the article, this is not the case with C++. So like if I have a C++ class and I think its trivial and its not imported into swift, it would be nice to have a clear message explaining that the type wasn't imported because XYZ, try ABC. Or maybe I though this was a value type that should be imported but really the methods im calling is a projection and we don't import those...

Maybe "high priority" isn't the right way to phrase it for a Roadmap. Id be curious to hear what role that plays in the Roadmap and what we are looking to support in terms of education of interop via the compiler

(I know @zoecarver has been adding these and I have hit these before and found them very helpful. I have also not hit them and wished they were there :))

[Huddie]

+1 -- example would be helpful :)

[Huddie]

Maybe there can be 2 broader sections.

Generally supported:

• Trivial types
• Reference types

Custom mappings:

• Iterators

In which case things would move form Custom mapping to "Generally Supported" as swift gets new features + we improve generally supported features to address issues that inhibit some item in Custom mapping from just being mapped

Something falls into Custom Mapping more because if we just generally imported it, it would be less safe in swift than C++ but we still want to support it, so we Custom map it

Not sure if that makes sense

[compnerd]

Yes, I think that would be helpful to many people.

[compnerd]

Suggested change

	* Iterators and ranges (by Egor)
	* Iterators and ranges

[zoecarver]

This and similar wont be in the final document.

[compnerd]

Suggested change

	* *The* Standard Library overlay (by Egor)
	* *The* Standard Library overlay

[compnerd]

I'm not sure I understand the emphasis on "This roadmap is".

[richardtop]

Agree, it's much "easier" to read without the emphasis, I also don't see a point in making it bold

[rjmccall]

I agree: if this going for some sort of drop-cap-like effect, it doesn't really work and would be out of place.

More importantly, though, this first sentence shouldn't be the first sentence. The way it's written draws attention to the statement this is a "sketch" and not a "final design"; that's good to include, but it should come later in the paragraph, after you've told us what the document is. "This document is a roadmap for the development of the "forward" half of C++ and Swift interoperability: using C++ APIs from Swift. It outlines..."

"some high-level topics" is vague and suggests you're eliding something that might be important. "This roadmap describes the current state of the art of using C++ from Swift", maybe?

The parallelism between these list items as things being outlined is weak, and it would be much better if you used a verb on each: "outlines some high-level topics", "lays out overarching goals", "defines a process", and "finally, investigates a collection of specific problems in importing C++ APIs and proposes solutions for them".

"Proposes solutions for them", if you choose to use it, would lead very nicely into a sentence about how this is not a "final design" and how everything in it will need to go through evolution.

[compnerd]

I think it may be better to just a) describe what "reverse" interop is, and then link a placeholder document that @hyp can fill out.

[compnerd]

Maybe move some of the suggested removals here, i.e. TODOs

[compnerd]

What makes mutex and files special? C has those as well.

[zoecarver]

I can just remove this. I was getting at another API pattern: types that use RAII to define a scope of use. But that should maybe be left for another document.

[compnerd]

Suggested change

	* Safety: because C++‘s un-oppionated APIs are funneled into Swift, they are generally forced through unsafe pointer which loses information. This loss of semantic information leads to mis-use, if more information was captured it would be more difficult, or even impossible, to make such mistakes.
	* Safety: because C++‘s un-opinionated APIs are funneled into Swift, they are generally forced through unsafe pointer which loses information. This loss of semantic information leads to misuse, if more information was captured it would be more difficult, or even impossible, to make such mistakes.

[compnerd]

Suggested change

	* Safety: the subtly different paradims for automatic lifetime management in Swift and C++ mean there is added unsafety to a language like c where lifetime is manually managed, or a language like objc where the lifetime matches swift’s paradims.
	* Safety: the subtly different paradigms for automatic lifetime management in Swift and C++ mean there is added unsafety to a language like C where lifetime is manually managed, or a language like ObjC where the lifetime matches Swift’s paradigms.

[compnerd]

Suggested change

	* Memory safety: lifetimes, reference counting, type safety: importing types as “Any” or “OpaquePointer” is not type safe. Also: use after free, bound checking, null pointer, etc.
	* Memory safety: lifetimes, reference counting, type safety: importing types as `Any` or `OpaquePointer` is not type safe. Also: use after free, bound checking, null pointer, etc.

[zoecarver]

Good compiler feedback

Good catch! This is an excellent observation and something that should 100% be called out in this document. I will add a section for that. Thanks @Huddie!

[AnnaZaks]

From this sentence it is not clear how much coverage of C++ types/APIs you expect from these categories. Not sure if you want to clarify that or not.

[dabrahams]

our goals, and C++ as a language, necessitate that not all APIs are imported

I don't know that this claim has been adequately supported by arguments. I understand that importing all APIs is a hard problem and we shouldn't expect it in the near term, but nothing I've read so far would lead me to the conclusion above.

[zoecarver]

Sure, I think I can update this. The point I'm trying to make here is that we want to start from a place of not importing all APIs. In this initial version of interop that is outlined by the roadmap, not all APIs will be imported. This is necessitated by the fact that we want specific mappings for API patterns, and the fact that not all C++ APIs fit into one of these patterns. Does that make sense?

Of course, I'm not opposed to eventually supporting all C++ API patterns. If someone wanted to add support for all of these, I don't see any reason we shouldn't. The document outlines exactly how this can be done over time in the evolution section.

[egorzhdan]

Iterator types must provide a comparison operator and increment operator

I think we should say "equality operator" instead of "comparison operator" and "pre-increment iterator" instead of "increment operator" to make it more precise.

But can we instead say that iterator types should conform to C++ named requirements for iterators without listing the specific requirements?

[dabrahams]

Suggested change

**C and Objective-C interoperability:** C++ interoperability will have to build off of the previous art of C and Objective-C interoperability. Starting with the name, C++ is a superset of C, which means C++ interoperability has a great starting place to build off of. C is a simple language without many high level constructs, so importing C APIs is, relatively, simple. On the other hand, Objective-C adds some high level constructs, such as reference types and a generics model. However, unlike C++, Objective-C has strong idioms; there are only so many patterns for doing some specific thing. Further, Objective-C was able to be updated to better interoperate with Swift, the generics model is a good example of this. So, while C and Objective-C interoperability is a good starting place, the model for importing these languages cannot be taken wholesale for C++ interoperability.

**C and Objective-C interoperability:** C++ interoperability will have to build on the previous art of C and Objective-C interoperability. Starting with the name, C++ is a superset of C, which means C++ interoperability has a great starting place. C is a simple language without many high level constructs, so importing C APIs is relatively simple. On the other hand, Objective-C adds some high level constructs, such as reference types and a generics model. However, unlike C++, Objective-C has strong idioms; there are only so many patterns for doing some specific thing. Further, Objective-C was able to be updated to better interoperate with Swift: the generics model is a good example. So, while C and Objective-C interoperability is a good starting place, the model for importing these languages cannot be taken wholesale for C++ interoperability.

[dabrahams]

This paragraph gives me the same sense, of putting off the real content.

[dabrahams]

Without some idea of which kinds of APIs you intend not to import, this statement is perhaps needlessly alarming. For example, I can live without the ability to use function-style macros from Swift, but I can't think of any other category of C++ APIs that obviously should be off the table, and you can make interop arbitrarily useless to me depending on where you draw that line.

Keep in mind that vector<bool> is a specialization, and if you're going to prevent me from using vector<T> in code that's generic on T, that would be pretty hard to swallow.

[mikepinkerton]

I disagree that it's needlessly alarming. If the goal of the first cut of this interop is to import only a manageable subset of C++, saying so in the goals section seems reasonable. It has to be said at some point, otherwise we're back to the previous (very challenging) plan of trying to import everything in a v1 implementation, which is what led us to here in the first place.

That said, I think concrete examples of what will and won't be imported will make this whole document easier to understand. That's a big part of what's lacking from all of this, so it still seems nebulous and scary and possibly what Dave is responding to.

[dabrahams]

You've set me up with the introduction that we're about to “discuss several of the most common C++ API patterns,” so when I get here I understand you to be categorizing C++ types. But the last sentence above leads to the opposite understanding.

[dabrahams]

reference types can be used to represent non-copyable types in C++

This idea conflicts with the idea that things should not become less safe when imported into Swift. In C++, non-copyable types are idiomatically used to enforce single-ownership semantics, on which the safety of other constructs may depend. Representing them directly as copyable references in Swift undermines that. One solution could be to import a non-copyable C++ T as a class Shared<T> in Swift, but the API necessary to make the indirection evident will be ugly until Swift itself gets new features… like at least partial direct support for non-copyable types.

[dabrahams]

In typical C++ fashion, there is no clear idiom for defining reference types.

This is not really a distinction between Swift and C++. I can create structs with reference semantics and classes with value semantics in Swift.

[dabrahams]

Also, Swift classes do not have the same semantics as C++ reference types. Typically when C++ is used with reference semantics, types are handled through explicit pointers, so shared_ptr<Foo> (or Foo*) is treated as a reference to Foo even if Foo itself would otherwise have value semantics. Idiomatic C++ actually gives you quite a bit more information than idiomatic Swift does in this respect.

[richardtop]

Pls explain how it's possible to create struct with reference semantics in Swift, as from my understanding, Swift's struct always "behaves" as if it was just a value type, i.e. it's copied when passed/dereferenced. Under the hood there might be some optimizations, such as copy-on-write in collections, for example, but at the surface level structs always behave like value types.

... unless there are mutating functions and/or inout parameters, but those apply to functions and not to the structs themselves. Or am I missing something else?

[zoecarver]

Hello all! Thank you for all of these extremely helpful comments :) I re-worked a lot of the document, and added quite a bit more to it. I have not addressed all of the comments above, but I am planning to spend most of tomorrow doing that (so don't be worried if your comment has not been addressed yet). I will let you know once I feel that I've addressed all the above comments, at that point, please feel raise the issue again if you feel it hasn't been addressed sufficiently.

Thanks again for all the feedback and thoughts!

[zoecarver]

It may be good to also discuss the possibility of having a "Swift reference type" or a foreign reference type that has Swift reference counting.

[rjmccall]

I would suggest starting with (trivial and non-trivial) value types, since they're the default patterns we're going to try to recognize.

You can explain the value/reference type distinction once and then assume it later; no need to repeat it in different sections.

[dabrahams]

I don't know how the Swift team is reacting to the Val object model description. If the feeling is that it's too distant and speculative to be relevant to this roadmap, I certainly understand that, but I thought it was worth pointing one small finger at 👉

[ephemer]

Thank you for the document. I didn't get through it all (and I understand I'm not necessarily its intended audience anyway) but I appreciate this being out in the open. It makes me realise how difficult this task really is.

I added some "editorial" comments. Hope you can feel my positive intention behind them :).

[zoecarver]

I added some "editorial" comments. Hope you can feel my positive intention behind them :).

Of course, thank you very much :)

[zoecarver]

Closing in favor of the Swift Evolution PR: swiftlang/swift-evolution#2057