diff --git a/proposals/0184-unsafe-pointers-add-missing.md b/proposals/0184-unsafe-pointers-add-missing.md index 60443cfe79..9b4e9fb0c7 100644 --- a/proposals/0184-unsafe-pointers-add-missing.md +++ b/proposals/0184-unsafe-pointers-add-missing.md @@ -1,76 +1,37 @@ # Unsafe[Mutable][Raw][Buffer]Pointer: add missing methods, adjust existing labels for clarity, and remove deallocation size -* Proposal: [SE-0184](0184-unsafe-pointers-add-missing.md) +* Proposal: [SE-0184](0184a-unsafe-pointers-part-1.md) * Author: [Kelvin Ma (“Taylor Swift”)](https://github.com/kelvin13) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Active review (September 1...7, 2017)** -* Implementation: [apple/swift#11464](https://github.com/apple/swift/pull/11464) +* Status: +* Implementation: [PR 12200](https://github.com/apple/swift/pull/12200) ## Introduction -Swift’s pointer types are an important interface for low-level memory manipulation, but the current API design is not very consistent, complete, or convenient. Many memory methods demand a `capacity:` or `count:` argument, forcing the user to manually track the size of the memory block, even though most of the time this is either unnecessary, or redundant as buffer pointers track this information natively. In some places, poor naming choices and overengineered function signatures compromise memory safety by leading users to believe that they have allocated or freed memory when in fact, they have not. +*This document is a spin-off from a much larger [original proposal](https://github.com/kelvin13/swift-evolution/blob/e888af466c9993de977f6999a131eadd33291b06/proposals/0184-unsafe-pointers-add-missing.md), which covers only those aspects of SE-1084 which do not deal with partial buffer memory state. Designing the partial buffer memory state API clearly requires more work, and has been left out of the scope of this document.* -This proposal seeks to improve the Swift pointer API by ironing out naming inconsistencies, adding sensible default argument values, adding missing methods, and reducing excessive verbosity, offering a more convenient, more sensible, and less bug-prone API. We also attempt to introduce a buffer pointer API that supports partial initialization without excessively compromising memory state safety. +Swift’s pointer types are an important interface for low-level memory manipulation, but the current API design is not very consistent, complete, or convenient. In some places, poor naming choices and overengineered function signatures compromise memory safety by leading users to believe that they have allocated or freed memory when in fact, they have not. This proposal seeks to improve the Swift pointer API by ironing out naming inconsistencies, adding missing methods, and reducing excessive verbosity, offering a more convenient, more sensible, and less bug-prone API. -Swift-evolution thread: [Pitch: Improved Swift pointers](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170710/038013.html), [Pitch: More Improved Swift pointers](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170717/038121.html) +Swift-evolution threads: [Pitch: Improved Swift pointers](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170710/038013.html), [Pitch: More Improved Swift pointers](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170717/038121.html) -Implementation branch: [**PR 11464**](https://github.com/apple/swift/pull/11464) +Implementation branch: [`kelvin13:se-0184a`](https://github.com/kelvin13/swift/tree/se-0184a) ## Background -There are four binary memorystate operations: *initialization*, *move-initialization*, *assignment*, and *move-assignment*. They can be grouped according to how they affect the source buffer and the destination buffer. **Copy** operations only read from the source buffer, leaving it unchanged. **Move** operations deinitialize the source memory, decrementing the reference count by 1 if the memory type is not a trivial type. **Retaining** operations initialize the destination memory, incrementing the reference count by 1 if applicable. **Releasing** operations deinitialize the destination memory before reinitializing it with the new values, resulting in a net change in the reference count of 0, if applicable. +There are four binary memorystate operations: *initialization*, *move-initialization*, *assignment*, and *move-assignment*, and two unary memorystate operations: *deinitialization* and *type rebinding*. The binary operations can be grouped according to how they affect the source buffer and the destination buffer. **Copy** operations only read from the source buffer, leaving it unchanged. **Move** operations deinitialize the source memory, decrementing the reference count by 1 if the memory type is not a trivial type. **Retaining** operations initialize the destination memory, incrementing the reference count by 1 if applicable. **Releasing** operations deinitialize the destination memory before reinitializing it with the new values, resulting in a net change in the reference count of 0, if applicable. | | Copy (+0) | Move (−1) | | -------------: |----------: | ---------: | | **Retaining (+1)** | initialize | move-initialize | | **Releasing (+0)** | assign | move-assign | -Note: deinitialization by itself is a unary operation; it decrements the reference count of the buffer by 1. +Raw pointers also have a unique operation, *bytewise-copying*, which we will lump together with the memorystate functions, but does not actually change a pointer’s memory state. -The four main types of Swift pointers we have currently support different subsets of this toolbox. - -### UnsafeMutablePointer - -| | Copy | Move | -| ------------- |----------: | ---------: | -| **Retaining** | `initialize(to:count:)`, `initialize(from:count:)` | `moveInitialize(from:count:)` | -| **Releasing** | `assign(from:count:)` | `moveAssign(from:count:)` | - -### UnsafeMutableRawPointer - -| | Copy | Move | -| ------------- |----------: | ---------: | -| **Retaining** | `initializeMemory(as:at:count:to:)`, `initializeMemory(as:from:count:)` | `moveInitializeMemory(as:from:count:)` | -| **Releasing** | | | - -### UnsafeMutableBufferPointer - -| | Copy | Move | -| ------------- |----------: | ---------: | -| **Retaining** | `initialize(from:)` | | -| **Releasing** | | | - - -### UnsafeMutableRawBufferPointer - -| | Copy | Move | -| ------------- |----------: | ---------: | -| **Retaining** | `initializeMemory(as:from:)` | | -| **Releasing** | | | - -There are unary memorystate operations such as *deinitialization* and *type rebinding*, which are not listed in the tables, but are still covered by this proposal. Raw pointers also have a unique operation, *bytewise-copying*, which we will lump together with the memorystate functions, but does not actually change a pointer’s memory state. +Most of these operations become more relevant in the discussion of partial buffer memory state, which is not in the scope of this document. This document only proposes changes related to memory allocation, type-rebinding, and two special *unary* forms of initialization and assignment which initialize memory to a fixed, repeating value. ## Motivation -Right now, `UnsafeMutableBufferPointer` is kind of a black box when it comes to producing and modifying instances of it. Much of the API present on `UnsafeMutablePointer` is absent on its buffer variant. To create, bind, allocate, initialize, deinitialize, and deallocate them, you have to extract `baseAddress`es and `count`s. This is unfortunate because `UnsafeMutableBufferPointer` provides a handy container for tracking the size of a memory buffer, but to actually make use of this information, the buffer pointer must be disassembled. In practice, this means the use of memory buffers requires frequent (and annoying) conversion back and forth between buffer pointers and base address–count pairs. For example, to move-initialize memory between two buffer pointers, you have to write this: - -```swift -buffer1.baseAddress?.moveInitialize(from: buffer2.baseAddress!, count: buffer1.count) -``` - -The `?` is sometimes exchanged with an `!` depending on the personality of the author, as normally, neither operator is meaningful here — the `baseAddress` is never `nil` if the buffer pointer was created around an instance of `UnsafeMutablePointer`. - -Memory buffer allocation is especially painful, since it requires the creation of a temporary `UnsafeMutablePointer` instance. This means that the following “idiom” is very common in Swift code: +Right now, `UnsafeMutableBufferPointer` is kind of a black box when it comes to producing and modifying instances of it. Much of the API present on `UnsafeMutablePointer` is absent on its buffer variant. To create, bind, allocate, initialize, and deallocate them, you have to extract `baseAddress`es and `count`s. This is unfortunate because `UnsafeMutableBufferPointer` provides a handy container for tracking the size of a memory buffer, but to actually make use of this information, the buffer pointer must be disassembled. In practice, this means the use of memory buffers requires frequent (and annoying) conversion back and forth between buffer pointers and base address–count pairs. For example, buffer allocation requires the creation of a temporary `UnsafeMutablePointer` instance. This means that the following “idiom” is very common in Swift code: ```swift let buffer = UnsafeMutableBufferPointer(start: UnsafeMutablePointer.allocate(capacity: byteCount), count: byteCount) @@ -94,6 +55,8 @@ defer } ``` +The `?` is sometimes exchanged with an `!` depending on the personality of the author, as normally, neither operator is meaningful here — the `baseAddress` is never `nil` if the buffer pointer was created around an instance of `UnsafeMutablePointer`. + This method is extremely problematic because nearly all users, on first seeing the signature of `deallocate(capacity:)`, will naturally conclude from the `capacity` label that `deallocate(capacity:)` is equivalent to some kind of `realloc()` that can only shrink the buffer. However this is not the actual behavior — `deallocate(capacity:)` actually *ignores* the `capacity` argument and just calls `free()` on `self`. The current API is not only awkward and suboptimal, it is *misleading*. You can write perfectly legal Swift code that shouldn’t segfault, but still can, for example ```swift @@ -116,18 +79,16 @@ defer which is functionally equivalent. However this will lead to disastrous source breakage if the implementation of `deallocate(capacity:)` is ever “corrected”. Since the API would not change, such code would still compile, but suddenly start failing at runtime. Thus, the current API, combined with incorrect documentation, is serving as a vector for introducing memory bugs into Swift code. -The Swift pointer API is incomplete in other ways too. For example, the `initialize(from:count:)` method on `UnsafeMutablePointer` has a repeated-value copy variant, `initialize(to:count:)`, but `assign(from:count:)` has no such variant, even though it would make just as much sense for it to have one. - Finally, some of the naming choices in the current API deserve a second look. While the original API intended to introduce a naming convention where `bytes` refers to uninitialized memory, `capacity` to uninitialized elements, and `count` to initialized elements, the actual usage of the three words does not always agree. In `copyBytes(from:count:)`, `count` refers to the number of *bytes*, which may or may not be initialized. Similarly, the `UnsafeMutableRawBufferPointer` `allocate(count:)` type method includes a `count` argument which actually refers to uninitialized bytes. The argument label `to:` is also excessively overloaded; sometimes it refers to a type `T.Type`, and sometimes it refers to a repeated value parameter. This becomes problematic when both parameters appear in the same method, as in `initializeMemory(as:at:count:to)`. ## Proposed solution -The goal of the API redesign is to bring all of the functionality in `UnsafeMutablePointer` and `UnsafeMutableRawPointer` to their buffer types, `UnsafeMutableBufferPointer` and `UnsafeMutableRawBufferPointer`. `UnsafeMutableRawBufferPointer` already contains some of this functionality, providing a useful blueprint for the proposed `UnsafeMutableBufferPointer` API. +The ultimate goal of the API redesign is to bring all of the functionality in `UnsafeMutablePointer` and `UnsafeMutableRawPointer` to their buffer types, `UnsafeMutableBufferPointer` and `UnsafeMutableRawBufferPointer`. Operations which are covered by this proposal are in **bold**. The full toolbox of methods that we could possibly support includes: - - allocation - - deallocation + - **allocation** + - **deallocation** - initialization - move-initialization @@ -136,64 +97,17 @@ The full toolbox of methods that we could possibly support includes: - move-assignment - deinitialization - - type rebinding + - **type rebinding** - bytewise copying Because copy operations (initialization and assignment) don’t mutate the source argument, they can also come in a form which takes a repeated-value source instead of a buffer source. - - initialization (repeated-value) - - assignment (repeated-value) + - **initialization (repeated-value)** + - **assignment (repeated-value)** `UnsafeMutablePointer` and `UnsafeMutableRawPointer` already contain repeated-value methods for initialization in the form of `initialize(to:count:)` and `initializeMemory(as:at:count:to:)`. This proposal will add the assignment analogues. For reasons explained later, the argument label for the repeated-value parameter will be referred to as `repeating:`, not `to:`. -In their most general form, these functions are written like this: - -``` -static -func allocate(as:count:) -> PointerType -func deallocate() - -func initialize(as:at:repeating:count:) -func initialize(as:at:from:count:) -func moveInitialize(as:at:from:count:) - -func assign(as:at:repeating:count:) -func assign(as:at:from:count:) -func moveAssign(as:at:from:count:) - -func deinitialize(as:at:count:) -func rebindMemory(as:count:) - -func copyBytes(at:from:count:) -``` - -where - - - **`as:`** refers to the element type - - **`at:`** refers to an offset from `self`, in strides of the element type - - **`repeating:`** refers to a repeating value - - **`from:`** refers to a second pointer which serves as the **source** - - **`count:`** refers the number of elements the operation operates on - -On actual pointer types, most of these parameters are unnecessary, and some of the methods themselves either don’t make sense to support, or are not practically useful. - - - it only makes sense for immutable pointer types to support deallocation and type rebinding. Note that Swift’s memory model does not require memory to be mutable for deallocation. - - - raw (untyped) pointers should not support any operations which involve deinitialization on `self`. This rules out deinitialization itself, as well as any releasing operations (assignment, move-assignment). - - - typed pointers don’t need an `as:` parameter (except for type rebinding) — they already have a type. It also doesn’t make sense for them to support byte-wise copying. - - - pointers for which it is syntactically easy to offset in strides (for example, by pointer arithmetic with `+`), don’t need to take an `at:` argument. - -> note: some of these conceptual argument labels have different names in the real API. `as:` is written as `to:` in the type-rebinding methods because it sounds better. `count:` is sometimes written as `capacity:` or `bytes:` to express the assumptions about the stride and initialization state of the memory in question. - -> * `bytes` refers to, well, a byte quantity that is *not assumed* to be initialized. -> * `capacity` refers to a strided quantity that is *not assumed* to be initialized. -> * `count` refers to a strided quantity that is *assumed* to be initialized. - -> note: we don’t bother supporting an `at:` offset in type rebinding operations since we don’t anticipate much use for such a feature. - ### `UnsafePointer` ``` @@ -201,7 +115,9 @@ func deallocate() func withMemoryRebound(to:capacity:_:) -> Result ``` -`UnsafePointer` does not get an allocator static method, since you almost always want a mutable pointer to newly allocated memory. Its type rebinding method is also written as a decorator, taking a trailing closure, for memory safety. `UnsafePointer` does not take `at:` arguments since `+` provides pointer arithmetic for it. +`UnsafePointer` does not get an allocator static method, since you almost always want a mutable pointer to newly allocated memory. Its type rebinding method is also written as a decorator, taking a trailing closure, for memory safety. + +Most immutable pointer types currently do not have a deallocation method. This proposal adds them, fixing [SR-3309](https://bugs.swift.org/browse/SR-3309). Note, immutable raw buffer pointers already support this API. ### `UnsafeMutablePointer` @@ -211,22 +127,27 @@ func allocate(capacity:) -> UnsafeMutablePointer func deallocate() func initialize(repeating:count:) -func initialize(from:count:) -func moveInitialize(from:count:) +func initialize(to:) func assign(repeating:count:) -func assign(from:count:) -func moveAssign(from:count:) -func deinitialize(count:) func withMemoryRebound(to:capacity:_:) -> Result ``` -Like `UnsafePointer`, `UnsafeMutablePointer`’s type rebinding method is written as a decorator, and its methods do not need `at:` arguments. +Like `UnsafePointer`, `UnsafeMutablePointer`’s type rebinding method is written as a decorator. -> note: the `count:` argument in `initialize(repeating:count:)`, `assign(repeating:count:)`, and `deinitialize(count:)` and the `capacity:` argument in `allocate(capacity:)` should receive a default value of `1`, since those methods are commonly used with a count of `1`. +Previously, the single-element repeated-initialization case was supported by a default argument of `1` on `initialize(repeating:count:)`’s `count:` parameter, but it was decided this was too confusing in terms of API readability. For example, calls to `initialize(repeating:count:)` and its corresponding method on `UnsafeMutableBufferPointer` were prone to look the same. -> `withMemoryRebound(to:capacity:_:)` should *not* receive a default `count:` value, to avoid misleading users about type binding in Swift’s memory model. (The entire memory block must be rebound, not just the first element under the pointer.) +```swift +plainPointer.initialize(repeating: pointee) +bufferPointer.initialize(repeating: repeatedValue) +``` + +Increasing API surface by adding this method is justified by the large number of calls to `initialize(to:count:)` in the standard library (and likely other code) which rely on the default argument of `1`. We do *not* need to add a corresponding `assignPointee(to:)` method since this can be done with the assignment operator. + +```swift +ptr.pointee = newValue +``` ### `UnsafeRawPointer` @@ -240,66 +161,17 @@ func bindMemory(to:capacity:) -> UnsafePointer ``` static -func allocate(bytes:alignedTo:) -> UnsafeMutableRawPointer +func allocate(byteCount:alignment:) -> UnsafeMutableRawPointer func deallocate() -func initializeMemory(as:at:repeating:count:) -> UnsafeMutablePointer -func initializeMemory(as:at:from:count:) -> UnsafeMutablePointer -func moveInitializeMemory(as:at:from:count:) -> UnsafeMutablePointer +func initializeMemory(as:repeating:count:) -> UnsafeMutablePointer func bindMemory(to:capacity:) -> UnsafeMutablePointer - -func copy(from:bytes:) -``` - -The `as:` argument in `allocate(bytes:alignedTo:)` is represented by an alignment parameter which takes an integer. This is more useful since we often need a computed alignment (like when aligning a structure) instead of a preset type alignment. - -> note: although raw pointers support pointer arithmetic, it is not done in strides, and `ptr + offset * MemoryLayout.stride` is painfully verbose, so these methods take an `at:` offset. - -> note: the `at:` arguments in `UnsafeMutableRawPointer` should all have a default value of `0`; i.e., no offset. - -------------- - -Buffer pointers are conceptually similar, except the `count:` argument is often unnecessary since they track their own length internally. This means you would call - -```swift -ptr1.initialize(repeating: value, count: count) ``` -on an `UnsafeMutablePointer`, but - -```swift -buffer1.initialize(repeating: value) -``` - -on an `UnsafeMutableBufferPointer`. - -Implementing unary operations like repeated-value initialization, repeated-value assignment, and type rebinding is straightforward. However, with binary operations like move-initialization, which involves both a source buffer and a destination buffer, the question of whose `count` to use becomes important. - -One option is to use the destination’s `count`, and set the precondition that source`.count` `>=` destination`.count`. The benefit to this is that the destination is always guaranteed to be fully initialized, so that a sizeless `deinitialize()` can be safely called on it. However, in the case of move-initialization and move-assignment, it can leave the source buffer partially *deinitialized* which is just as big a problem. It is also not very useful in practice, since real collections tend to grow monotonically, periodically moving their contents into larger and larger buffers. - -A better option is to use the source’s `count`, combined with an `at:` offset, and set the precondition that `offset` `+` source`.count` `<=` destination`.count`. This *`at:from:`* system inspired by existing `UnsafeMutableRawPointer` APIs is an extremely useful system for supporting partially initialized buffer pointers by allowing us to initialize, assign, and move buffers in segments. For example, it would now be easy to concatenate multiple buffers into one. - -```swift -let pixels:Int = scanlines.map{ $0.count }.reduce(0, +) -var image = UnsafeMutableBufferPointer.allocate(capacity: pixels) - -var filled:Int = 0 -for scanline:UnsafeMutableBufferPointer in scanlines -{ - image.moveInitialize(at: filled, from: scanline) - filled += scanline.count -} - -image.deinitialize(at: 0, count: filled) -image.deallocate() -``` +Currently, `UnsafeMutableRawPointer`’s methods take an `at:` offset argument that is interpreted in strides. This argument is not currently in use in the entire Swift standard library, and we believe that it is not useful in practice. -Under this system, it will be impossible to leave part of a source buffer deinitialized, and every segment of a destination buffer will be accessible (instead of only segments starting at index `0`.) - -> note: we use `at:` instead of `+` because pointer arithmetic does not play well with the nillable buffer pointer `baseAddress`. - -> note: while deinitialization can be performed on a buffer pointer using its own `count` property, we have decided it’s better to explicitly ask for the `at:count:` pair to be consistent with real use patterns and the rest of the proposed API which operates on segments of `self`. +Unlike `UnsafeMutablePointer`, we do not add a single-instance initialize method to `UnsafeMutableRawPointer`, as such a method would probably not be useful. However, we still remove the default argument of `1` from the `count:` argument in `initializeMemory(as:repeating:count:)` to prevent confusion with calls to its buffer variant. ### `UnsafeBufferPointer` @@ -310,6 +182,8 @@ func withMemoryRebound(to:_:) -> Result The buffer type rebind method dynamically computes the new count by performing multiplication and integer division, since the target type may have a different stride than the original type. This is in line with existing precedent in the generic buffer method `initializeMemory(as:from:)` on `UnsafeMutableRawBufferPointer`. +> Note: **calling `deallocate()` on a buffer pointer is only defined behavior if the buffer pointer references a complete heap memory block**. This operation may become supported in a wider variety of cases in the future if Swift gets a more sophisticated heap allocation backend. + ### `UnsafeMutableBufferPointer` ``` @@ -318,21 +192,14 @@ func allocate(capacity:) -> UnsafeMutableBufferPointer func deallocate() func initialize(repeating:) -func initialize(at:from:) -func moveInitialize(at:from:) func assign(repeating:) -func assign(at:from:) -func moveAssign(at:from:) -func deinitialize(at:count) func withMemoryRebound(to:_:) -> Result ``` The buffer type rebind method works the same way as in `UnsafeBufferPointer`. (Type rebinding never cares about mutability.) -> note: the `at:` arguments in `UnsafeMutableBufferPointer` and `UnsafeMutableRawBufferPointer` should *not* receive default values, as they are an integral part of the buffer pointer memory state safety system, and so it is important they appear at the call site. - ### `UnsafeRawBufferPointer` ``` @@ -345,59 +212,29 @@ func bindMemory(to:) -> UnsafeBufferPointer ``` static -func allocate(bytes:alignedTo:) -> UnsafeMutableRawBufferPointer +func allocate(byteCount:alignment:) -> UnsafeMutableRawBufferPointer func deallocate() func initializeMemory(as:repeating:) -> UnsafeMutableBufferPointer -func initializeMemory(as:at:from:) -> UnsafeMutableBufferPointer -func moveInitializeMemory(as:at:from:) -> UnsafeMutableBufferPointer func bindMemory(to:) -> UnsafeMutableBufferPointer - -func copyBytes(at:from:) ``` > note: `initializeMemory(as:repeating:)` performs integer division on `self.count` (just like `bindMemory(to:)`) -> note: the return values of `initializeMemory(as:repeating:)`, `initializeMemory(as:at:from:)`, and `moveInitializeMemory(as:at:from:)` should all be marked as `@discardableResult`. - -> note: `copyBytes(from:)` takes an `at:` argument because just like `UnsafeBufferPointer`, offsetting its `baseAddress` is not syntactically easy. The function name also has “`Bytes`” in it since it’s missing the `bytes:` argument label, and we need to emphasize that it only performs a bytewise copy. - -## Detailed changes - -The proposed new API attempts to build on the existing API wherever possible. With the exception of `deallocate()` (which has good justification to replace `deallocate(capacity:)` and `deallocate(bytes:alignedTo:)`), all changes are either pure additive changes, or renames which are trivial to automigrate. This reduces the amount of source breakage. - -- **fix the ordering of the arguments in `initializeMemory(as:at:count:to:)` and rename the argument `to:` to `repeating:` in all repeated-value copy functions** +> note: the return value of `initializeMemory(as:repeating:)` should be marked as `@discardableResult`. -The ordering of the `to:` and `count:` argument labels in the `initializeMemory(as:at:count:to:)` method on `UnsafeMutableRawPointer` contradicts the rest of the Swift pointer API, where `to:` precedes `count:`. +We also make several miscellaneous changes to the API in order to tidy things up. -Because the ordering `initializeMemory(as:at:to:count:)` conflicts with the use of `to:` as the argument label for a target type, this argument should be renamed to `repeating:`. The word `repeating:` is much more clear in terms of describing the methods’ behavior, and is consistent with the use of the word in the `Array` API. +- **rename `copyBytes(from:count:)` and `copyBytes(from:)` to `copyMemory(from:byteCount:)` and `copyMemory(from:)`** -- **add the repeated-value copy assignment method `assign(repeating:count:)`** - -This addresses the missing assignment analogue to the `initialize(to:count:)` method. - -- **rename `copyBytes(from:count:)` to `copy(from:bytes:)` on `UnsafeMutableRawPointer`** - -We will enforce the convention of the use of the words `bytes`, `count`, and `capacity`. - -Since this makes the word “bytes” occur twice in `copyBytes(from:bytes:)`, we should drop the “Bytes” suffix and further rename the method to `copy(from:bytes:)`. Since `UnsafeMutableRawPointer` is inherently untyped, it is obvious that any memory transfer operation on it is a bytewise operation so the “Bytes” suffix adds only verbosity and no clarity. An unsized version of this method will also be added to `UnsafeMutableRawBufferPointer`, but keeping the “`Bytes`” suffix. - -We do not rename the `count` property on `UnsafeMutableRawBufferPointer` to `bytes` since this could be confused with the actual buffer data. - -- **rename `count` in `UnsafeMutableRawBufferPointer.allocate(count:)` to `bytes` and add an `alignedTo` parameter to make it `UnsafeMutableRawBufferPointer.allocate(bytes:alignedTo:)`** - -This brings it in line with the `UnsafeMutableRawPointer` allocator, and avoids the contradictory and inconsistent use of `count` to represent a byte quantity. Currently `UnsafeMutableRawBufferPointer.allocate(count:)` aligns to the size of `UInt`, an assumption not shared by its plain variant. +This brings the method names in line with the rest of the raw pointer API. - **add an `init(mutating:)` initializer to `UnsafeMutableBufferPointer`** This makes it much easier to make a mutable copy of an immutable buffer pointer. Such an initializer already exists on `UnsafeMutableRawBufferPointer`, so adding one to `UnsafeMutableBufferPointer` is also necessary for consistency. The reverse initializer, from `UnsafeMutableBufferPointer` to `UnsafeBufferPointer` should also be added for completeness. -- **add a mutable overload to the `copy(from:)` method on `UnsafeMutableRawBufferPointer`, the `initialize(at:from:)` and `assign(at:from:)` methods on `UnsafeMutableBufferPointer`, and the `initializeMemory(as:at:from:)` method on `UnsafeMutableRawBufferPointer`** - -Currently, for plain pointers, there is a compiler subtyping relationship between `UnsafePointer` and `UnsafeMutablePointer`. No such relationship exists between `UnsafeBufferPointer` and `UnsafeMutableBufferPointer` or their raw counterparts, so it is necessary to provide mutable overloads for these functions. - -- **add `deallocate()` to all pointer types, replacing any existing deallocation methods** +- **deprecate the sized deallocation API** Removing `capacity` from `deallocate(capacity:)` will end the confusion over what `deallocate()` does, making it obvious that `deallocate()` will free the *entire* memory block at `self`, just as if `free()` were called on it. @@ -405,7 +242,9 @@ The old `deallocate(capacity:)` method should be marked as `deprecated` and even Along similar lines, the `bytes` and `alignedTo` parameters should be removed from the `deallocate(bytes:alignedTo:)` method on `UnsafeMutableRawPointer` and `UnsafeRawPointer`. -An unsized `deallocate()` method should be added to all pointer types, even immutable ones, as Swift’s memory model does not require memory to be mutable for deallocation. This fixes [SR-3309](https://bugs.swift.org/browse/SR-3309). Note, immutable raw buffer pointers already support this API. +As discussed earlier, an unsized `deallocate()` method should be added to all pointer types, even immutable ones, as Swift’s memory model does not require memory to be mutable for deallocation. + +> note: the deallocation size parameters were originally included in early versions of Swift in order to support a more sophisticated hypothetical heap allocator backend that we wanted to have in the future. (Swift currently calls `malloc(_:)` and `free()`.) While such a backend would theoretically run more efficiently than the C backend, overengineering Swift to support it in the future has proven to be a detriment to users right now. By removing the size parameters now, we make it easier and safer to reintroduce such an API in the future without inadvertently causing silent source breakage. > note: changes to deallocation methods are not listed in the type-by-type overview below. All items in the following list are either non-source breaking, or trivially automigratable. @@ -417,6 +256,12 @@ An unsized `deallocate()` method should be added to all pointer types, even immu func withMemoryRebound(to:capacity:_:) -> Result ``` +#### New methods + +```diff ++++ func deallocate() +``` + ### `UnsafeMutablePointer` #### Existing methods @@ -425,13 +270,6 @@ func withMemoryRebound(to:capacity:_:) -> Result static func allocate(capacity:) -> UnsafeMutablePointer -func initialize(from:count:) -func moveInitialize(from:count:) - -func assign(from:count:) -func moveAssign(from:count:) - -func deinitialize(count:) func withMemoryRebound(to:capacity:_:) -> Result ``` @@ -445,6 +283,9 @@ func withMemoryRebound(to:capacity:_:) -> Result #### New methods ```diff ++++ func deallocate() + ++++ func initialize(to:) +++ func assign(repeating:count:) ``` @@ -456,39 +297,41 @@ func withMemoryRebound(to:capacity:_:) -> Result func bindMemory(to:capacity:) -> UnsafePointer ``` +#### New methods + +```diff ++++ func deallocate() +``` + ### `UnsafeMutableRawPointer` #### Existing methods ``` -static -func allocate(bytes:alignedTo:) -> UnsafeMutableRawPointer - func bindMemory(to:capacity:) -> UnsafeMutablePointer ``` -#### Renamed methods +#### New methods ```diff ---- func initializeMemory(as:at:count:to:) -> UnsafeMutablePointer -+++ func initializeMemory(as:at:repeating:count:) -> UnsafeMutablePointer - ---- func copyBytes(from:count:) -+++ func copy(from:bytes:) ++++ func deallocate() ``` -#### New arguments +#### Renamed methods and dropped arguments ```diff ---- func initializeMemory(as:from:count:) -> UnsafeMutablePointer ---- func moveInitializeMemory(as:from:count:) -> UnsafeMutablePointer -+++ func initializeMemory(as:at:from:count:) -> UnsafeMutablePointer -+++ func moveInitializeMemory(as:at:from:count:) -> UnsafeMutablePointer -``` +--- static +--- func allocate(bytes:alignedTo:) -> UnsafeMutableRawPointer + ++++ static ++++ func allocate(byteCount:alignment:) -> UnsafeMutableRawPointer -> note: The new `at:` argument has a backwards-compatible default argument of `0`. This not only makes sense, and is consistent with the existing default value of `at:` in `initializeMemory(as:at:count:to:)`, it also prevents source breakage. +--- func initializeMemory(as:at:count:to:) -> UnsafeMutablePointer ++++ func initializeMemory(as:repeating:count:) -> UnsafeMutablePointer -> note: We are adding a *new* default argument of `MemoryLayout.alignment` for the `alignment` parameter in `allocate(bytes:alignedTo:)`. The rationale is that Swift is introducing a language-level default guarantee of word-aligned storage, so the default argument reflects Swift’s memory model. Higher alignments (such as 16-byte alignment) should be specified explicitly by the user. +--- func copyBytes(from:count:) ++++ func copyMemory(from:byteCount:) +``` ### `UnsafeBufferPointer` @@ -510,14 +353,8 @@ func bindMemory(to:capacity:) -> UnsafeMutablePointer +++ func deallocate() +++ func initialize(repeating:) -+++ func initialize(at:from:) -+++ func moveInitialize(at:from:) - +++ func assign(repeating:) -+++ func assign(at:from:) -+++ func moveAssign(at:from:) -+++ func deinitialize(at:count:) +++ func withMemoryRebound(to:_:) -> Result ``` @@ -543,44 +380,31 @@ deallocate() deallocate() ``` -#### New/renamed arguments +#### Renamed methods and new/renamed arguments ```diff --- static --- func allocate(count:) -> UnsafeMutableRawBufferPointer +++ static -+++ func allocate(bytes:alignedTo:) -> UnsafeMutableRawBufferPointer ++++ func allocate(byteCount:alignment:) -> UnsafeMutableRawBufferPointer --- func copyBytes(from:) -+++ func copyBytes(at:from:) ++++ func copyMemory(from:) ``` #### New methods ```diff +++ func initializeMemory(as:repeating:) -> UnsafeMutableBufferPointer -+++ func initializeMemory(as:at:from:) -> UnsafeMutableBufferPointer -+++ func moveInitializeMemory(as:at:from:) -> UnsafeMutableBufferPointer +++ func bindMemory(to:) -> UnsafeMutableBufferPointer ``` -> note: for backwards compatibility, the `alignedTo:` argument in `allocate(bytes:alignedTo:)` should take a default value of `MemoryLayout.alignment`. This requires [SR-5664](https://bugs.swift.org/browse/SR-5664) to be fixed before it will work properly. - -> note: The new `at:` argument in `copyBytes(at:from:)` has a backwards-compatible default argument of `0`. This poses no risk to memory state safety, since this method can only perform a bytewise copy anyways. - ## What this proposal does not do -- **attempt to fully support partial initialization** +- **attempt to fully partial initialization** -This proposal attempts to design a buffer interface that provides some semblance of memory state safety. However, it does not fully address issues relating to ergonomics such as - - - overloading `+` for buffer pointers, allowing “pointer arithmetic” to be performed on buffer pointers - - easier buffer pointer slicing, which does not produce wasteful `MutableRandomAccessSlice>` structures - -nor does it attempt to design a higher level buffer type which would be able to provide stronger memory state guarantees. - -We expect possible solutions to these problems to purely additive, and would not require modifying the methods this proposal will introduce. +This proposal does not attempt to fill in most of the memory state APIs for buffer pointers, as doing so necessitates designing a partial initialization system, as well as a possible buffer slice rework. - **address problems relating to the generic `Sequence` buffer API** @@ -599,23 +423,22 @@ struct UnsafePointer struct UnsafeMutablePointer { ---- static func allocate(capacity:Int) -> UnsafeMutablePointer -+++ static func allocate(capacity:Int = 1) -> UnsafeMutablePointer + static func allocate(capacity:Int) -> UnsafeMutablePointer --- func deallocate(capacity:Int) +++ func deallocate() --- func initialize(to:Pointee, count:Int = 1) -+++ func initialize(repeating:Pointee, count:Int = 1) ++++ func initialize(repeating:Pointee, count:Int) ++++ func initialize(to:Pointee) func initialize(from:UnsafePointer, count:Int) - moveInitialize(from:UnsafeMutablePointer, count:Int) + func moveInitialize(from:UnsafeMutablePointer, count:Int) -+++ func assign(repeating:Pointee, count:Int = 1) ++++ func assign(repeating:Pointee, count:Int) func assign(from:UnsafePointer, count:Int) func moveAssign(from:UnsafeMutablePointer, count:Int) ---- func deinitialize(count:Int) -+++ func deinitialize(count:Int = 1) + func deinitialize(count:Int) func withMemoryRebound(to:T.Type, capacity:Int, _ body:(UnsafeMutablePointer) -> Result) -> Result @@ -634,26 +457,21 @@ struct UnsafeMutableRawPointer --- static --- func allocate(bytes:Int, alignedTo:Int) -> UnsafeMutableRawPointer +++ static -+++ func allocate(bytes:Int, alignedTo:Int = MemoryLayout.alignment) -+++ -> UnsafeMutableRawPointer ++++ func allocate(byteCount:Int, alignment:Int) -> UnsafeMutableRawPointer --- func deallocate(bytes:Int, alignedTo:Int) +++ func deallocate() --- func initializeMemory(as:T.Type, at:Int = 0, count:Int = 1, to:T) -> UnsafeMutablePointer -+++ func initializeMemory(as:T.Type, at:Int = 0, repeating:T, count:Int = 1) -> UnsafeMutablePointer ++++ func initializeMemory(as:T.Type, repeating:T, count:Int) -> UnsafeMutablePointer ---- func initializeMemory(as:T.Type, from:UnsafePointer, count:Int) -> UnsafeMutablePointer ---- func moveInitializeMemory(as:T.Type, from:UnsafeMutablePointer, count:Int) ---- -> UnsafeMutablePointer -+++ func initializeMemory(as:T.Type, at:Int = 0, from:UnsafePointer, count:Int) -+++ -> UnsafeMutablePointer -+++ func moveInitializeMemory(as:T.Type, at:Int = 0, from:UnsafeMutablePointer, count:Int) -+++ -> UnsafeMutablePointer + func initializeMemory(as:T.Type, from:UnsafePointer, count:Int) -> UnsafeMutablePointer + func moveInitializeMemory(as:T.Type, from:UnsafeMutablePointer, count:Int) + -> UnsafeMutablePointer func bindMemory(to:T.Type, count:Int) -> UnsafeMutablePointer --- func copyBytes(from:UnsafeRawPointer, count:Int) -+++ func copy(from:UnsafeRawPointer, bytes:Int) ++++ func copyMemory(from:UnsafeRawPointer, byteCount:Int) } struct UnsafeBufferPointer @@ -674,16 +492,7 @@ struct UnsafeMutableBufferPointer +++ func allocate(capacity:Int) -> UnsafeMutableBufferPointer +++ func initialize(repeating:Element) -+++ func initialize(at:Int, from:UnsafeBufferPointer) -+++ func initialize(at:Int, from:UnsafeMutableBufferPointer) -+++ func moveInitialize(at:Int, from:UnsafeMutableBufferPointer) - +++ func assign(repeating:Element) -+++ func assign(at:Int, from:UnsafeBufferPointer) -+++ func assign(at:Int, from:UnsafeMutableBufferPointer) -+++ func moveAssign(at:Int, from:UnsafeMutableBufferPointer) - -+++ func deinitialize(at:Int, count:Int) +++ func withMemoryRebound +++ (to:T.Type, _ body:(UnsafeMutableBufferPointer) -> Result) @@ -699,56 +508,55 @@ struct UnsafeRawBufferPointer struct UnsafeMutableRawBufferPointer { --- static func allocate(count:Int) -> UnsafeMutableRawBufferPointer -+++ static func allocate(bytes:Int, alignedTo:Int = MemoryLayout.alignment) -+++ -> UnsafeMutableRawBufferPointer ++++ static func allocate(byteCount:Int, alignment:Int) -> UnsafeMutableRawBufferPointer func deallocate() +++ func initializeMemory(as:T.Type, repeating:T) -> UnsafeMutableBufferPointer -+++ func initializeMemory(as:T.Type, at:Int, from:UnsafeBufferPointer) -+++ -> UnsafeMutableBufferPointer -+++ func initializeMemory(as:T.Type, at:Int, from:UnsafeMutableBufferPointer) -+++ -> UnsafeMutableBufferPointer -+++ func moveInitializeMemory(as:T.Type, at:Int, from:UnsafeMutableBufferPointer) -+++ -> UnsafeMutableBufferPointer +++ func bindMemory(to:T.Type) -> UnsafeMutableBufferPointer --- func copyBytes(from:UnsafeRawBufferPointer) -+++ func copyBytes(at:Int = 0, from:UnsafeRawBufferPointer) -+++ func copyBytes(at:Int = 0, from:UnsafeMutableRawBufferPointer) ++++ func copyMemory(from:UnsafeRawBufferPointer) } ``` ## Source compatibility -- **fix the ordering of the arguments in `initializeMemory(as:at:count:to:)` and rename the argument `to:` to `repeating:` in all repeated-value copy functions** +Everything is additive except the following. Can we deprecate all of +the original functions in Swift 5, then drop those from the binary +later in Swift 6? -This change is source breaking but can be trivially automigrated. - -- **add the repeated-value copy assignment method `assign(repeating:count:)`** +- **add `deallocate()` to all pointer types, replacing any existing deallocation methods** -This change is purely additive. +The migrator needs to drop the existing `capacity` and `alignedTo` arguments. -- **rename `copyBytes(from:count:)` to `copy(from:bytes:)` on `UnsafeMutableRawPointer`** +- **in `UnsafeMutableRawPointer.allocate(count:alignedTo:)` rename `count` to `byteCount` and `alignedTo` to `alignment`** -This change is source breaking but can be trivially automigrated. +- **in `UnsafeMutableRawBufferPointer.allocate(count:)` rename `count` to `byteCount` and add an `alignment` parameter** -- **rename `count` in `UnsafeMutableRawBufferPointer.allocate(count:)` to `bytes` and add an `alignedTo` parameter to make it `UnsafeMutableRawBufferPointer.allocate(bytes:alignedTo:)`** +This change is source breaking but can be trivially automigrated. The +`alignment:` parameter can be filled in with `MemoryLayout.stride`. -This change is source breaking but can be trivially automigrated. The `alignedTo:` parameter can be filled in with `MemoryLayout.stride`. If [SR-5664](https://bugs.swift.org/browse/SR-5664) is fixed, `MemoryLayout.stride` can even be provided as a default argument. +- **fix the arguments to `initialize(repeating:Pointee, count:Int)`** -- **add an `init(mutating:)` initializer to `UnsafeMutableBufferPointer`** +Note: initialize(to:Pointee) is backward compatible whenever the +caller relied on a default `count = 1`. -This change is purely additive. +An annotation could otherwise rename `to` to `repeating`, but we don't +want that to interfere with the default count case, so this might need to be a migrator rule. -- **add a mutable overload to the `copy(from:)` method on `UnsafeMutableRawBufferPointer`, the `initialize(at:from:)` and `assign(at:from:)` methods on `UnsafeMutableBufferPointer`, and the `initializeMemory(as:at:from:)` method on `UnsafeMutableRawBufferPointer`** +- **fix the ordering of the arguments in `initializeMemory(as:at:count:to:)`, rename `to:` to `repeating:`, and remove the `at:` argument** -This change is purely additive. +This change is source breaking but can be trivially automigrated. The +`to` argument changes position and is relabeled as `repeating`. -- **add `deallocate()` to all pointer types, replacing any existing deallocation methods** +The migrator could be taught to convert the `at:` argument into +pointer arithmetic on `self`. However, we found no code on Github that +uses the `at:` argument, so it is low priority. -This change is source-breaking, but this is a Good Thing™. The current API encourages incorrect code to be written, and sets us up for potentially catastrophic source breakage down the road should the implementations of `deallocate(capacity:)` and `deallocate(bytes:alignedTo:)` ever be “fixed”, so users should be forced to stop using them as soon as possible. +- **rename `copyBytes(from:count:)` to `copyMemory(from:byteCount:)`** +This change is source breaking but can be trivially automigrated. ## Effect on ABI stability @@ -760,8 +568,6 @@ Some proposed changes in this proposal change the public API. Removing sized deallocators right now will break ABI, but offers increased ABI and API stability in the future as reallocator methods can be added in the future without having to rename `deallocate(capacity:)` which currently occupies a “reallocator” name, but has “`free()`” behavior. -This proposal seeks to tackle all the breaking changes required for such an overhaul of Swift pointers, and leaves unanswered only additive changes that still need to be made in the future, reducing the need for future breakage. - ## Alternatives considered - **keeping sized deallocators and fixing the stdlib implementation instead** @@ -779,4 +585,3 @@ The label `value:` or `toValue:` doesn’t fully capture the repeating nature of ```swift ptr.initialize(value: value) ``` -