From e888af466c9993de977f6999a131eadd33291b06 Mon Sep 17 00:00:00 2001 From: taylor swift Date: Wed, 13 Sep 2017 18:57:53 -0500 Subject: [PATCH 1/7] amended se-184 --- proposals/0184-unsafe-pointers-add-missing.md | 147 +++++++++--------- 1 file changed, 74 insertions(+), 73 deletions(-) diff --git a/proposals/0184-unsafe-pointers-add-missing.md b/proposals/0184-unsafe-pointers-add-missing.md index 60443cfe79..99566daf36 100644 --- a/proposals/0184-unsafe-pointers-add-missing.md +++ b/proposals/0184-unsafe-pointers-add-missing.md @@ -10,7 +10,7 @@ Swift’s pointer types are an important interface for low-level memory manipulation, but the current API design is not very consistent, complete, or convenient. Many memory methods demand a `capacity:` or `count:` argument, forcing the user to manually track the size of the memory block, even though most of the time this is either unnecessary, or redundant as buffer pointers track this information natively. In some places, poor naming choices and overengineered function signatures compromise memory safety by leading users to believe that they have allocated or freed memory when in fact, they have not. -This proposal seeks to improve the Swift pointer API by ironing out naming inconsistencies, adding sensible default argument values, adding missing methods, and reducing excessive verbosity, offering a more convenient, more sensible, and less bug-prone API. We also attempt to introduce a buffer pointer API that supports partial initialization without excessively compromising memory state safety. +This proposal seeks to improve the Swift pointer API by ironing out naming inconsistencies, adding missing methods, and reducing excessive verbosity, offering a more convenient, more sensible, and less bug-prone API. We also attempt to introduce a buffer pointer API that supports partial initialization without excessively compromising memory state safety. Swift-evolution thread: [Pitch: Improved Swift pointers](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170710/038013.html), [Pitch: More Improved Swift pointers](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170717/038121.html) @@ -154,15 +154,15 @@ static func allocate(as:count:) -> PointerType func deallocate() -func initialize(as:at:repeating:count:) -func initialize(as:at:from:count:) -func moveInitialize(as:at:from:count:) +func initialize(at:as:repeating:count:) +func initialize(at:as:from:count:) +func moveInitialize(at:as:from:count:) -func assign(as:at:repeating:count:) -func assign(as:at:from:count:) -func moveAssign(as:at:from:count:) +func assign(at:as:repeating:count:) +func assign(at:as:from:count:) +func moveAssign(at:as:from:count:) -func deinitialize(as:at:count:) +func deinitialize(at:as:count:) func rebindMemory(as:count:) func copyBytes(at:from:count:) @@ -171,7 +171,7 @@ func copyBytes(at:from:count:) where - **`as:`** refers to the element type - - **`at:`** refers to an offset from `self`, in strides of the element type + - **`at:`** refers to an offset from `self`, in strides of the element type, if any - **`repeating:`** refers to a repeating value - **`from:`** refers to a second pointer which serves as the **source** - **`count:`** refers the number of elements the operation operates on @@ -184,7 +184,9 @@ On actual pointer types, most of these parameters are unnecessary, and some of t - typed pointers don’t need an `as:` parameter (except for type rebinding) — they already have a type. It also doesn’t make sense for them to support byte-wise copying. - - pointers for which it is syntactically easy to offset in strides (for example, by pointer arithmetic with `+`), don’t need to take an `at:` argument. + - pointers for which it is syntactically easy to offset in strides, or in the case of raw pointers, bytes (for example, by pointer arithmetic with `+`), don’t need to take an `at:` argument. + +This proposal moves the `at:` parameter to the front of the parameter list. (Where this parameter used to appear in `UnsafeMutableRawPointer`, it came after the `as:` parameter.) The rationale for this is that this proposal redefines the `at:` parameter in terms of pointer arithmetic offsets, and pointer arithmetic is written “first” from left to right. Since some of our pointer types will use `at:` and others won’t, we want the offset value to occur roughly in the same reading order across all our pointer types. > note: some of these conceptual argument labels have different names in the real API. `as:` is written as `to:` in the type-rebinding methods because it sounds better. `count:` is sometimes written as `capacity:` or `bytes:` to express the assumptions about the stride and initialization state of the memory in question. @@ -211,6 +213,7 @@ func allocate(capacity:) -> UnsafeMutablePointer func deallocate() func initialize(repeating:count:) +func initializePointee(to:) func initialize(from:count:) func moveInitialize(from:count:) @@ -222,11 +225,20 @@ func deinitialize(count:) func withMemoryRebound(to:capacity:_:) -> Result ``` -Like `UnsafePointer`, `UnsafeMutablePointer`’s type rebinding method is written as a decorator, and its methods do not need `at:` arguments. +Like `UnsafePointer`, `UnsafeMutablePointer`’s type rebinding method is written as a decorator, and its methods do not need `at:` arguments. -> note: the `count:` argument in `initialize(repeating:count:)`, `assign(repeating:count:)`, and `deinitialize(count:)` and the `capacity:` argument in `allocate(capacity:)` should receive a default value of `1`, since those methods are commonly used with a count of `1`. +Unlike earlier versions of this proposal, we propose adding a method `initializePointee(to:)` to `UnsafeMutablePointer`. Previously, the single-element initialization case was supported by a default argument of `1` on `initialize(repeating:count:)`’s `count:` parameter, but it was decided this was too confusing in terms of API readability. For example, calls to `initialize(repeating:count:)` and its corresponding method on `UnsafeMutableBufferPointer` were prone to look the same. -> `withMemoryRebound(to:capacity:_:)` should *not* receive a default `count:` value, to avoid misleading users about type binding in Swift’s memory model. (The entire memory block must be rebound, not just the first element under the pointer.) +```swift +plainPointer.initialize(repeating: pointee) +bufferPointer.initialize(repeating: repeatedValue) +``` + +Increasing API surface by adding this method is justified by the large number of calls to `initialize(to:count:)` in the standard library (and likely other code) which rely on the default argument of `1`. We do *not* need to add a corresponding `assignPointee(to:)` method since this can be done with the assignment operator. + +```swift +ptr.pointee = newValue +``` ### `UnsafeRawPointer` @@ -243,24 +255,24 @@ static func allocate(bytes:alignedTo:) -> UnsafeMutableRawPointer func deallocate() -func initializeMemory(as:at:repeating:count:) -> UnsafeMutablePointer -func initializeMemory(as:at:from:count:) -> UnsafeMutablePointer -func moveInitializeMemory(as:at:from:count:) -> UnsafeMutablePointer +func initializeMemory(as:repeating:count:) -> UnsafeMutablePointer +func initializeMemory(as:from:count:) -> UnsafeMutablePointer +func moveInitializeMemory(as:from:count:) -> UnsafeMutablePointer func bindMemory(to:capacity:) -> UnsafeMutablePointer -func copy(from:bytes:) +func copyMemory(from:bytes:) ``` The `as:` argument in `allocate(bytes:alignedTo:)` is represented by an alignment parameter which takes an integer. This is more useful since we often need a computed alignment (like when aligning a structure) instead of a preset type alignment. -> note: although raw pointers support pointer arithmetic, it is not done in strides, and `ptr + offset * MemoryLayout.stride` is painfully verbose, so these methods take an `at:` offset. +Currently, `UnsafeMutableRawPointer`’s methods take an `at:` offset argument that is interpreted in strides. This argument is not currently in use in the entire Swift standard library, and we believe that it is not useful in practice. This proposal replaces it with a `atByteOffset:` argument which takes a byte offset, a much more useful parameter. Since a byte offset off of a `UnsafeMutableRawPointer` can easily be obtained through pointer arithmetic, we do not actually need such an argument here. -> note: the `at:` arguments in `UnsafeMutableRawPointer` should all have a default value of `0`; i.e., no offset. +Unlike `UnsafeMutablePointer`, we do not add a single-instance initialize method to `UnsafeMutableRawPointer`, as such a method would probably not be useful. However, we still remove the default argument of `1` from the `count:` argument in `initializeMemory(as:repeating:count:)` to prevent confusion with calls to its buffer variant. ------------- -Buffer pointers are conceptually similar, except the `count:` argument is often unnecessary since they track their own length internally. This means you would call +Buffer pointers are conceptually similar to plain pointers, except the `count:` argument is often unnecessary since they track their own length internally. This means you would call ```swift ptr1.initialize(repeating: value, count: count) @@ -295,7 +307,9 @@ image.deinitialize(at: 0, count: filled) image.deallocate() ``` -Under this system, it will be impossible to leave part of a source buffer deinitialized, and every segment of a destination buffer will be accessible (instead of only segments starting at index `0`.) +Under this system, it will be impossible to leave part of a source buffer deinitialized, and every segment of a destination buffer will be accessible (instead of only segments starting at index `0`.) + +For now, **calling `deallocate()` on a buffer pointer is only defined behavior if the buffer pointer references a complete heap memory block**. This operation may become supported in a wider variety of cases in the future if Swift gets a more sophisticated heap allocation backend. > note: we use `at:` instead of `+` because pointer arithmetic does not play well with the nillable buffer pointer `baseAddress`. @@ -349,19 +363,19 @@ func allocate(bytes:alignedTo:) -> UnsafeMutableRawBufferPointer func deallocate() func initializeMemory(as:repeating:) -> UnsafeMutableBufferPointer -func initializeMemory(as:at:from:) -> UnsafeMutableBufferPointer -func moveInitializeMemory(as:at:from:) -> UnsafeMutableBufferPointer +func initializeMemory(atByteOffset:as:from:) -> UnsafeMutableBufferPointer +func moveInitializeMemory(atByteOffset:as:from:) -> UnsafeMutableBufferPointer func bindMemory(to:) -> UnsafeMutableBufferPointer -func copyBytes(at:from:) +func copyMemory(at:from:) ``` > note: `initializeMemory(as:repeating:)` performs integer division on `self.count` (just like `bindMemory(to:)`) > note: the return values of `initializeMemory(as:repeating:)`, `initializeMemory(as:at:from:)`, and `moveInitializeMemory(as:at:from:)` should all be marked as `@discardableResult`. -> note: `copyBytes(from:)` takes an `at:` argument because just like `UnsafeBufferPointer`, offsetting its `baseAddress` is not syntactically easy. The function name also has “`Bytes`” in it since it’s missing the `bytes:` argument label, and we need to emphasize that it only performs a bytewise copy. +> note: even though the `at:` argument in `copyMemory(at:from:)` is in terms of bytes, it is not written as `atByteOffset` since there is no type object parameter in the function signature that could suggest that the offset is in typed strides. ## Detailed changes @@ -377,13 +391,11 @@ Because the ordering `initializeMemory(as:at:to:count:)` conflicts with This addresses the missing assignment analogue to the `initialize(to:count:)` method. -- **rename `copyBytes(from:count:)` to `copy(from:bytes:)` on `UnsafeMutableRawPointer`** - -We will enforce the convention of the use of the words `bytes`, `count`, and `capacity`. +- **rename `copyBytes(from:count:)` and `copyBytes(from:)` to `copyMemory(from:bytes:)` and `copyMemory(at:from:)`** -Since this makes the word “bytes” occur twice in `copyBytes(from:bytes:)`, we should drop the “Bytes” suffix and further rename the method to `copy(from:bytes:)`. Since `UnsafeMutableRawPointer` is inherently untyped, it is obvious that any memory transfer operation on it is a bytewise operation so the “Bytes” suffix adds only verbosity and no clarity. An unsized version of this method will also be added to `UnsafeMutableRawBufferPointer`, but keeping the “`Bytes`” suffix. +This brings the method names in line with the rest of the raw pointer API. -We do not rename the `count` property on `UnsafeMutableRawBufferPointer` to `bytes` since this could be confused with the actual buffer data. +> note: we do not change the `copyBytes(from:)` collection method. - **rename `count` in `UnsafeMutableRawBufferPointer.allocate(count:)` to `bytes` and add an `alignedTo` parameter to make it `UnsafeMutableRawBufferPointer.allocate(bytes:alignedTo:)`** @@ -393,7 +405,7 @@ This brings it in line with the `UnsafeMutableRawPointer` allocator, and avoids This makes it much easier to make a mutable copy of an immutable buffer pointer. Such an initializer already exists on `UnsafeMutableRawBufferPointer`, so adding one to `UnsafeMutableBufferPointer` is also necessary for consistency. The reverse initializer, from `UnsafeMutableBufferPointer` to `UnsafeBufferPointer` should also be added for completeness. -- **add a mutable overload to the `copy(from:)` method on `UnsafeMutableRawBufferPointer`, the `initialize(at:from:)` and `assign(at:from:)` methods on `UnsafeMutableBufferPointer`, and the `initializeMemory(as:at:from:)` method on `UnsafeMutableRawBufferPointer`** +- **add a mutable overload to the `copyMemory(at:from:)` method on `UnsafeMutableRawBufferPointer`, the `initialize(at:from:)` and `assign(at:from:)` methods on `UnsafeMutableBufferPointer`, and the `initializeMemory(atByteOffset:as:from:)` method on `UnsafeMutableRawBufferPointer`** Currently, for plain pointers, there is a compiler subtyping relationship between `UnsafePointer` and `UnsafeMutablePointer`. No such relationship exists between `UnsafeBufferPointer` and `UnsafeMutableBufferPointer` or their raw counterparts, so it is necessary to provide mutable overloads for these functions. @@ -407,6 +419,8 @@ Along similar lines, the `bytes` and `alignedTo` parameters should be removed fr An unsized `deallocate()` method should be added to all pointer types, even immutable ones, as Swift’s memory model does not require memory to be mutable for deallocation. This fixes [SR-3309](https://bugs.swift.org/browse/SR-3309). Note, immutable raw buffer pointers already support this API. +> note: the deallocation size parameters were originally included in early versions of Swift in order to support a more sophisticated hypothetical heap allocator backend that we wanted to have in the future. (Swift currently calls `malloc(_:)` and `free()`.) While such a backend would theoretically run more efficiently than the C backend, overengineering Swift to support it in the future has proven to be a detriment to users right now. By removing the size parameters now, we make it easier and safer to reintroduce such an API in the future without inadvertently causing silent source breakage. + > note: changes to deallocation methods are not listed in the type-by-type overview below. All items in the following list are either non-source breaking, or trivially automigratable. ### `UnsafePointer` @@ -445,6 +459,7 @@ func withMemoryRebound(to:capacity:_:) -> Result #### New methods ```diff ++++ func initializePointee(to:) +++ func assign(repeating:count:) ``` @@ -464,30 +479,22 @@ func bindMemory(to:capacity:) -> UnsafePointer static func allocate(bytes:alignedTo:) -> UnsafeMutableRawPointer +func initializeMemory(as:from:count:) -> UnsafeMutablePointer +func moveInitializeMemory(as:from:count:) -> UnsafeMutablePointer + func bindMemory(to:capacity:) -> UnsafeMutablePointer ``` -#### Renamed methods +#### Renamed methods and dropped arguments ```diff --- func initializeMemory(as:at:count:to:) -> UnsafeMutablePointer -+++ func initializeMemory(as:at:repeating:count:) -> UnsafeMutablePointer ++++ func initializeMemory(as:repeating:count:) -> UnsafeMutablePointer --- func copyBytes(from:count:) -+++ func copy(from:bytes:) ++++ func copyMemory(from:bytes:) ``` -#### New arguments - -```diff ---- func initializeMemory(as:from:count:) -> UnsafeMutablePointer ---- func moveInitializeMemory(as:from:count:) -> UnsafeMutablePointer -+++ func initializeMemory(as:at:from:count:) -> UnsafeMutablePointer -+++ func moveInitializeMemory(as:at:from:count:) -> UnsafeMutablePointer -``` - -> note: The new `at:` argument has a backwards-compatible default argument of `0`. This not only makes sense, and is consistent with the existing default value of `at:` in `initializeMemory(as:at:count:to:)`, it also prevents source breakage. - > note: We are adding a *new* default argument of `MemoryLayout.alignment` for the `alignment` parameter in `allocate(bytes:alignedTo:)`. The rationale is that Swift is introducing a language-level default guarantee of word-aligned storage, so the default argument reflects Swift’s memory model. Higher alignments (such as 16-byte alignment) should be specified explicitly by the user. ### `UnsafeBufferPointer` @@ -543,7 +550,7 @@ deallocate() deallocate() ``` -#### New/renamed arguments +#### Renamed methods and new/renamed arguments ```diff --- static @@ -552,22 +559,22 @@ deallocate() +++ func allocate(bytes:alignedTo:) -> UnsafeMutableRawBufferPointer --- func copyBytes(from:) -+++ func copyBytes(at:from:) ++++ func copyMemory(at:from:) ``` #### New methods ```diff +++ func initializeMemory(as:repeating:) -> UnsafeMutableBufferPointer -+++ func initializeMemory(as:at:from:) -> UnsafeMutableBufferPointer -+++ func moveInitializeMemory(as:at:from:) -> UnsafeMutableBufferPointer ++++ func initializeMemory(atByteOffset:as:from:) -> UnsafeMutableBufferPointer ++++ func moveInitializeMemory(atByteOffset:as:from:) -> UnsafeMutableBufferPointer +++ func bindMemory(to:) -> UnsafeMutableBufferPointer ``` > note: for backwards compatibility, the `alignedTo:` argument in `allocate(bytes:alignedTo:)` should take a default value of `MemoryLayout.alignment`. This requires [SR-5664](https://bugs.swift.org/browse/SR-5664) to be fixed before it will work properly. -> note: The new `at:` argument in `copyBytes(at:from:)` has a backwards-compatible default argument of `0`. This poses no risk to memory state safety, since this method can only perform a bytewise copy anyways. +> note: The new `at:` argument in `copyMemory(at:from:)` has a backwards-compatible default argument of `0`. This poses no risk to memory state safety, since this method can only perform a bytewise copy anyways. ## What this proposal does not do @@ -599,23 +606,22 @@ struct UnsafePointer struct UnsafeMutablePointer { ---- static func allocate(capacity:Int) -> UnsafeMutablePointer -+++ static func allocate(capacity:Int = 1) -> UnsafeMutablePointer + static func allocate(capacity:Int) -> UnsafeMutablePointer --- func deallocate(capacity:Int) +++ func deallocate() --- func initialize(to:Pointee, count:Int = 1) -+++ func initialize(repeating:Pointee, count:Int = 1) ++++ func initialize(repeating:Pointee, count:Int) ++++ func initializePointee(to:Pointee) func initialize(from:UnsafePointer, count:Int) moveInitialize(from:UnsafeMutablePointer, count:Int) -+++ func assign(repeating:Pointee, count:Int = 1) ++++ func assign(repeating:Pointee, count:Int) func assign(from:UnsafePointer, count:Int) func moveAssign(from:UnsafeMutablePointer, count:Int) ---- func deinitialize(count:Int) -+++ func deinitialize(count:Int = 1) + func deinitialize(count:Int) func withMemoryRebound(to:T.Type, capacity:Int, _ body:(UnsafeMutablePointer) -> Result) -> Result @@ -640,20 +646,16 @@ struct UnsafeMutableRawPointer +++ func deallocate() --- func initializeMemory(as:T.Type, at:Int = 0, count:Int = 1, to:T) -> UnsafeMutablePointer -+++ func initializeMemory(as:T.Type, at:Int = 0, repeating:T, count:Int = 1) -> UnsafeMutablePointer ++++ func initializeMemory(as:T.Type, repeating:T, count:Int) -> UnsafeMutablePointer ---- func initializeMemory(as:T.Type, from:UnsafePointer, count:Int) -> UnsafeMutablePointer ---- func moveInitializeMemory(as:T.Type, from:UnsafeMutablePointer, count:Int) ---- -> UnsafeMutablePointer -+++ func initializeMemory(as:T.Type, at:Int = 0, from:UnsafePointer, count:Int) -+++ -> UnsafeMutablePointer -+++ func moveInitializeMemory(as:T.Type, at:Int = 0, from:UnsafeMutablePointer, count:Int) -+++ -> UnsafeMutablePointer + func initializeMemory(as:T.Type, from:UnsafePointer, count:Int) -> UnsafeMutablePointer + func moveInitializeMemory(as:T.Type, from:UnsafeMutablePointer, count:Int) + -> UnsafeMutablePointer func bindMemory(to:T.Type, count:Int) -> UnsafeMutablePointer --- func copyBytes(from:UnsafeRawPointer, count:Int) -+++ func copy(from:UnsafeRawPointer, bytes:Int) ++++ func copyMemory(from:UnsafeRawPointer, bytes:Int) } struct UnsafeBufferPointer @@ -704,18 +706,18 @@ struct UnsafeMutableRawBufferPointer func deallocate() +++ func initializeMemory(as:T.Type, repeating:T) -> UnsafeMutableBufferPointer -+++ func initializeMemory(as:T.Type, at:Int, from:UnsafeBufferPointer) ++++ func initializeMemory(atByteOffset:Int, as:T.Type, from:UnsafeBufferPointer) +++ -> UnsafeMutableBufferPointer -+++ func initializeMemory(as:T.Type, at:Int, from:UnsafeMutableBufferPointer) ++++ func initializeMemory(atByteOffset:Int, as:T.Type, from:UnsafeMutableBufferPointer) +++ -> UnsafeMutableBufferPointer -+++ func moveInitializeMemory(as:T.Type, at:Int, from:UnsafeMutableBufferPointer) ++++ func moveInitializeMemory(atByteOffset:Int, as:T.Type, from:UnsafeMutableBufferPointer) +++ -> UnsafeMutableBufferPointer +++ func bindMemory(to:T.Type) -> UnsafeMutableBufferPointer --- func copyBytes(from:UnsafeRawBufferPointer) -+++ func copyBytes(at:Int = 0, from:UnsafeRawBufferPointer) -+++ func copyBytes(at:Int = 0, from:UnsafeMutableRawBufferPointer) ++++ func copyMemory(at:Int = 0, from:UnsafeRawBufferPointer) ++++ func copyMemory(at:Int = 0, from:UnsafeMutableRawBufferPointer) } ``` @@ -729,7 +731,7 @@ This change is source breaking but can be trivially automigrated. This change is purely additive. -- **rename `copyBytes(from:count:)` to `copy(from:bytes:)` on `UnsafeMutableRawPointer`** +- **rename `copyBytes(from:count:)` and `copyBytes(from:)` to `copyMemory(from:bytes:)` and `copyMemory(at:from:)`** This change is source breaking but can be trivially automigrated. @@ -741,7 +743,7 @@ This change is source breaking but can be trivially automigrated. The `alignedTo This change is purely additive. -- **add a mutable overload to the `copy(from:)` method on `UnsafeMutableRawBufferPointer`, the `initialize(at:from:)` and `assign(at:from:)` methods on `UnsafeMutableBufferPointer`, and the `initializeMemory(as:at:from:)` method on `UnsafeMutableRawBufferPointer`** +- **add a mutable overload to the `copyMemory(at:from:)` method on `UnsafeMutableRawBufferPointer`, the `initialize(at:from:)` and `assign(at:from:)` methods on `UnsafeMutableBufferPointer`, and the `initializeMemory(atByteOffset:as:from:)` method on `UnsafeMutableRawBufferPointer`** This change is purely additive. @@ -779,4 +781,3 @@ The label `value:` or `toValue:` doesn’t fully capture the repeating nature of ```swift ptr.initialize(value: value) ``` - From 676d53b09b4ac7452a68923a4b94475a9bce8d96 Mon Sep 17 00:00:00 2001 From: taylor swift Date: Sat, 30 Sep 2017 02:01:48 -0500 Subject: [PATCH 2/7] unsafe pointers part 1 --- proposals/0184b-unsafe-pointers-part-1.md | 567 ++++++++++++++++++++++ 1 file changed, 567 insertions(+) create mode 100644 proposals/0184b-unsafe-pointers-part-1.md diff --git a/proposals/0184b-unsafe-pointers-part-1.md b/proposals/0184b-unsafe-pointers-part-1.md new file mode 100644 index 0000000000..57bae528ad --- /dev/null +++ b/proposals/0184b-unsafe-pointers-part-1.md @@ -0,0 +1,567 @@ +# Unsafe[Mutable][Raw][Buffer]Pointer: add missing methods, adjust existing labels for clarity, and remove deallocation size + +* Proposal: [SE-0184](0184-unsafe-pointers-part-1.md) +* Author: [Kelvin Ma (“Taylor Swift”)](https://github.com/kelvin13) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: +* Implementation: + +## Introduction + +*This document is a spin-off from a larger [original proposal](0184-unsafe-pointers-add-missing.md), SE-0184, which covers only those aspects of SE-1084 which do not deal with partial buffer memory state. Designing the partial buffer memory state API clearly requires more work, and has been left out of the scope of this document.* + +Swift’s pointer types are an important interface for low-level memory manipulation, but the current API design is not very consistent, complete, or convenient. In some places, poor naming choices and overengineered function signatures compromise memory safety by leading users to believe that they have allocated or freed memory when in fact, they have not. This proposal seeks to improve the Swift pointer API by ironing out naming inconsistencies, adding missing methods, and reducing excessive verbosity, offering a more convenient, more sensible, and less bug-prone API. + +Swift-evolution threads: [Pitch: Improved Swift pointers](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170710/038013.html), [Pitch: More Improved Swift pointers](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170717/038121.html) + +Implementation branch: + +## Background + +There are four binary memorystate operations: *initialization*, *move-initialization*, *assignment*, and *move-assignment*, and two unary memorystate operations: *deinitialization* and *type rebinding*. The binary operations can be grouped according to how they affect the source buffer and the destination buffer. **Copy** operations only read from the source buffer, leaving it unchanged. **Move** operations deinitialize the source memory, decrementing the reference count by 1 if the memory type is not a trivial type. **Retaining** operations initialize the destination memory, incrementing the reference count by 1 if applicable. **Releasing** operations deinitialize the destination memory before reinitializing it with the new values, resulting in a net change in the reference count of 0, if applicable. + +| | Copy (+0) | Move (−1) | +| -------------: |----------: | ---------: | +| **Retaining (+1)** | initialize | move-initialize | +| **Releasing (+0)** | assign | move-assign | + +Raw pointers also have a unique operation, *bytewise-copying*, which we will lump together with the memorystate functions, but does not actually change a pointer’s memory state. + +Most of these operations become more relevant in the discussion of partial buffer memory state, which is not in the scope of this document. This document only proposes changes related to memory allocation, type-rebinding, and two special *unary* forms of initialization and assignment which initialize memory to a fixed, repeating value. + +## Motivation + +Right now, `UnsafeMutableBufferPointer` is kind of a black box when it comes to producing and modifying instances of it. Much of the API present on `UnsafeMutablePointer` is absent on its buffer variant. To create, bind, allocate, initialize, and deallocate them, you have to extract `baseAddress`es and `count`s. This is unfortunate because `UnsafeMutableBufferPointer` provides a handy container for tracking the size of a memory buffer, but to actually make use of this information, the buffer pointer must be disassembled. In practice, this means the use of memory buffers requires frequent (and annoying) conversion back and forth between buffer pointers and base address–count pairs. For example, buffer allocation requires the creation of a temporary `UnsafeMutablePointer` instance. This means that the following “idiom” is very common in Swift code: + +```swift +let buffer = UnsafeMutableBufferPointer(start: UnsafeMutablePointer.allocate(capacity: byteCount), count: byteCount) +``` + +Aside from being extremely long and unwieldy, and requiring the creation of a temporary, `byteCount` must appear twice. + +You can’t even cast buffer pointer types to their mutable or immutable forms without creating a temporary. + +```swift + +var mutableBuffer = UnsafeMutableBufferPointer(start: UnsafeMutablePointer(mutating: immutableBuffer.baseAddress!), count: immutableBuffer.count) +``` + +Currently, memory is deallocated by an instance method on `UnsafeMutablePointer`, `deallocate(count:)`. Like much of the Swift pointer API, performing this operation on a buffer pointer requires extracting `baseAddress!` and `count`. It is very common for the allocation code above to be immediately followed by: + +```swift +defer +{ + buffer.baseAddress?.deallocate(capacity: buffer.count) +} +``` + +The `?` is sometimes exchanged with an `!` depending on the personality of the author, as normally, neither operator is meaningful here — the `baseAddress` is never `nil` if the buffer pointer was created around an instance of `UnsafeMutablePointer`. + +This method is extremely problematic because nearly all users, on first seeing the signature of `deallocate(capacity:)`, will naturally conclude from the `capacity` label that `deallocate(capacity:)` is equivalent to some kind of `realloc()` that can only shrink the buffer. However this is not the actual behavior — `deallocate(capacity:)` actually *ignores* the `capacity` argument and just calls `free()` on `self`. The current API is not only awkward and suboptimal, it is *misleading*. You can write perfectly legal Swift code that shouldn’t segfault, but still can, for example + +```swift +var ptr = UnsafeMutablePointer.allocate(capacity: 1000000) +ptr.initialize(to: 13, count: 1000000) +ptr.deallocate(capacity: 500000) // deallocate the second half of the memory block +ptr[0] // segmentation fault +``` + +where the first 500000 addresses should still be valid if the [documentation](https://developer.apple.com/documentation/swift/unsafemutablepointer/2295090-deallocate) is to be read literally. + +Users who are *aware* of this behavior may also choose to disregard the `capacity` argument and write things like this: + +```swift +defer +{ + buffer.baseAddress?.deallocate(capacity: 42) +} +``` + +which is functionally equivalent. However this will lead to disastrous source breakage if the implementation of `deallocate(capacity:)` is ever “corrected”. Since the API would not change, such code would still compile, but suddenly start failing at runtime. Thus, the current API, combined with incorrect documentation, is serving as a vector for introducing memory bugs into Swift code. + +Finally, some of the naming choices in the current API deserve a second look. While the original API intended to introduce a naming convention where `bytes` refers to uninitialized memory, `capacity` to uninitialized elements, and `count` to initialized elements, the actual usage of the three words does not always agree. In `copyBytes(from:count:)`, `count` refers to the number of *bytes*, which may or may not be initialized. Similarly, the `UnsafeMutableRawBufferPointer` `allocate(count:)` type method includes a `count` argument which actually refers to uninitialized bytes. The argument label `to:` is also excessively overloaded; sometimes it refers to a type `T.Type`, and sometimes it refers to a repeated value parameter. This becomes problematic when both parameters appear in the same method, as in `initializeMemory(as:at:count:to)`. + +## Proposed solution + +The ultimate goal of the API redesign is to bring all of the functionality in `UnsafeMutablePointer` and `UnsafeMutableRawPointer` to their buffer types, `UnsafeMutableBufferPointer` and `UnsafeMutableRawBufferPointer`. Operations which are covered by this proposal are in **bold**. + +The full toolbox of methods that we could possibly support includes: + + - **allocation** + - **deallocation** + + - initialization + - move-initialization + + - assignment + - move-assignment + + - deinitialization + + - **type rebinding** + - bytewise copying + +Because copy operations (initialization and assignment) don’t mutate the source argument, they can also come in a form which takes a repeated-value source instead of a buffer source. + + - **initialization (repeated-value)** + - **assignment (repeated-value)** + +`UnsafeMutablePointer` and `UnsafeMutableRawPointer` already contain repeated-value methods for initialization in the form of `initialize(to:count:)` and `initializeMemory(as:at:count:to:)`. This proposal will add the assignment analogues. For reasons explained later, the argument label for the repeated-value parameter will be referred to as `repeating:`, not `to:`. + +### `UnsafePointer` + +``` +func deallocate() +func withMemoryRebound(to:capacity:_:) -> Result +``` + +`UnsafePointer` does not get an allocator static method, since you almost always want a mutable pointer to newly allocated memory. Its type rebinding method is also written as a decorator, taking a trailing closure, for memory safety. + +Most immutable pointer types currently do not have a deallocation method. This proposal adds them, fixing [SR-3309](https://bugs.swift.org/browse/SR-3309). Note, immutable raw buffer pointers already support this API. + +### `UnsafeMutablePointer` + +``` +static +func allocate(capacity:) -> UnsafeMutablePointer +func deallocate() + +func initialize(repeating:count:) +func initialize(to:) + +func assign(repeating:count:) + +func withMemoryRebound(to:capacity:_:) -> Result +``` + +Like `UnsafePointer`, `UnsafeMutablePointer`’s type rebinding method is written as a decorator. + +Previously, the single-element repeated-initialization case was supported by a default argument of `1` on `initialize(repeating:count:)`’s `count:` parameter, but it was decided this was too confusing in terms of API readability. For example, calls to `initialize(repeating:count:)` and its corresponding method on `UnsafeMutableBufferPointer` were prone to look the same. + +```swift +plainPointer.initialize(repeating: pointee) +bufferPointer.initialize(repeating: repeatedValue) +``` + +Increasing API surface by adding this method is justified by the large number of calls to `initialize(to:count:)` in the standard library (and likely other code) which rely on the default argument of `1`. We do *not* need to add a corresponding `assignPointee(to:)` method since this can be done with the assignment operator. + +```swift +ptr.pointee = newValue +``` + +### `UnsafeRawPointer` + +``` +func deallocate() + +func bindMemory(to:capacity:) -> UnsafePointer +``` + +### `UnsafeMutableRawPointer` + +``` +static +func allocate(byteCount:alignment:) -> UnsafeMutableRawPointer +func deallocate() + +func initializeMemory(as:repeating:count:) -> UnsafeMutablePointer + +func bindMemory(to:capacity:) -> UnsafeMutablePointer +``` + +Currently, `UnsafeMutableRawPointer`’s methods take an `at:` offset argument that is interpreted in strides. This argument is not currently in use in the entire Swift standard library, and we believe that it is not useful in practice. + +Unlike `UnsafeMutablePointer`, we do not add a single-instance initialize method to `UnsafeMutableRawPointer`, as such a method would probably not be useful. However, we still remove the default argument of `1` from the `count:` argument in `initializeMemory(as:repeating:count:)` to prevent confusion with calls to its buffer variant. + +### `UnsafeBufferPointer` + +``` +func deallocate() +func withMemoryRebound(to:_:) -> Result +``` + +The buffer type rebind method dynamically computes the new count by performing multiplication and integer division, since the target type may have a different stride than the original type. This is in line with existing precedent in the generic buffer method `initializeMemory(as:from:)` on `UnsafeMutableRawBufferPointer`. + +> Note: **calling `deallocate()` on a buffer pointer is only defined behavior if the buffer pointer references a complete heap memory block**. This operation may become supported in a wider variety of cases in the future if Swift gets a more sophisticated heap allocation backend. + +### `UnsafeMutableBufferPointer` + +``` +static +func allocate(capacity:) -> UnsafeMutableBufferPointer +func deallocate() + +func initialize(repeating:) + +func assign(repeating:) + +func withMemoryRebound(to:_:) -> Result +``` + +The buffer type rebind method works the same way as in `UnsafeBufferPointer`. (Type rebinding never cares about mutability.) + +### `UnsafeRawBufferPointer` + +``` +func deallocate() + +func bindMemory(to:) -> UnsafeBufferPointer +``` + +### `UnsafeMutableRawBufferPointer` + +``` +static +func allocate(byteCount:alignment:) -> UnsafeMutableRawBufferPointer +func deallocate() + +func initializeMemory(as:repeating:) -> UnsafeMutableBufferPointer + +func bindMemory(to:) -> UnsafeMutableBufferPointer +``` + +> note: `initializeMemory(as:repeating:)` performs integer division on `self.count` (just like `bindMemory(to:)`) + +> note: the return value of `initializeMemory(as:repeating:)` should be marked as `@discardableResult`. + +We also make several miscellaneous changes to the API in order to tidy things up. + +- **rename `copyBytes(from:count:)` and `copyBytes(from:)` to `copyMemory(from:byteCount:)` and `copyMemory(from:)`** + +This brings the method names in line with the rest of the raw pointer API. + +- **add an `init(mutating:)` initializer to `UnsafeMutableBufferPointer`** + +This makes it much easier to make a mutable copy of an immutable buffer pointer. Such an initializer already exists on `UnsafeMutableRawBufferPointer`, so adding one to `UnsafeMutableBufferPointer` is also necessary for consistency. The reverse initializer, from `UnsafeMutableBufferPointer` to `UnsafeBufferPointer` should also be added for completeness. + +- **deprecate the sized deallocation API** + +Removing `capacity` from `deallocate(capacity:)` will end the confusion over what `deallocate()` does, making it obvious that `deallocate()` will free the *entire* memory block at `self`, just as if `free()` were called on it. + +The old `deallocate(capacity:)` method should be marked as `deprecated` and eventually removed since it currently encourages dangerously incorrect code. This avoids misleading future users, encourages current users to address this potentially catastrophic memory bug, and leaves the possibility open for us to add a `deallocate(capacity:)` method in the future, or perhaps even a `reallocate(toCapacity:)` method. + +Along similar lines, the `bytes` and `alignedTo` parameters should be removed from the `deallocate(bytes:alignedTo:)` method on `UnsafeMutableRawPointer` and `UnsafeRawPointer`. + +As discussed earlier, an unsized `deallocate()` method should be added to all pointer types, even immutable ones, as Swift’s memory model does not require memory to be mutable for deallocation. + +> note: the deallocation size parameters were originally included in early versions of Swift in order to support a more sophisticated hypothetical heap allocator backend that we wanted to have in the future. (Swift currently calls `malloc(_:)` and `free()`.) While such a backend would theoretically run more efficiently than the C backend, overengineering Swift to support it in the future has proven to be a detriment to users right now. By removing the size parameters now, we make it easier and safer to reintroduce such an API in the future without inadvertently causing silent source breakage. + +> note: changes to deallocation methods are not listed in the type-by-type overview below. All items in the following list are either non-source breaking, or trivially automigratable. + +### `UnsafePointer` + +#### Existing methods + +``` +func withMemoryRebound(to:capacity:_:) -> Result +``` + +### `UnsafeMutablePointer` + +#### Existing methods + +``` +static +func allocate(capacity:) -> UnsafeMutablePointer + +func withMemoryRebound(to:capacity:_:) -> Result +``` + +#### Renamed methods + +```diff +--- func initialize(to:count:) ++++ func initialize(repeating:count:) +``` + +#### New methods + +```diff ++++ func initialize(to:) ++++ func assign(repeating:count:) +``` + +### `UnsafeRawPointer` + +#### Existing methods + +``` +func bindMemory(to:capacity:) -> UnsafePointer +``` + +### `UnsafeMutableRawPointer` + +#### Existing methods + +``` +func bindMemory(to:capacity:) -> UnsafeMutablePointer +``` + +#### Renamed methods and dropped arguments + +```diff +--- static +--- func allocate(bytes:alignedTo:) -> UnsafeMutableRawPointer + ++++ static ++++ func allocate(byteCount:alignment:) -> UnsafeMutableRawPointer + +--- func initializeMemory(as:at:count:to:) -> UnsafeMutablePointer ++++ func initializeMemory(as:repeating:count:) -> UnsafeMutablePointer + +--- func copyBytes(from:count:) ++++ func copyMemory(from:byteCount:) +``` + +### `UnsafeBufferPointer` + +#### New methods + +```diff ++++ func deallocate() + ++++ withMemoryRebound(to:_:) -> Result +``` + +### `UnsafeMutableBufferPointer` + +#### New methods + +```diff ++++ static ++++ func allocate(capacity:) -> UnsafeMutableBufferPointer ++++ func deallocate() + ++++ func initialize(repeating:) ++++ func assign(repeating:) + ++++ func withMemoryRebound(to:_:) -> Result +``` + +### `UnsafeRawBufferPointer` + +#### Existing methods + +``` +deallocate() +``` + +#### New methods + +```diff ++++ func bindMemory(to:) -> UnsafeBufferPointer +``` + +### `UnsafeMutableRawBufferPointer` + +#### Existing methods + +``` +deallocate() +``` + +#### Renamed methods and new/renamed arguments + +```diff +--- static +--- func allocate(count:) -> UnsafeMutableRawBufferPointer ++++ static ++++ func allocate(byteCount:alignment:) -> UnsafeMutableRawBufferPointer + +--- func copyBytes(from:) ++++ func copyMemory(from:) +``` + +#### New methods + +```diff ++++ func initializeMemory(as:repeating:) -> UnsafeMutableBufferPointer + ++++ func bindMemory(to:) -> UnsafeMutableBufferPointer +``` + +## What this proposal does not do + +- **attempt to fully partial initialization** + +This proposal does not attempt to fill in most of the memory state APIs for buffer pointers, as doing so necessitates designing a partial initialization system, as well as a possible buffer slice rework. + +- **address problems relating to the generic `Sequence` buffer API** + +Buffer pointers are currently missing generic `assign(from:S)` and `initializeMemory(as:S.Element.Type, from:S)` methods. The existing protocol oriented API also lacks polish and is inconvenient to use. (For example, it returns tuples.) This is an issue that can be tackled separately from the lower-level buffer-pointer-to-buffer-pointer API. + +## Detailed design + +```diff +struct UnsafePointer +{ ++++ func deallocate() + + func withMemoryRebound(to:T.Type, capacity:Int, _ body:(UnsafePointer) -> Result) + -> Result +} + +struct UnsafeMutablePointer +{ + static func allocate(capacity:Int) -> UnsafeMutablePointer + +--- func deallocate(capacity:Int) ++++ func deallocate() + +--- func initialize(to:Pointee, count:Int = 1) ++++ func initialize(repeating:Pointee, count:Int) ++++ func initialize(to:Pointee) + func initialize(from:UnsafePointer, count:Int) + func moveInitialize(from:UnsafeMutablePointer, count:Int) + ++++ func assign(repeating:Pointee, count:Int) + func assign(from:UnsafePointer, count:Int) + func moveAssign(from:UnsafeMutablePointer, count:Int) + + func deinitialize(count:Int) + + func withMemoryRebound(to:T.Type, capacity:Int, _ body:(UnsafeMutablePointer) -> Result) + -> Result +} + +struct UnsafeRawPointer +{ +--- func deallocate(bytes:Int, alignedTo:Int) ++++ func deallocate() + + func bindMemory(to:T.Type, count:Int) -> UnsafeMutablePointer +} + +struct UnsafeMutableRawPointer +{ +--- static +--- func allocate(bytes:Int, alignedTo:Int) -> UnsafeMutableRawPointer ++++ static ++++ func allocate(byteCount:Int, alignment:Int) -> UnsafeMutableRawPointer +--- func deallocate(bytes:Int, alignedTo:Int) ++++ func deallocate() + +--- func initializeMemory(as:T.Type, at:Int = 0, count:Int = 1, to:T) -> UnsafeMutablePointer ++++ func initializeMemory(as:T.Type, repeating:T, count:Int) -> UnsafeMutablePointer + + func initializeMemory(as:T.Type, from:UnsafePointer, count:Int) -> UnsafeMutablePointer + func moveInitializeMemory(as:T.Type, from:UnsafeMutablePointer, count:Int) + -> UnsafeMutablePointer + + func bindMemory(to:T.Type, count:Int) -> UnsafeMutablePointer + +--- func copyBytes(from:UnsafeRawPointer, count:Int) ++++ func copyMemory(from:UnsafeRawPointer, byteCount:Int) +} + +struct UnsafeBufferPointer +{ ++++ init(_:UnsafeMutableBufferPointer) + ++++ func deallocate() + ++++ func withMemoryRebound ++++ (to:T.Type, _ body:(UnsafeBufferPointer) -> Result) +} + +struct UnsafeMutableBufferPointer +{ ++++ init(mutating:UnsafeBufferPointer) + ++++ static ++++ func allocate(capacity:Int) -> UnsafeMutableBufferPointer + ++++ func initialize(repeating:Element) ++++ func assign(repeating:Element) + ++++ func withMemoryRebound ++++ (to:T.Type, _ body:(UnsafeMutableBufferPointer) -> Result) +} + +struct UnsafeRawBufferPointer +{ + func deallocate() + ++++ func bindMemory(to:T.Type) -> UnsafeBufferPointer +} + +struct UnsafeMutableRawBufferPointer +{ +--- static func allocate(count:Int) -> UnsafeMutableRawBufferPointer ++++ static func allocate(byteCount:Int, alignment:Int) -> UnsafeMutableRawBufferPointer + func deallocate() + ++++ func initializeMemory(as:T.Type, repeating:T) -> UnsafeMutableBufferPointer + ++++ func bindMemory(to:T.Type) -> UnsafeMutableBufferPointer + +--- func copyBytes(from:UnsafeRawBufferPointer) ++++ func copyMemory(from:UnsafeRawBufferPointer) +} +``` + +## Source compatibility + +Everything is additive except the following. Can we deprecate all of +the original functions in Swift 5, then drop those from the binary +later in Swift 6? + +- **add `deallocate()` to all pointer types, replacing any existing deallocation methods** + +The migrator needs to drop the existing `capacity` and `alignedTo` arguments. + +- **in `UnsafeMutableRawPointer.allocate(count:alignedTo:)` rename `count` to `byteCount` and `alignedTo` to `alignment`** + +- **in `UnsafeMutableRawBufferPointer.allocate(count:)` rename `count` to `byteCount` and add an `alignment` parameter** + +This change is source breaking but can be trivially automigrated. The +`alignment:` parameter can be filled in with `MemoryLayout.stride`. + +- **fix the arguments to `initialize(repeating:Pointee, count:Int)`** + +Note: initialize(to:Pointee) is backward compatible whenever the +caller relied on a default `count = 1`. + +An annotation could otherwise rename `to` to `repeating`, but we don't +want that to interfere with the default count case, so this might need to be a migrator rule. + +- **fix the ordering of the arguments in `initializeMemory(as:at:count:to:)`, rename `to:` to `repeating:`, and remove the `at:` argument** + +This change is source breaking but can be trivially automigrated. The +`to` argument changes position and is relabeled as `repeating`. + +The migrator could be taught to convert the `at:` argument into +pointer arithmetic on `self`. However, we found no code on Github that +uses the `at:` argument, so it is low priority. + +- **rename `copyBytes(from:count:)` to `copyMemory(from:byteCount:)`** + +This change is source breaking but can be trivially automigrated. + +## Effect on ABI stability + +Removing sized deallocators changes the existing ABI, as will renaming some of the methods and their argument labels. + +## Effect on API resilience + +Some proposed changes in this proposal change the public API. + +Removing sized deallocators right now will break ABI, but offers increased ABI and API stability in the future as reallocator methods can be added in the future without having to rename `deallocate(capacity:)` which currently occupies a “reallocator” name, but has “`free()`” behavior. + +## Alternatives considered + +- **keeping sized deallocators and fixing the stdlib implementation instead** + +Instead of dropping the `capacity` parameter from `deallocate(capacity:)`, we could fix the underlying implementation so that the function actually deallocates `capacity`’s worth of memory. However this would be catastrophically, and silently, source-breaking as existing code would continue compiling, but suddenly start leaking or segfaulting at runtime. `deallocate(capacity:)` can always be added back at a later date without breaking ABI or API, once users have been forced to address this potential bug. + +- **adding an initializer `UnsafeMutableBufferPointer.init(allocatingCount:)` instead of a type method to `UnsafeMutableBufferPointer`** + +The allocator could be expressed as an initializer instead of a type method. However since allocation involves acquisition of an external resource, perhaps it is better to keep with the existing convention that allocation is performed differently than regular buffer pointer construction. + +- **using the argument label `value:` instead of `repeating:` in methods such as `initialize(repeating:count:)` (originally `initialize(to:count:)`)** + +The label `value:` or `toValue:` doesn’t fully capture the repeating nature of the argument, and is inconsistent with `Array.init(repeating:count:)`. While `value:` sounds less strange when `count == 1`, on consistency and technical correctness, `repeating:` is the better term. Furthermore, `value` is a common variable name, meaning that function calls with `value:` as the label would be prone to looking like this: + +```swift +ptr.initialize(value: value) +``` From 30269a0c168b91e9e318007cdfb3fa34ab9860c1 Mon Sep 17 00:00:00 2001 From: taylor swift Date: Sat, 30 Sep 2017 02:04:09 -0500 Subject: [PATCH 3/7] rename to 0184a scheme --- ...nsafe-pointers-part-1.md => 0184a-unsafe-pointers-part-1.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename proposals/{0184b-unsafe-pointers-part-1.md => 0184a-unsafe-pointers-part-1.md} (99%) diff --git a/proposals/0184b-unsafe-pointers-part-1.md b/proposals/0184a-unsafe-pointers-part-1.md similarity index 99% rename from proposals/0184b-unsafe-pointers-part-1.md rename to proposals/0184a-unsafe-pointers-part-1.md index 57bae528ad..039f8e648f 100644 --- a/proposals/0184b-unsafe-pointers-part-1.md +++ b/proposals/0184a-unsafe-pointers-part-1.md @@ -1,6 +1,6 @@ # Unsafe[Mutable][Raw][Buffer]Pointer: add missing methods, adjust existing labels for clarity, and remove deallocation size -* Proposal: [SE-0184](0184-unsafe-pointers-part-1.md) +* Proposal: [SE-0184 A](0184a-unsafe-pointers-part-1.md) * Author: [Kelvin Ma (“Taylor Swift”)](https://github.com/kelvin13) * Review Manager: [Doug Gregor](https://github.com/DougGregor) * Status: From ccaf83e3f9169d1b6a1c0bc00473b27fcfec0e67 Mon Sep 17 00:00:00 2001 From: taylor swift Date: Sat, 30 Sep 2017 02:12:42 -0500 Subject: [PATCH 4/7] add deallocate() to list --- proposals/0184a-unsafe-pointers-part-1.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/proposals/0184a-unsafe-pointers-part-1.md b/proposals/0184a-unsafe-pointers-part-1.md index 039f8e648f..669330ed4a 100644 --- a/proposals/0184a-unsafe-pointers-part-1.md +++ b/proposals/0184a-unsafe-pointers-part-1.md @@ -256,6 +256,12 @@ As discussed earlier, an unsized `deallocate()` method should be added to all po func withMemoryRebound(to:capacity:_:) -> Result ``` +#### New methods + +```diff ++++ func deallocate() +``` + ### `UnsafeMutablePointer` #### Existing methods @@ -277,6 +283,8 @@ func withMemoryRebound(to:capacity:_:) -> Result #### New methods ```diff ++++ func deallocate() + +++ func initialize(to:) +++ func assign(repeating:count:) ``` @@ -289,6 +297,12 @@ func withMemoryRebound(to:capacity:_:) -> Result func bindMemory(to:capacity:) -> UnsafePointer ``` +#### New methods + +```diff ++++ func deallocate() +``` + ### `UnsafeMutableRawPointer` #### Existing methods @@ -297,6 +311,12 @@ func bindMemory(to:capacity:) -> UnsafePointer func bindMemory(to:capacity:) -> UnsafeMutablePointer ``` +#### New methods + +```diff ++++ func deallocate() +``` + #### Renamed methods and dropped arguments ```diff From 25a2ac63a237dfb7c40cd4923e7135c638558dd7 Mon Sep 17 00:00:00 2001 From: Kelvin Date: Sat, 30 Sep 2017 14:52:54 -0500 Subject: [PATCH 5/7] implementation branch --- proposals/0184a-unsafe-pointers-part-1.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/0184a-unsafe-pointers-part-1.md b/proposals/0184a-unsafe-pointers-part-1.md index 669330ed4a..db7eb40db5 100644 --- a/proposals/0184a-unsafe-pointers-part-1.md +++ b/proposals/0184a-unsafe-pointers-part-1.md @@ -4,7 +4,7 @@ * Author: [Kelvin Ma (“Taylor Swift”)](https://github.com/kelvin13) * Review Manager: [Doug Gregor](https://github.com/DougGregor) * Status: -* Implementation: +* Implementation: [PR 12200](https://github.com/apple/swift/pull/12200) ## Introduction @@ -14,7 +14,7 @@ Swift’s pointer types are an important interface for low-level memory manipula Swift-evolution threads: [Pitch: Improved Swift pointers](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170710/038013.html), [Pitch: More Improved Swift pointers](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170717/038121.html) -Implementation branch: +Implementation branch: [`kelvin13:se-0184a`](https://github.com/kelvin13/swift/tree/se-0184a) ## Background From a3d2c0828550efade30e662c3d7ac61f17941b86 Mon Sep 17 00:00:00 2001 From: Kelvin Date: Wed, 4 Oct 2017 17:49:41 -0500 Subject: [PATCH 6/7] merge se 0184a into 0184 --- proposals/0184-unsafe-pointers-add-missing.md | 384 +++++------------- 1 file changed, 94 insertions(+), 290 deletions(-) diff --git a/proposals/0184-unsafe-pointers-add-missing.md b/proposals/0184-unsafe-pointers-add-missing.md index 99566daf36..9b4e9fb0c7 100644 --- a/proposals/0184-unsafe-pointers-add-missing.md +++ b/proposals/0184-unsafe-pointers-add-missing.md @@ -1,76 +1,37 @@ # Unsafe[Mutable][Raw][Buffer]Pointer: add missing methods, adjust existing labels for clarity, and remove deallocation size -* Proposal: [SE-0184](0184-unsafe-pointers-add-missing.md) +* Proposal: [SE-0184](0184a-unsafe-pointers-part-1.md) * Author: [Kelvin Ma (“Taylor Swift”)](https://github.com/kelvin13) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Active review (September 1...7, 2017)** -* Implementation: [apple/swift#11464](https://github.com/apple/swift/pull/11464) +* Status: +* Implementation: [PR 12200](https://github.com/apple/swift/pull/12200) ## Introduction -Swift’s pointer types are an important interface for low-level memory manipulation, but the current API design is not very consistent, complete, or convenient. Many memory methods demand a `capacity:` or `count:` argument, forcing the user to manually track the size of the memory block, even though most of the time this is either unnecessary, or redundant as buffer pointers track this information natively. In some places, poor naming choices and overengineered function signatures compromise memory safety by leading users to believe that they have allocated or freed memory when in fact, they have not. +*This document is a spin-off from a much larger [original proposal](https://github.com/kelvin13/swift-evolution/blob/e888af466c9993de977f6999a131eadd33291b06/proposals/0184-unsafe-pointers-add-missing.md), which covers only those aspects of SE-1084 which do not deal with partial buffer memory state. Designing the partial buffer memory state API clearly requires more work, and has been left out of the scope of this document.* -This proposal seeks to improve the Swift pointer API by ironing out naming inconsistencies, adding missing methods, and reducing excessive verbosity, offering a more convenient, more sensible, and less bug-prone API. We also attempt to introduce a buffer pointer API that supports partial initialization without excessively compromising memory state safety. +Swift’s pointer types are an important interface for low-level memory manipulation, but the current API design is not very consistent, complete, or convenient. In some places, poor naming choices and overengineered function signatures compromise memory safety by leading users to believe that they have allocated or freed memory when in fact, they have not. This proposal seeks to improve the Swift pointer API by ironing out naming inconsistencies, adding missing methods, and reducing excessive verbosity, offering a more convenient, more sensible, and less bug-prone API. -Swift-evolution thread: [Pitch: Improved Swift pointers](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170710/038013.html), [Pitch: More Improved Swift pointers](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170717/038121.html) +Swift-evolution threads: [Pitch: Improved Swift pointers](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170710/038013.html), [Pitch: More Improved Swift pointers](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170717/038121.html) -Implementation branch: [**PR 11464**](https://github.com/apple/swift/pull/11464) +Implementation branch: [`kelvin13:se-0184a`](https://github.com/kelvin13/swift/tree/se-0184a) ## Background -There are four binary memorystate operations: *initialization*, *move-initialization*, *assignment*, and *move-assignment*. They can be grouped according to how they affect the source buffer and the destination buffer. **Copy** operations only read from the source buffer, leaving it unchanged. **Move** operations deinitialize the source memory, decrementing the reference count by 1 if the memory type is not a trivial type. **Retaining** operations initialize the destination memory, incrementing the reference count by 1 if applicable. **Releasing** operations deinitialize the destination memory before reinitializing it with the new values, resulting in a net change in the reference count of 0, if applicable. +There are four binary memorystate operations: *initialization*, *move-initialization*, *assignment*, and *move-assignment*, and two unary memorystate operations: *deinitialization* and *type rebinding*. The binary operations can be grouped according to how they affect the source buffer and the destination buffer. **Copy** operations only read from the source buffer, leaving it unchanged. **Move** operations deinitialize the source memory, decrementing the reference count by 1 if the memory type is not a trivial type. **Retaining** operations initialize the destination memory, incrementing the reference count by 1 if applicable. **Releasing** operations deinitialize the destination memory before reinitializing it with the new values, resulting in a net change in the reference count of 0, if applicable. | | Copy (+0) | Move (−1) | | -------------: |----------: | ---------: | | **Retaining (+1)** | initialize | move-initialize | | **Releasing (+0)** | assign | move-assign | -Note: deinitialization by itself is a unary operation; it decrements the reference count of the buffer by 1. +Raw pointers also have a unique operation, *bytewise-copying*, which we will lump together with the memorystate functions, but does not actually change a pointer’s memory state. -The four main types of Swift pointers we have currently support different subsets of this toolbox. - -### UnsafeMutablePointer - -| | Copy | Move | -| ------------- |----------: | ---------: | -| **Retaining** | `initialize(to:count:)`, `initialize(from:count:)` | `moveInitialize(from:count:)` | -| **Releasing** | `assign(from:count:)` | `moveAssign(from:count:)` | - -### UnsafeMutableRawPointer - -| | Copy | Move | -| ------------- |----------: | ---------: | -| **Retaining** | `initializeMemory(as:at:count:to:)`, `initializeMemory(as:from:count:)` | `moveInitializeMemory(as:from:count:)` | -| **Releasing** | | | - -### UnsafeMutableBufferPointer - -| | Copy | Move | -| ------------- |----------: | ---------: | -| **Retaining** | `initialize(from:)` | | -| **Releasing** | | | - - -### UnsafeMutableRawBufferPointer - -| | Copy | Move | -| ------------- |----------: | ---------: | -| **Retaining** | `initializeMemory(as:from:)` | | -| **Releasing** | | | - -There are unary memorystate operations such as *deinitialization* and *type rebinding*, which are not listed in the tables, but are still covered by this proposal. Raw pointers also have a unique operation, *bytewise-copying*, which we will lump together with the memorystate functions, but does not actually change a pointer’s memory state. +Most of these operations become more relevant in the discussion of partial buffer memory state, which is not in the scope of this document. This document only proposes changes related to memory allocation, type-rebinding, and two special *unary* forms of initialization and assignment which initialize memory to a fixed, repeating value. ## Motivation -Right now, `UnsafeMutableBufferPointer` is kind of a black box when it comes to producing and modifying instances of it. Much of the API present on `UnsafeMutablePointer` is absent on its buffer variant. To create, bind, allocate, initialize, deinitialize, and deallocate them, you have to extract `baseAddress`es and `count`s. This is unfortunate because `UnsafeMutableBufferPointer` provides a handy container for tracking the size of a memory buffer, but to actually make use of this information, the buffer pointer must be disassembled. In practice, this means the use of memory buffers requires frequent (and annoying) conversion back and forth between buffer pointers and base address–count pairs. For example, to move-initialize memory between two buffer pointers, you have to write this: - -```swift -buffer1.baseAddress?.moveInitialize(from: buffer2.baseAddress!, count: buffer1.count) -``` - -The `?` is sometimes exchanged with an `!` depending on the personality of the author, as normally, neither operator is meaningful here — the `baseAddress` is never `nil` if the buffer pointer was created around an instance of `UnsafeMutablePointer`. - -Memory buffer allocation is especially painful, since it requires the creation of a temporary `UnsafeMutablePointer` instance. This means that the following “idiom” is very common in Swift code: +Right now, `UnsafeMutableBufferPointer` is kind of a black box when it comes to producing and modifying instances of it. Much of the API present on `UnsafeMutablePointer` is absent on its buffer variant. To create, bind, allocate, initialize, and deallocate them, you have to extract `baseAddress`es and `count`s. This is unfortunate because `UnsafeMutableBufferPointer` provides a handy container for tracking the size of a memory buffer, but to actually make use of this information, the buffer pointer must be disassembled. In practice, this means the use of memory buffers requires frequent (and annoying) conversion back and forth between buffer pointers and base address–count pairs. For example, buffer allocation requires the creation of a temporary `UnsafeMutablePointer` instance. This means that the following “idiom” is very common in Swift code: ```swift let buffer = UnsafeMutableBufferPointer(start: UnsafeMutablePointer.allocate(capacity: byteCount), count: byteCount) @@ -94,6 +55,8 @@ defer } ``` +The `?` is sometimes exchanged with an `!` depending on the personality of the author, as normally, neither operator is meaningful here — the `baseAddress` is never `nil` if the buffer pointer was created around an instance of `UnsafeMutablePointer`. + This method is extremely problematic because nearly all users, on first seeing the signature of `deallocate(capacity:)`, will naturally conclude from the `capacity` label that `deallocate(capacity:)` is equivalent to some kind of `realloc()` that can only shrink the buffer. However this is not the actual behavior — `deallocate(capacity:)` actually *ignores* the `capacity` argument and just calls `free()` on `self`. The current API is not only awkward and suboptimal, it is *misleading*. You can write perfectly legal Swift code that shouldn’t segfault, but still can, for example ```swift @@ -116,18 +79,16 @@ defer which is functionally equivalent. However this will lead to disastrous source breakage if the implementation of `deallocate(capacity:)` is ever “corrected”. Since the API would not change, such code would still compile, but suddenly start failing at runtime. Thus, the current API, combined with incorrect documentation, is serving as a vector for introducing memory bugs into Swift code. -The Swift pointer API is incomplete in other ways too. For example, the `initialize(from:count:)` method on `UnsafeMutablePointer` has a repeated-value copy variant, `initialize(to:count:)`, but `assign(from:count:)` has no such variant, even though it would make just as much sense for it to have one. - Finally, some of the naming choices in the current API deserve a second look. While the original API intended to introduce a naming convention where `bytes` refers to uninitialized memory, `capacity` to uninitialized elements, and `count` to initialized elements, the actual usage of the three words does not always agree. In `copyBytes(from:count:)`, `count` refers to the number of *bytes*, which may or may not be initialized. Similarly, the `UnsafeMutableRawBufferPointer` `allocate(count:)` type method includes a `count` argument which actually refers to uninitialized bytes. The argument label `to:` is also excessively overloaded; sometimes it refers to a type `T.Type`, and sometimes it refers to a repeated value parameter. This becomes problematic when both parameters appear in the same method, as in `initializeMemory(as:at:count:to)`. ## Proposed solution -The goal of the API redesign is to bring all of the functionality in `UnsafeMutablePointer` and `UnsafeMutableRawPointer` to their buffer types, `UnsafeMutableBufferPointer` and `UnsafeMutableRawBufferPointer`. `UnsafeMutableRawBufferPointer` already contains some of this functionality, providing a useful blueprint for the proposed `UnsafeMutableBufferPointer` API. +The ultimate goal of the API redesign is to bring all of the functionality in `UnsafeMutablePointer` and `UnsafeMutableRawPointer` to their buffer types, `UnsafeMutableBufferPointer` and `UnsafeMutableRawBufferPointer`. Operations which are covered by this proposal are in **bold**. The full toolbox of methods that we could possibly support includes: - - allocation - - deallocation + - **allocation** + - **deallocation** - initialization - move-initialization @@ -136,66 +97,17 @@ The full toolbox of methods that we could possibly support includes: - move-assignment - deinitialization - - type rebinding + - **type rebinding** - bytewise copying Because copy operations (initialization and assignment) don’t mutate the source argument, they can also come in a form which takes a repeated-value source instead of a buffer source. - - initialization (repeated-value) - - assignment (repeated-value) + - **initialization (repeated-value)** + - **assignment (repeated-value)** `UnsafeMutablePointer` and `UnsafeMutableRawPointer` already contain repeated-value methods for initialization in the form of `initialize(to:count:)` and `initializeMemory(as:at:count:to:)`. This proposal will add the assignment analogues. For reasons explained later, the argument label for the repeated-value parameter will be referred to as `repeating:`, not `to:`. -In their most general form, these functions are written like this: - -``` -static -func allocate(as:count:) -> PointerType -func deallocate() - -func initialize(at:as:repeating:count:) -func initialize(at:as:from:count:) -func moveInitialize(at:as:from:count:) - -func assign(at:as:repeating:count:) -func assign(at:as:from:count:) -func moveAssign(at:as:from:count:) - -func deinitialize(at:as:count:) -func rebindMemory(as:count:) - -func copyBytes(at:from:count:) -``` - -where - - - **`as:`** refers to the element type - - **`at:`** refers to an offset from `self`, in strides of the element type, if any - - **`repeating:`** refers to a repeating value - - **`from:`** refers to a second pointer which serves as the **source** - - **`count:`** refers the number of elements the operation operates on - -On actual pointer types, most of these parameters are unnecessary, and some of the methods themselves either don’t make sense to support, or are not practically useful. - - - it only makes sense for immutable pointer types to support deallocation and type rebinding. Note that Swift’s memory model does not require memory to be mutable for deallocation. - - - raw (untyped) pointers should not support any operations which involve deinitialization on `self`. This rules out deinitialization itself, as well as any releasing operations (assignment, move-assignment). - - - typed pointers don’t need an `as:` parameter (except for type rebinding) — they already have a type. It also doesn’t make sense for them to support byte-wise copying. - - - pointers for which it is syntactically easy to offset in strides, or in the case of raw pointers, bytes (for example, by pointer arithmetic with `+`), don’t need to take an `at:` argument. - -This proposal moves the `at:` parameter to the front of the parameter list. (Where this parameter used to appear in `UnsafeMutableRawPointer`, it came after the `as:` parameter.) The rationale for this is that this proposal redefines the `at:` parameter in terms of pointer arithmetic offsets, and pointer arithmetic is written “first” from left to right. Since some of our pointer types will use `at:` and others won’t, we want the offset value to occur roughly in the same reading order across all our pointer types. - -> note: some of these conceptual argument labels have different names in the real API. `as:` is written as `to:` in the type-rebinding methods because it sounds better. `count:` is sometimes written as `capacity:` or `bytes:` to express the assumptions about the stride and initialization state of the memory in question. - -> * `bytes` refers to, well, a byte quantity that is *not assumed* to be initialized. -> * `capacity` refers to a strided quantity that is *not assumed* to be initialized. -> * `count` refers to a strided quantity that is *assumed* to be initialized. - -> note: we don’t bother supporting an `at:` offset in type rebinding operations since we don’t anticipate much use for such a feature. - ### `UnsafePointer` ``` @@ -203,7 +115,9 @@ func deallocate() func withMemoryRebound(to:capacity:_:) -> Result ``` -`UnsafePointer` does not get an allocator static method, since you almost always want a mutable pointer to newly allocated memory. Its type rebinding method is also written as a decorator, taking a trailing closure, for memory safety. `UnsafePointer` does not take `at:` arguments since `+` provides pointer arithmetic for it. +`UnsafePointer` does not get an allocator static method, since you almost always want a mutable pointer to newly allocated memory. Its type rebinding method is also written as a decorator, taking a trailing closure, for memory safety. + +Most immutable pointer types currently do not have a deallocation method. This proposal adds them, fixing [SR-3309](https://bugs.swift.org/browse/SR-3309). Note, immutable raw buffer pointers already support this API. ### `UnsafeMutablePointer` @@ -213,21 +127,16 @@ func allocate(capacity:) -> UnsafeMutablePointer func deallocate() func initialize(repeating:count:) -func initializePointee(to:) -func initialize(from:count:) -func moveInitialize(from:count:) +func initialize(to:) func assign(repeating:count:) -func assign(from:count:) -func moveAssign(from:count:) -func deinitialize(count:) func withMemoryRebound(to:capacity:_:) -> Result ``` -Like `UnsafePointer`, `UnsafeMutablePointer`’s type rebinding method is written as a decorator, and its methods do not need `at:` arguments. +Like `UnsafePointer`, `UnsafeMutablePointer`’s type rebinding method is written as a decorator. -Unlike earlier versions of this proposal, we propose adding a method `initializePointee(to:)` to `UnsafeMutablePointer`. Previously, the single-element initialization case was supported by a default argument of `1` on `initialize(repeating:count:)`’s `count:` parameter, but it was decided this was too confusing in terms of API readability. For example, calls to `initialize(repeating:count:)` and its corresponding method on `UnsafeMutableBufferPointer` were prone to look the same. +Previously, the single-element repeated-initialization case was supported by a default argument of `1` on `initialize(repeating:count:)`’s `count:` parameter, but it was decided this was too confusing in terms of API readability. For example, calls to `initialize(repeating:count:)` and its corresponding method on `UnsafeMutableBufferPointer` were prone to look the same. ```swift plainPointer.initialize(repeating: pointee) @@ -252,69 +161,18 @@ func bindMemory(to:capacity:) -> UnsafePointer ``` static -func allocate(bytes:alignedTo:) -> UnsafeMutableRawPointer +func allocate(byteCount:alignment:) -> UnsafeMutableRawPointer func deallocate() func initializeMemory(as:repeating:count:) -> UnsafeMutablePointer -func initializeMemory(as:from:count:) -> UnsafeMutablePointer -func moveInitializeMemory(as:from:count:) -> UnsafeMutablePointer func bindMemory(to:capacity:) -> UnsafeMutablePointer - -func copyMemory(from:bytes:) ``` -The `as:` argument in `allocate(bytes:alignedTo:)` is represented by an alignment parameter which takes an integer. This is more useful since we often need a computed alignment (like when aligning a structure) instead of a preset type alignment. - -Currently, `UnsafeMutableRawPointer`’s methods take an `at:` offset argument that is interpreted in strides. This argument is not currently in use in the entire Swift standard library, and we believe that it is not useful in practice. This proposal replaces it with a `atByteOffset:` argument which takes a byte offset, a much more useful parameter. Since a byte offset off of a `UnsafeMutableRawPointer` can easily be obtained through pointer arithmetic, we do not actually need such an argument here. +Currently, `UnsafeMutableRawPointer`’s methods take an `at:` offset argument that is interpreted in strides. This argument is not currently in use in the entire Swift standard library, and we believe that it is not useful in practice. Unlike `UnsafeMutablePointer`, we do not add a single-instance initialize method to `UnsafeMutableRawPointer`, as such a method would probably not be useful. However, we still remove the default argument of `1` from the `count:` argument in `initializeMemory(as:repeating:count:)` to prevent confusion with calls to its buffer variant. -------------- - -Buffer pointers are conceptually similar to plain pointers, except the `count:` argument is often unnecessary since they track their own length internally. This means you would call - -```swift -ptr1.initialize(repeating: value, count: count) -``` - -on an `UnsafeMutablePointer`, but - -```swift -buffer1.initialize(repeating: value) -``` - -on an `UnsafeMutableBufferPointer`. - -Implementing unary operations like repeated-value initialization, repeated-value assignment, and type rebinding is straightforward. However, with binary operations like move-initialization, which involves both a source buffer and a destination buffer, the question of whose `count` to use becomes important. - -One option is to use the destination’s `count`, and set the precondition that source`.count` `>=` destination`.count`. The benefit to this is that the destination is always guaranteed to be fully initialized, so that a sizeless `deinitialize()` can be safely called on it. However, in the case of move-initialization and move-assignment, it can leave the source buffer partially *deinitialized* which is just as big a problem. It is also not very useful in practice, since real collections tend to grow monotonically, periodically moving their contents into larger and larger buffers. - -A better option is to use the source’s `count`, combined with an `at:` offset, and set the precondition that `offset` `+` source`.count` `<=` destination`.count`. This *`at:from:`* system inspired by existing `UnsafeMutableRawPointer` APIs is an extremely useful system for supporting partially initialized buffer pointers by allowing us to initialize, assign, and move buffers in segments. For example, it would now be easy to concatenate multiple buffers into one. - -```swift -let pixels:Int = scanlines.map{ $0.count }.reduce(0, +) -var image = UnsafeMutableBufferPointer.allocate(capacity: pixels) - -var filled:Int = 0 -for scanline:UnsafeMutableBufferPointer in scanlines -{ - image.moveInitialize(at: filled, from: scanline) - filled += scanline.count -} - -image.deinitialize(at: 0, count: filled) -image.deallocate() -``` - -Under this system, it will be impossible to leave part of a source buffer deinitialized, and every segment of a destination buffer will be accessible (instead of only segments starting at index `0`.) - -For now, **calling `deallocate()` on a buffer pointer is only defined behavior if the buffer pointer references a complete heap memory block**. This operation may become supported in a wider variety of cases in the future if Swift gets a more sophisticated heap allocation backend. - -> note: we use `at:` instead of `+` because pointer arithmetic does not play well with the nillable buffer pointer `baseAddress`. - -> note: while deinitialization can be performed on a buffer pointer using its own `count` property, we have decided it’s better to explicitly ask for the `at:count:` pair to be consistent with real use patterns and the rest of the proposed API which operates on segments of `self`. - ### `UnsafeBufferPointer` ``` @@ -324,6 +182,8 @@ func withMemoryRebound(to:_:) -> Result The buffer type rebind method dynamically computes the new count by performing multiplication and integer division, since the target type may have a different stride than the original type. This is in line with existing precedent in the generic buffer method `initializeMemory(as:from:)` on `UnsafeMutableRawBufferPointer`. +> Note: **calling `deallocate()` on a buffer pointer is only defined behavior if the buffer pointer references a complete heap memory block**. This operation may become supported in a wider variety of cases in the future if Swift gets a more sophisticated heap allocation backend. + ### `UnsafeMutableBufferPointer` ``` @@ -332,21 +192,14 @@ func allocate(capacity:) -> UnsafeMutableBufferPointer func deallocate() func initialize(repeating:) -func initialize(at:from:) -func moveInitialize(at:from:) func assign(repeating:) -func assign(at:from:) -func moveAssign(at:from:) -func deinitialize(at:count) func withMemoryRebound(to:_:) -> Result ``` The buffer type rebind method works the same way as in `UnsafeBufferPointer`. (Type rebinding never cares about mutability.) -> note: the `at:` arguments in `UnsafeMutableBufferPointer` and `UnsafeMutableRawBufferPointer` should *not* receive default values, as they are an integral part of the buffer pointer memory state safety system, and so it is important they appear at the call site. - ### `UnsafeRawBufferPointer` ``` @@ -359,57 +212,29 @@ func bindMemory(to:) -> UnsafeBufferPointer ``` static -func allocate(bytes:alignedTo:) -> UnsafeMutableRawBufferPointer +func allocate(byteCount:alignment:) -> UnsafeMutableRawBufferPointer func deallocate() func initializeMemory(as:repeating:) -> UnsafeMutableBufferPointer -func initializeMemory(atByteOffset:as:from:) -> UnsafeMutableBufferPointer -func moveInitializeMemory(atByteOffset:as:from:) -> UnsafeMutableBufferPointer func bindMemory(to:) -> UnsafeMutableBufferPointer - -func copyMemory(at:from:) ``` > note: `initializeMemory(as:repeating:)` performs integer division on `self.count` (just like `bindMemory(to:)`) -> note: the return values of `initializeMemory(as:repeating:)`, `initializeMemory(as:at:from:)`, and `moveInitializeMemory(as:at:from:)` should all be marked as `@discardableResult`. - -> note: even though the `at:` argument in `copyMemory(at:from:)` is in terms of bytes, it is not written as `atByteOffset` since there is no type object parameter in the function signature that could suggest that the offset is in typed strides. - -## Detailed changes +> note: the return value of `initializeMemory(as:repeating:)` should be marked as `@discardableResult`. -The proposed new API attempts to build on the existing API wherever possible. With the exception of `deallocate()` (which has good justification to replace `deallocate(capacity:)` and `deallocate(bytes:alignedTo:)`), all changes are either pure additive changes, or renames which are trivial to automigrate. This reduces the amount of source breakage. +We also make several miscellaneous changes to the API in order to tidy things up. -- **fix the ordering of the arguments in `initializeMemory(as:at:count:to:)` and rename the argument `to:` to `repeating:` in all repeated-value copy functions** - -The ordering of the `to:` and `count:` argument labels in the `initializeMemory(as:at:count:to:)` method on `UnsafeMutableRawPointer` contradicts the rest of the Swift pointer API, where `to:` precedes `count:`. - -Because the ordering `initializeMemory(as:at:to:count:)` conflicts with the use of `to:` as the argument label for a target type, this argument should be renamed to `repeating:`. The word `repeating:` is much more clear in terms of describing the methods’ behavior, and is consistent with the use of the word in the `Array` API. - -- **add the repeated-value copy assignment method `assign(repeating:count:)`** - -This addresses the missing assignment analogue to the `initialize(to:count:)` method. - -- **rename `copyBytes(from:count:)` and `copyBytes(from:)` to `copyMemory(from:bytes:)` and `copyMemory(at:from:)`** +- **rename `copyBytes(from:count:)` and `copyBytes(from:)` to `copyMemory(from:byteCount:)` and `copyMemory(from:)`** This brings the method names in line with the rest of the raw pointer API. -> note: we do not change the `copyBytes(from:)` collection method. - -- **rename `count` in `UnsafeMutableRawBufferPointer.allocate(count:)` to `bytes` and add an `alignedTo` parameter to make it `UnsafeMutableRawBufferPointer.allocate(bytes:alignedTo:)`** - -This brings it in line with the `UnsafeMutableRawPointer` allocator, and avoids the contradictory and inconsistent use of `count` to represent a byte quantity. Currently `UnsafeMutableRawBufferPointer.allocate(count:)` aligns to the size of `UInt`, an assumption not shared by its plain variant. - - **add an `init(mutating:)` initializer to `UnsafeMutableBufferPointer`** This makes it much easier to make a mutable copy of an immutable buffer pointer. Such an initializer already exists on `UnsafeMutableRawBufferPointer`, so adding one to `UnsafeMutableBufferPointer` is also necessary for consistency. The reverse initializer, from `UnsafeMutableBufferPointer` to `UnsafeBufferPointer` should also be added for completeness. -- **add a mutable overload to the `copyMemory(at:from:)` method on `UnsafeMutableRawBufferPointer`, the `initialize(at:from:)` and `assign(at:from:)` methods on `UnsafeMutableBufferPointer`, and the `initializeMemory(atByteOffset:as:from:)` method on `UnsafeMutableRawBufferPointer`** - -Currently, for plain pointers, there is a compiler subtyping relationship between `UnsafePointer` and `UnsafeMutablePointer`. No such relationship exists between `UnsafeBufferPointer` and `UnsafeMutableBufferPointer` or their raw counterparts, so it is necessary to provide mutable overloads for these functions. - -- **add `deallocate()` to all pointer types, replacing any existing deallocation methods** +- **deprecate the sized deallocation API** Removing `capacity` from `deallocate(capacity:)` will end the confusion over what `deallocate()` does, making it obvious that `deallocate()` will free the *entire* memory block at `self`, just as if `free()` were called on it. @@ -417,7 +242,7 @@ The old `deallocate(capacity:)` method should be marked as `deprecated` and even Along similar lines, the `bytes` and `alignedTo` parameters should be removed from the `deallocate(bytes:alignedTo:)` method on `UnsafeMutableRawPointer` and `UnsafeRawPointer`. -An unsized `deallocate()` method should be added to all pointer types, even immutable ones, as Swift’s memory model does not require memory to be mutable for deallocation. This fixes [SR-3309](https://bugs.swift.org/browse/SR-3309). Note, immutable raw buffer pointers already support this API. +As discussed earlier, an unsized `deallocate()` method should be added to all pointer types, even immutable ones, as Swift’s memory model does not require memory to be mutable for deallocation. > note: the deallocation size parameters were originally included in early versions of Swift in order to support a more sophisticated hypothetical heap allocator backend that we wanted to have in the future. (Swift currently calls `malloc(_:)` and `free()`.) While such a backend would theoretically run more efficiently than the C backend, overengineering Swift to support it in the future has proven to be a detriment to users right now. By removing the size parameters now, we make it easier and safer to reintroduce such an API in the future without inadvertently causing silent source breakage. @@ -431,6 +256,12 @@ An unsized `deallocate()` method should be added to all pointer types, even immu func withMemoryRebound(to:capacity:_:) -> Result ``` +#### New methods + +```diff ++++ func deallocate() +``` + ### `UnsafeMutablePointer` #### Existing methods @@ -439,13 +270,6 @@ func withMemoryRebound(to:capacity:_:) -> Result static func allocate(capacity:) -> UnsafeMutablePointer -func initialize(from:count:) -func moveInitialize(from:count:) - -func assign(from:count:) -func moveAssign(from:count:) - -func deinitialize(count:) func withMemoryRebound(to:capacity:_:) -> Result ``` @@ -459,7 +283,9 @@ func withMemoryRebound(to:capacity:_:) -> Result #### New methods ```diff -+++ func initializePointee(to:) ++++ func deallocate() + ++++ func initialize(to:) +++ func assign(repeating:count:) ``` @@ -471,32 +297,42 @@ func withMemoryRebound(to:capacity:_:) -> Result func bindMemory(to:capacity:) -> UnsafePointer ``` +#### New methods + +```diff ++++ func deallocate() +``` + ### `UnsafeMutableRawPointer` #### Existing methods ``` -static -func allocate(bytes:alignedTo:) -> UnsafeMutableRawPointer +func bindMemory(to:capacity:) -> UnsafeMutablePointer +``` -func initializeMemory(as:from:count:) -> UnsafeMutablePointer -func moveInitializeMemory(as:from:count:) -> UnsafeMutablePointer +#### New methods -func bindMemory(to:capacity:) -> UnsafeMutablePointer +```diff ++++ func deallocate() ``` #### Renamed methods and dropped arguments ```diff +--- static +--- func allocate(bytes:alignedTo:) -> UnsafeMutableRawPointer + ++++ static ++++ func allocate(byteCount:alignment:) -> UnsafeMutableRawPointer + --- func initializeMemory(as:at:count:to:) -> UnsafeMutablePointer +++ func initializeMemory(as:repeating:count:) -> UnsafeMutablePointer --- func copyBytes(from:count:) -+++ func copyMemory(from:bytes:) ++++ func copyMemory(from:byteCount:) ``` -> note: We are adding a *new* default argument of `MemoryLayout.alignment` for the `alignment` parameter in `allocate(bytes:alignedTo:)`. The rationale is that Swift is introducing a language-level default guarantee of word-aligned storage, so the default argument reflects Swift’s memory model. Higher alignments (such as 16-byte alignment) should be specified explicitly by the user. - ### `UnsafeBufferPointer` #### New methods @@ -517,14 +353,8 @@ func bindMemory(to:capacity:) -> UnsafeMutablePointer +++ func deallocate() +++ func initialize(repeating:) -+++ func initialize(at:from:) -+++ func moveInitialize(at:from:) - +++ func assign(repeating:) -+++ func assign(at:from:) -+++ func moveAssign(at:from:) -+++ func deinitialize(at:count:) +++ func withMemoryRebound(to:_:) -> Result ``` @@ -556,38 +386,25 @@ deallocate() --- static --- func allocate(count:) -> UnsafeMutableRawBufferPointer +++ static -+++ func allocate(bytes:alignedTo:) -> UnsafeMutableRawBufferPointer ++++ func allocate(byteCount:alignment:) -> UnsafeMutableRawBufferPointer --- func copyBytes(from:) -+++ func copyMemory(at:from:) ++++ func copyMemory(from:) ``` #### New methods ```diff +++ func initializeMemory(as:repeating:) -> UnsafeMutableBufferPointer -+++ func initializeMemory(atByteOffset:as:from:) -> UnsafeMutableBufferPointer -+++ func moveInitializeMemory(atByteOffset:as:from:) -> UnsafeMutableBufferPointer +++ func bindMemory(to:) -> UnsafeMutableBufferPointer ``` -> note: for backwards compatibility, the `alignedTo:` argument in `allocate(bytes:alignedTo:)` should take a default value of `MemoryLayout.alignment`. This requires [SR-5664](https://bugs.swift.org/browse/SR-5664) to be fixed before it will work properly. - -> note: The new `at:` argument in `copyMemory(at:from:)` has a backwards-compatible default argument of `0`. This poses no risk to memory state safety, since this method can only perform a bytewise copy anyways. - ## What this proposal does not do -- **attempt to fully support partial initialization** - -This proposal attempts to design a buffer interface that provides some semblance of memory state safety. However, it does not fully address issues relating to ergonomics such as - - - overloading `+` for buffer pointers, allowing “pointer arithmetic” to be performed on buffer pointers - - easier buffer pointer slicing, which does not produce wasteful `MutableRandomAccessSlice>` structures - -nor does it attempt to design a higher level buffer type which would be able to provide stronger memory state guarantees. +- **attempt to fully partial initialization** -We expect possible solutions to these problems to purely additive, and would not require modifying the methods this proposal will introduce. +This proposal does not attempt to fill in most of the memory state APIs for buffer pointers, as doing so necessitates designing a partial initialization system, as well as a possible buffer slice rework. - **address problems relating to the generic `Sequence` buffer API** @@ -613,9 +430,9 @@ struct UnsafeMutablePointer --- func initialize(to:Pointee, count:Int = 1) +++ func initialize(repeating:Pointee, count:Int) -+++ func initializePointee(to:Pointee) ++++ func initialize(to:Pointee) func initialize(from:UnsafePointer, count:Int) - moveInitialize(from:UnsafeMutablePointer, count:Int) + func moveInitialize(from:UnsafeMutablePointer, count:Int) +++ func assign(repeating:Pointee, count:Int) func assign(from:UnsafePointer, count:Int) @@ -640,8 +457,7 @@ struct UnsafeMutableRawPointer --- static --- func allocate(bytes:Int, alignedTo:Int) -> UnsafeMutableRawPointer +++ static -+++ func allocate(bytes:Int, alignedTo:Int = MemoryLayout.alignment) -+++ -> UnsafeMutableRawPointer ++++ func allocate(byteCount:Int, alignment:Int) -> UnsafeMutableRawPointer --- func deallocate(bytes:Int, alignedTo:Int) +++ func deallocate() @@ -655,7 +471,7 @@ struct UnsafeMutableRawPointer func bindMemory(to:T.Type, count:Int) -> UnsafeMutablePointer --- func copyBytes(from:UnsafeRawPointer, count:Int) -+++ func copyMemory(from:UnsafeRawPointer, bytes:Int) ++++ func copyMemory(from:UnsafeRawPointer, byteCount:Int) } struct UnsafeBufferPointer @@ -676,16 +492,7 @@ struct UnsafeMutableBufferPointer +++ func allocate(capacity:Int) -> UnsafeMutableBufferPointer +++ func initialize(repeating:Element) -+++ func initialize(at:Int, from:UnsafeBufferPointer) -+++ func initialize(at:Int, from:UnsafeMutableBufferPointer) -+++ func moveInitialize(at:Int, from:UnsafeMutableBufferPointer) - +++ func assign(repeating:Element) -+++ func assign(at:Int, from:UnsafeBufferPointer) -+++ func assign(at:Int, from:UnsafeMutableBufferPointer) -+++ func moveAssign(at:Int, from:UnsafeMutableBufferPointer) - -+++ func deinitialize(at:Int, count:Int) +++ func withMemoryRebound +++ (to:T.Type, _ body:(UnsafeMutableBufferPointer) -> Result) @@ -701,56 +508,55 @@ struct UnsafeRawBufferPointer struct UnsafeMutableRawBufferPointer { --- static func allocate(count:Int) -> UnsafeMutableRawBufferPointer -+++ static func allocate(bytes:Int, alignedTo:Int = MemoryLayout.alignment) -+++ -> UnsafeMutableRawBufferPointer ++++ static func allocate(byteCount:Int, alignment:Int) -> UnsafeMutableRawBufferPointer func deallocate() +++ func initializeMemory(as:T.Type, repeating:T) -> UnsafeMutableBufferPointer -+++ func initializeMemory(atByteOffset:Int, as:T.Type, from:UnsafeBufferPointer) -+++ -> UnsafeMutableBufferPointer -+++ func initializeMemory(atByteOffset:Int, as:T.Type, from:UnsafeMutableBufferPointer) -+++ -> UnsafeMutableBufferPointer -+++ func moveInitializeMemory(atByteOffset:Int, as:T.Type, from:UnsafeMutableBufferPointer) -+++ -> UnsafeMutableBufferPointer +++ func bindMemory(to:T.Type) -> UnsafeMutableBufferPointer --- func copyBytes(from:UnsafeRawBufferPointer) -+++ func copyMemory(at:Int = 0, from:UnsafeRawBufferPointer) -+++ func copyMemory(at:Int = 0, from:UnsafeMutableRawBufferPointer) ++++ func copyMemory(from:UnsafeRawBufferPointer) } ``` ## Source compatibility -- **fix the ordering of the arguments in `initializeMemory(as:at:count:to:)` and rename the argument `to:` to `repeating:` in all repeated-value copy functions** +Everything is additive except the following. Can we deprecate all of +the original functions in Swift 5, then drop those from the binary +later in Swift 6? -This change is source breaking but can be trivially automigrated. +- **add `deallocate()` to all pointer types, replacing any existing deallocation methods** -- **add the repeated-value copy assignment method `assign(repeating:count:)`** +The migrator needs to drop the existing `capacity` and `alignedTo` arguments. -This change is purely additive. +- **in `UnsafeMutableRawPointer.allocate(count:alignedTo:)` rename `count` to `byteCount` and `alignedTo` to `alignment`** -- **rename `copyBytes(from:count:)` and `copyBytes(from:)` to `copyMemory(from:bytes:)` and `copyMemory(at:from:)`** +- **in `UnsafeMutableRawBufferPointer.allocate(count:)` rename `count` to `byteCount` and add an `alignment` parameter** -This change is source breaking but can be trivially automigrated. +This change is source breaking but can be trivially automigrated. The +`alignment:` parameter can be filled in with `MemoryLayout.stride`. -- **rename `count` in `UnsafeMutableRawBufferPointer.allocate(count:)` to `bytes` and add an `alignedTo` parameter to make it `UnsafeMutableRawBufferPointer.allocate(bytes:alignedTo:)`** +- **fix the arguments to `initialize(repeating:Pointee, count:Int)`** -This change is source breaking but can be trivially automigrated. The `alignedTo:` parameter can be filled in with `MemoryLayout.stride`. If [SR-5664](https://bugs.swift.org/browse/SR-5664) is fixed, `MemoryLayout.stride` can even be provided as a default argument. +Note: initialize(to:Pointee) is backward compatible whenever the +caller relied on a default `count = 1`. -- **add an `init(mutating:)` initializer to `UnsafeMutableBufferPointer`** +An annotation could otherwise rename `to` to `repeating`, but we don't +want that to interfere with the default count case, so this might need to be a migrator rule. -This change is purely additive. +- **fix the ordering of the arguments in `initializeMemory(as:at:count:to:)`, rename `to:` to `repeating:`, and remove the `at:` argument** -- **add a mutable overload to the `copyMemory(at:from:)` method on `UnsafeMutableRawBufferPointer`, the `initialize(at:from:)` and `assign(at:from:)` methods on `UnsafeMutableBufferPointer`, and the `initializeMemory(atByteOffset:as:from:)` method on `UnsafeMutableRawBufferPointer`** +This change is source breaking but can be trivially automigrated. The +`to` argument changes position and is relabeled as `repeating`. -This change is purely additive. +The migrator could be taught to convert the `at:` argument into +pointer arithmetic on `self`. However, we found no code on Github that +uses the `at:` argument, so it is low priority. -- **add `deallocate()` to all pointer types, replacing any existing deallocation methods** - -This change is source-breaking, but this is a Good Thing™. The current API encourages incorrect code to be written, and sets us up for potentially catastrophic source breakage down the road should the implementations of `deallocate(capacity:)` and `deallocate(bytes:alignedTo:)` ever be “fixed”, so users should be forced to stop using them as soon as possible. +- **rename `copyBytes(from:count:)` to `copyMemory(from:byteCount:)`** +This change is source breaking but can be trivially automigrated. ## Effect on ABI stability @@ -762,8 +568,6 @@ Some proposed changes in this proposal change the public API. Removing sized deallocators right now will break ABI, but offers increased ABI and API stability in the future as reallocator methods can be added in the future without having to rename `deallocate(capacity:)` which currently occupies a “reallocator” name, but has “`free()`” behavior. -This proposal seeks to tackle all the breaking changes required for such an overhaul of Swift pointers, and leaves unanswered only additive changes that still need to be made in the future, reducing the need for future breakage. - ## Alternatives considered - **keeping sized deallocators and fixing the stdlib implementation instead** From 6af5822fcf3a8af12de029bbec8e7ccdd92bb730 Mon Sep 17 00:00:00 2001 From: Kelvin Date: Wed, 4 Oct 2017 17:50:58 -0500 Subject: [PATCH 7/7] merged into se 1084 primary document --- proposals/0184a-unsafe-pointers-part-1.md | 587 ---------------------- 1 file changed, 587 deletions(-) delete mode 100644 proposals/0184a-unsafe-pointers-part-1.md diff --git a/proposals/0184a-unsafe-pointers-part-1.md b/proposals/0184a-unsafe-pointers-part-1.md deleted file mode 100644 index db7eb40db5..0000000000 --- a/proposals/0184a-unsafe-pointers-part-1.md +++ /dev/null @@ -1,587 +0,0 @@ -# Unsafe[Mutable][Raw][Buffer]Pointer: add missing methods, adjust existing labels for clarity, and remove deallocation size - -* Proposal: [SE-0184 A](0184a-unsafe-pointers-part-1.md) -* Author: [Kelvin Ma (“Taylor Swift”)](https://github.com/kelvin13) -* Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: -* Implementation: [PR 12200](https://github.com/apple/swift/pull/12200) - -## Introduction - -*This document is a spin-off from a larger [original proposal](0184-unsafe-pointers-add-missing.md), SE-0184, which covers only those aspects of SE-1084 which do not deal with partial buffer memory state. Designing the partial buffer memory state API clearly requires more work, and has been left out of the scope of this document.* - -Swift’s pointer types are an important interface for low-level memory manipulation, but the current API design is not very consistent, complete, or convenient. In some places, poor naming choices and overengineered function signatures compromise memory safety by leading users to believe that they have allocated or freed memory when in fact, they have not. This proposal seeks to improve the Swift pointer API by ironing out naming inconsistencies, adding missing methods, and reducing excessive verbosity, offering a more convenient, more sensible, and less bug-prone API. - -Swift-evolution threads: [Pitch: Improved Swift pointers](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170710/038013.html), [Pitch: More Improved Swift pointers](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170717/038121.html) - -Implementation branch: [`kelvin13:se-0184a`](https://github.com/kelvin13/swift/tree/se-0184a) - -## Background - -There are four binary memorystate operations: *initialization*, *move-initialization*, *assignment*, and *move-assignment*, and two unary memorystate operations: *deinitialization* and *type rebinding*. The binary operations can be grouped according to how they affect the source buffer and the destination buffer. **Copy** operations only read from the source buffer, leaving it unchanged. **Move** operations deinitialize the source memory, decrementing the reference count by 1 if the memory type is not a trivial type. **Retaining** operations initialize the destination memory, incrementing the reference count by 1 if applicable. **Releasing** operations deinitialize the destination memory before reinitializing it with the new values, resulting in a net change in the reference count of 0, if applicable. - -| | Copy (+0) | Move (−1) | -| -------------: |----------: | ---------: | -| **Retaining (+1)** | initialize | move-initialize | -| **Releasing (+0)** | assign | move-assign | - -Raw pointers also have a unique operation, *bytewise-copying*, which we will lump together with the memorystate functions, but does not actually change a pointer’s memory state. - -Most of these operations become more relevant in the discussion of partial buffer memory state, which is not in the scope of this document. This document only proposes changes related to memory allocation, type-rebinding, and two special *unary* forms of initialization and assignment which initialize memory to a fixed, repeating value. - -## Motivation - -Right now, `UnsafeMutableBufferPointer` is kind of a black box when it comes to producing and modifying instances of it. Much of the API present on `UnsafeMutablePointer` is absent on its buffer variant. To create, bind, allocate, initialize, and deallocate them, you have to extract `baseAddress`es and `count`s. This is unfortunate because `UnsafeMutableBufferPointer` provides a handy container for tracking the size of a memory buffer, but to actually make use of this information, the buffer pointer must be disassembled. In practice, this means the use of memory buffers requires frequent (and annoying) conversion back and forth between buffer pointers and base address–count pairs. For example, buffer allocation requires the creation of a temporary `UnsafeMutablePointer` instance. This means that the following “idiom” is very common in Swift code: - -```swift -let buffer = UnsafeMutableBufferPointer(start: UnsafeMutablePointer.allocate(capacity: byteCount), count: byteCount) -``` - -Aside from being extremely long and unwieldy, and requiring the creation of a temporary, `byteCount` must appear twice. - -You can’t even cast buffer pointer types to their mutable or immutable forms without creating a temporary. - -```swift - -var mutableBuffer = UnsafeMutableBufferPointer(start: UnsafeMutablePointer(mutating: immutableBuffer.baseAddress!), count: immutableBuffer.count) -``` - -Currently, memory is deallocated by an instance method on `UnsafeMutablePointer`, `deallocate(count:)`. Like much of the Swift pointer API, performing this operation on a buffer pointer requires extracting `baseAddress!` and `count`. It is very common for the allocation code above to be immediately followed by: - -```swift -defer -{ - buffer.baseAddress?.deallocate(capacity: buffer.count) -} -``` - -The `?` is sometimes exchanged with an `!` depending on the personality of the author, as normally, neither operator is meaningful here — the `baseAddress` is never `nil` if the buffer pointer was created around an instance of `UnsafeMutablePointer`. - -This method is extremely problematic because nearly all users, on first seeing the signature of `deallocate(capacity:)`, will naturally conclude from the `capacity` label that `deallocate(capacity:)` is equivalent to some kind of `realloc()` that can only shrink the buffer. However this is not the actual behavior — `deallocate(capacity:)` actually *ignores* the `capacity` argument and just calls `free()` on `self`. The current API is not only awkward and suboptimal, it is *misleading*. You can write perfectly legal Swift code that shouldn’t segfault, but still can, for example - -```swift -var ptr = UnsafeMutablePointer.allocate(capacity: 1000000) -ptr.initialize(to: 13, count: 1000000) -ptr.deallocate(capacity: 500000) // deallocate the second half of the memory block -ptr[0] // segmentation fault -``` - -where the first 500000 addresses should still be valid if the [documentation](https://developer.apple.com/documentation/swift/unsafemutablepointer/2295090-deallocate) is to be read literally. - -Users who are *aware* of this behavior may also choose to disregard the `capacity` argument and write things like this: - -```swift -defer -{ - buffer.baseAddress?.deallocate(capacity: 42) -} -``` - -which is functionally equivalent. However this will lead to disastrous source breakage if the implementation of `deallocate(capacity:)` is ever “corrected”. Since the API would not change, such code would still compile, but suddenly start failing at runtime. Thus, the current API, combined with incorrect documentation, is serving as a vector for introducing memory bugs into Swift code. - -Finally, some of the naming choices in the current API deserve a second look. While the original API intended to introduce a naming convention where `bytes` refers to uninitialized memory, `capacity` to uninitialized elements, and `count` to initialized elements, the actual usage of the three words does not always agree. In `copyBytes(from:count:)`, `count` refers to the number of *bytes*, which may or may not be initialized. Similarly, the `UnsafeMutableRawBufferPointer` `allocate(count:)` type method includes a `count` argument which actually refers to uninitialized bytes. The argument label `to:` is also excessively overloaded; sometimes it refers to a type `T.Type`, and sometimes it refers to a repeated value parameter. This becomes problematic when both parameters appear in the same method, as in `initializeMemory(as:at:count:to)`. - -## Proposed solution - -The ultimate goal of the API redesign is to bring all of the functionality in `UnsafeMutablePointer` and `UnsafeMutableRawPointer` to their buffer types, `UnsafeMutableBufferPointer` and `UnsafeMutableRawBufferPointer`. Operations which are covered by this proposal are in **bold**. - -The full toolbox of methods that we could possibly support includes: - - - **allocation** - - **deallocation** - - - initialization - - move-initialization - - - assignment - - move-assignment - - - deinitialization - - - **type rebinding** - - bytewise copying - -Because copy operations (initialization and assignment) don’t mutate the source argument, they can also come in a form which takes a repeated-value source instead of a buffer source. - - - **initialization (repeated-value)** - - **assignment (repeated-value)** - -`UnsafeMutablePointer` and `UnsafeMutableRawPointer` already contain repeated-value methods for initialization in the form of `initialize(to:count:)` and `initializeMemory(as:at:count:to:)`. This proposal will add the assignment analogues. For reasons explained later, the argument label for the repeated-value parameter will be referred to as `repeating:`, not `to:`. - -### `UnsafePointer` - -``` -func deallocate() -func withMemoryRebound(to:capacity:_:) -> Result -``` - -`UnsafePointer` does not get an allocator static method, since you almost always want a mutable pointer to newly allocated memory. Its type rebinding method is also written as a decorator, taking a trailing closure, for memory safety. - -Most immutable pointer types currently do not have a deallocation method. This proposal adds them, fixing [SR-3309](https://bugs.swift.org/browse/SR-3309). Note, immutable raw buffer pointers already support this API. - -### `UnsafeMutablePointer` - -``` -static -func allocate(capacity:) -> UnsafeMutablePointer -func deallocate() - -func initialize(repeating:count:) -func initialize(to:) - -func assign(repeating:count:) - -func withMemoryRebound(to:capacity:_:) -> Result -``` - -Like `UnsafePointer`, `UnsafeMutablePointer`’s type rebinding method is written as a decorator. - -Previously, the single-element repeated-initialization case was supported by a default argument of `1` on `initialize(repeating:count:)`’s `count:` parameter, but it was decided this was too confusing in terms of API readability. For example, calls to `initialize(repeating:count:)` and its corresponding method on `UnsafeMutableBufferPointer` were prone to look the same. - -```swift -plainPointer.initialize(repeating: pointee) -bufferPointer.initialize(repeating: repeatedValue) -``` - -Increasing API surface by adding this method is justified by the large number of calls to `initialize(to:count:)` in the standard library (and likely other code) which rely on the default argument of `1`. We do *not* need to add a corresponding `assignPointee(to:)` method since this can be done with the assignment operator. - -```swift -ptr.pointee = newValue -``` - -### `UnsafeRawPointer` - -``` -func deallocate() - -func bindMemory(to:capacity:) -> UnsafePointer -``` - -### `UnsafeMutableRawPointer` - -``` -static -func allocate(byteCount:alignment:) -> UnsafeMutableRawPointer -func deallocate() - -func initializeMemory(as:repeating:count:) -> UnsafeMutablePointer - -func bindMemory(to:capacity:) -> UnsafeMutablePointer -``` - -Currently, `UnsafeMutableRawPointer`’s methods take an `at:` offset argument that is interpreted in strides. This argument is not currently in use in the entire Swift standard library, and we believe that it is not useful in practice. - -Unlike `UnsafeMutablePointer`, we do not add a single-instance initialize method to `UnsafeMutableRawPointer`, as such a method would probably not be useful. However, we still remove the default argument of `1` from the `count:` argument in `initializeMemory(as:repeating:count:)` to prevent confusion with calls to its buffer variant. - -### `UnsafeBufferPointer` - -``` -func deallocate() -func withMemoryRebound(to:_:) -> Result -``` - -The buffer type rebind method dynamically computes the new count by performing multiplication and integer division, since the target type may have a different stride than the original type. This is in line with existing precedent in the generic buffer method `initializeMemory(as:from:)` on `UnsafeMutableRawBufferPointer`. - -> Note: **calling `deallocate()` on a buffer pointer is only defined behavior if the buffer pointer references a complete heap memory block**. This operation may become supported in a wider variety of cases in the future if Swift gets a more sophisticated heap allocation backend. - -### `UnsafeMutableBufferPointer` - -``` -static -func allocate(capacity:) -> UnsafeMutableBufferPointer -func deallocate() - -func initialize(repeating:) - -func assign(repeating:) - -func withMemoryRebound(to:_:) -> Result -``` - -The buffer type rebind method works the same way as in `UnsafeBufferPointer`. (Type rebinding never cares about mutability.) - -### `UnsafeRawBufferPointer` - -``` -func deallocate() - -func bindMemory(to:) -> UnsafeBufferPointer -``` - -### `UnsafeMutableRawBufferPointer` - -``` -static -func allocate(byteCount:alignment:) -> UnsafeMutableRawBufferPointer -func deallocate() - -func initializeMemory(as:repeating:) -> UnsafeMutableBufferPointer - -func bindMemory(to:) -> UnsafeMutableBufferPointer -``` - -> note: `initializeMemory(as:repeating:)` performs integer division on `self.count` (just like `bindMemory(to:)`) - -> note: the return value of `initializeMemory(as:repeating:)` should be marked as `@discardableResult`. - -We also make several miscellaneous changes to the API in order to tidy things up. - -- **rename `copyBytes(from:count:)` and `copyBytes(from:)` to `copyMemory(from:byteCount:)` and `copyMemory(from:)`** - -This brings the method names in line with the rest of the raw pointer API. - -- **add an `init(mutating:)` initializer to `UnsafeMutableBufferPointer`** - -This makes it much easier to make a mutable copy of an immutable buffer pointer. Such an initializer already exists on `UnsafeMutableRawBufferPointer`, so adding one to `UnsafeMutableBufferPointer` is also necessary for consistency. The reverse initializer, from `UnsafeMutableBufferPointer` to `UnsafeBufferPointer` should also be added for completeness. - -- **deprecate the sized deallocation API** - -Removing `capacity` from `deallocate(capacity:)` will end the confusion over what `deallocate()` does, making it obvious that `deallocate()` will free the *entire* memory block at `self`, just as if `free()` were called on it. - -The old `deallocate(capacity:)` method should be marked as `deprecated` and eventually removed since it currently encourages dangerously incorrect code. This avoids misleading future users, encourages current users to address this potentially catastrophic memory bug, and leaves the possibility open for us to add a `deallocate(capacity:)` method in the future, or perhaps even a `reallocate(toCapacity:)` method. - -Along similar lines, the `bytes` and `alignedTo` parameters should be removed from the `deallocate(bytes:alignedTo:)` method on `UnsafeMutableRawPointer` and `UnsafeRawPointer`. - -As discussed earlier, an unsized `deallocate()` method should be added to all pointer types, even immutable ones, as Swift’s memory model does not require memory to be mutable for deallocation. - -> note: the deallocation size parameters were originally included in early versions of Swift in order to support a more sophisticated hypothetical heap allocator backend that we wanted to have in the future. (Swift currently calls `malloc(_:)` and `free()`.) While such a backend would theoretically run more efficiently than the C backend, overengineering Swift to support it in the future has proven to be a detriment to users right now. By removing the size parameters now, we make it easier and safer to reintroduce such an API in the future without inadvertently causing silent source breakage. - -> note: changes to deallocation methods are not listed in the type-by-type overview below. All items in the following list are either non-source breaking, or trivially automigratable. - -### `UnsafePointer` - -#### Existing methods - -``` -func withMemoryRebound(to:capacity:_:) -> Result -``` - -#### New methods - -```diff -+++ func deallocate() -``` - -### `UnsafeMutablePointer` - -#### Existing methods - -``` -static -func allocate(capacity:) -> UnsafeMutablePointer - -func withMemoryRebound(to:capacity:_:) -> Result -``` - -#### Renamed methods - -```diff ---- func initialize(to:count:) -+++ func initialize(repeating:count:) -``` - -#### New methods - -```diff -+++ func deallocate() - -+++ func initialize(to:) -+++ func assign(repeating:count:) -``` - -### `UnsafeRawPointer` - -#### Existing methods - -``` -func bindMemory(to:capacity:) -> UnsafePointer -``` - -#### New methods - -```diff -+++ func deallocate() -``` - -### `UnsafeMutableRawPointer` - -#### Existing methods - -``` -func bindMemory(to:capacity:) -> UnsafeMutablePointer -``` - -#### New methods - -```diff -+++ func deallocate() -``` - -#### Renamed methods and dropped arguments - -```diff ---- static ---- func allocate(bytes:alignedTo:) -> UnsafeMutableRawPointer - -+++ static -+++ func allocate(byteCount:alignment:) -> UnsafeMutableRawPointer - ---- func initializeMemory(as:at:count:to:) -> UnsafeMutablePointer -+++ func initializeMemory(as:repeating:count:) -> UnsafeMutablePointer - ---- func copyBytes(from:count:) -+++ func copyMemory(from:byteCount:) -``` - -### `UnsafeBufferPointer` - -#### New methods - -```diff -+++ func deallocate() - -+++ withMemoryRebound(to:_:) -> Result -``` - -### `UnsafeMutableBufferPointer` - -#### New methods - -```diff -+++ static -+++ func allocate(capacity:) -> UnsafeMutableBufferPointer -+++ func deallocate() - -+++ func initialize(repeating:) -+++ func assign(repeating:) - -+++ func withMemoryRebound(to:_:) -> Result -``` - -### `UnsafeRawBufferPointer` - -#### Existing methods - -``` -deallocate() -``` - -#### New methods - -```diff -+++ func bindMemory(to:) -> UnsafeBufferPointer -``` - -### `UnsafeMutableRawBufferPointer` - -#### Existing methods - -``` -deallocate() -``` - -#### Renamed methods and new/renamed arguments - -```diff ---- static ---- func allocate(count:) -> UnsafeMutableRawBufferPointer -+++ static -+++ func allocate(byteCount:alignment:) -> UnsafeMutableRawBufferPointer - ---- func copyBytes(from:) -+++ func copyMemory(from:) -``` - -#### New methods - -```diff -+++ func initializeMemory(as:repeating:) -> UnsafeMutableBufferPointer - -+++ func bindMemory(to:) -> UnsafeMutableBufferPointer -``` - -## What this proposal does not do - -- **attempt to fully partial initialization** - -This proposal does not attempt to fill in most of the memory state APIs for buffer pointers, as doing so necessitates designing a partial initialization system, as well as a possible buffer slice rework. - -- **address problems relating to the generic `Sequence` buffer API** - -Buffer pointers are currently missing generic `assign(from:S)` and `initializeMemory(as:S.Element.Type, from:S)` methods. The existing protocol oriented API also lacks polish and is inconvenient to use. (For example, it returns tuples.) This is an issue that can be tackled separately from the lower-level buffer-pointer-to-buffer-pointer API. - -## Detailed design - -```diff -struct UnsafePointer -{ -+++ func deallocate() - - func withMemoryRebound(to:T.Type, capacity:Int, _ body:(UnsafePointer) -> Result) - -> Result -} - -struct UnsafeMutablePointer -{ - static func allocate(capacity:Int) -> UnsafeMutablePointer - ---- func deallocate(capacity:Int) -+++ func deallocate() - ---- func initialize(to:Pointee, count:Int = 1) -+++ func initialize(repeating:Pointee, count:Int) -+++ func initialize(to:Pointee) - func initialize(from:UnsafePointer, count:Int) - func moveInitialize(from:UnsafeMutablePointer, count:Int) - -+++ func assign(repeating:Pointee, count:Int) - func assign(from:UnsafePointer, count:Int) - func moveAssign(from:UnsafeMutablePointer, count:Int) - - func deinitialize(count:Int) - - func withMemoryRebound(to:T.Type, capacity:Int, _ body:(UnsafeMutablePointer) -> Result) - -> Result -} - -struct UnsafeRawPointer -{ ---- func deallocate(bytes:Int, alignedTo:Int) -+++ func deallocate() - - func bindMemory(to:T.Type, count:Int) -> UnsafeMutablePointer -} - -struct UnsafeMutableRawPointer -{ ---- static ---- func allocate(bytes:Int, alignedTo:Int) -> UnsafeMutableRawPointer -+++ static -+++ func allocate(byteCount:Int, alignment:Int) -> UnsafeMutableRawPointer ---- func deallocate(bytes:Int, alignedTo:Int) -+++ func deallocate() - ---- func initializeMemory(as:T.Type, at:Int = 0, count:Int = 1, to:T) -> UnsafeMutablePointer -+++ func initializeMemory(as:T.Type, repeating:T, count:Int) -> UnsafeMutablePointer - - func initializeMemory(as:T.Type, from:UnsafePointer, count:Int) -> UnsafeMutablePointer - func moveInitializeMemory(as:T.Type, from:UnsafeMutablePointer, count:Int) - -> UnsafeMutablePointer - - func bindMemory(to:T.Type, count:Int) -> UnsafeMutablePointer - ---- func copyBytes(from:UnsafeRawPointer, count:Int) -+++ func copyMemory(from:UnsafeRawPointer, byteCount:Int) -} - -struct UnsafeBufferPointer -{ -+++ init(_:UnsafeMutableBufferPointer) - -+++ func deallocate() - -+++ func withMemoryRebound -+++ (to:T.Type, _ body:(UnsafeBufferPointer) -> Result) -} - -struct UnsafeMutableBufferPointer -{ -+++ init(mutating:UnsafeBufferPointer) - -+++ static -+++ func allocate(capacity:Int) -> UnsafeMutableBufferPointer - -+++ func initialize(repeating:Element) -+++ func assign(repeating:Element) - -+++ func withMemoryRebound -+++ (to:T.Type, _ body:(UnsafeMutableBufferPointer) -> Result) -} - -struct UnsafeRawBufferPointer -{ - func deallocate() - -+++ func bindMemory(to:T.Type) -> UnsafeBufferPointer -} - -struct UnsafeMutableRawBufferPointer -{ ---- static func allocate(count:Int) -> UnsafeMutableRawBufferPointer -+++ static func allocate(byteCount:Int, alignment:Int) -> UnsafeMutableRawBufferPointer - func deallocate() - -+++ func initializeMemory(as:T.Type, repeating:T) -> UnsafeMutableBufferPointer - -+++ func bindMemory(to:T.Type) -> UnsafeMutableBufferPointer - ---- func copyBytes(from:UnsafeRawBufferPointer) -+++ func copyMemory(from:UnsafeRawBufferPointer) -} -``` - -## Source compatibility - -Everything is additive except the following. Can we deprecate all of -the original functions in Swift 5, then drop those from the binary -later in Swift 6? - -- **add `deallocate()` to all pointer types, replacing any existing deallocation methods** - -The migrator needs to drop the existing `capacity` and `alignedTo` arguments. - -- **in `UnsafeMutableRawPointer.allocate(count:alignedTo:)` rename `count` to `byteCount` and `alignedTo` to `alignment`** - -- **in `UnsafeMutableRawBufferPointer.allocate(count:)` rename `count` to `byteCount` and add an `alignment` parameter** - -This change is source breaking but can be trivially automigrated. The -`alignment:` parameter can be filled in with `MemoryLayout.stride`. - -- **fix the arguments to `initialize(repeating:Pointee, count:Int)`** - -Note: initialize(to:Pointee) is backward compatible whenever the -caller relied on a default `count = 1`. - -An annotation could otherwise rename `to` to `repeating`, but we don't -want that to interfere with the default count case, so this might need to be a migrator rule. - -- **fix the ordering of the arguments in `initializeMemory(as:at:count:to:)`, rename `to:` to `repeating:`, and remove the `at:` argument** - -This change is source breaking but can be trivially automigrated. The -`to` argument changes position and is relabeled as `repeating`. - -The migrator could be taught to convert the `at:` argument into -pointer arithmetic on `self`. However, we found no code on Github that -uses the `at:` argument, so it is low priority. - -- **rename `copyBytes(from:count:)` to `copyMemory(from:byteCount:)`** - -This change is source breaking but can be trivially automigrated. - -## Effect on ABI stability - -Removing sized deallocators changes the existing ABI, as will renaming some of the methods and their argument labels. - -## Effect on API resilience - -Some proposed changes in this proposal change the public API. - -Removing sized deallocators right now will break ABI, but offers increased ABI and API stability in the future as reallocator methods can be added in the future without having to rename `deallocate(capacity:)` which currently occupies a “reallocator” name, but has “`free()`” behavior. - -## Alternatives considered - -- **keeping sized deallocators and fixing the stdlib implementation instead** - -Instead of dropping the `capacity` parameter from `deallocate(capacity:)`, we could fix the underlying implementation so that the function actually deallocates `capacity`’s worth of memory. However this would be catastrophically, and silently, source-breaking as existing code would continue compiling, but suddenly start leaking or segfaulting at runtime. `deallocate(capacity:)` can always be added back at a later date without breaking ABI or API, once users have been forced to address this potential bug. - -- **adding an initializer `UnsafeMutableBufferPointer.init(allocatingCount:)` instead of a type method to `UnsafeMutableBufferPointer`** - -The allocator could be expressed as an initializer instead of a type method. However since allocation involves acquisition of an external resource, perhaps it is better to keep with the existing convention that allocation is performed differently than regular buffer pointer construction. - -- **using the argument label `value:` instead of `repeating:` in methods such as `initialize(repeating:count:)` (originally `initialize(to:count:)`)** - -The label `value:` or `toValue:` doesn’t fully capture the repeating nature of the argument, and is inconsistent with `Array.init(repeating:count:)`. While `value:` sounds less strange when `count == 1`, on consistency and technical correctness, `repeating:` is the better term. Furthermore, `value` is a common variable name, meaning that function calls with `value:` as the label would be prone to looking like this: - -```swift -ptr.initialize(value: value) -```