[se-0405] Implement API additions #68419
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -493,6 +493,116 @@ extension String { | |
| self = String._fromNonContiguousUnsafeBitcastUTF8Repairing(codeUnits).0 | ||
| } | ||
|
|
||
| /// Creates a new string by copying and validating the sequence of | ||
| /// code units passed in, according to the specified encoding. | ||
| /// | ||
| /// This initializer does not try to repair ill-formed code unit sequences. | ||
| /// If any are found, the result of the initializer is `nil`. | ||
| /// | ||
| /// The following example calls this initializer with the contents of two | ||
| /// different arrays---first with a well-formed UTF-8 code unit sequence and | ||
| /// then with an ill-formed UTF-16 code unit sequence. | ||
| /// | ||
| /// let validUTF8: [UInt8] = [67, 97, 0, 102, 195, 169] | ||
| /// let valid = String(validating: validUTF8, as: UTF8.self) | ||
| /// print(valid ?? "nil") | ||
| /// // Prints "Café" | ||
| /// | ||
| /// let invalidUTF16: [UInt16] = [0x41, 0x42, 0xd801] | ||
| /// let invalid = String(validating: invalidUTF16, as: UTF16.self) | ||
| /// print(invalid ?? "nil") | ||
| /// // Prints "nil" | ||
| /// | ||
| /// - Parameters: | ||
| /// - codeUnits: A sequence of code units that encode a `String` | ||
| /// - encoding: A conformer to `Unicode.Encoding` to be used | ||
| /// to decode `codeUnits`. | ||
| @inlinable | ||
| @available(SwiftStdlib 5.11, *) | ||
| public init?<Encoding: Unicode.Encoding>( | ||
| validating codeUnits: some Sequence<Encoding.CodeUnit>, | ||
| as encoding: Encoding.Type | ||
| ) { | ||
| let contiguousResult = codeUnits.withContiguousStorageIfAvailable { | ||
| String._validate($0, as: Encoding.self) | ||
| } | ||
| if let validationResult = contiguousResult { | ||
| guard let validatedString = validationResult else { | ||
| return nil | ||
| } | ||
| self = validatedString | ||
| return | ||
| } | ||
|
|
||
| // slow-path | ||
| var transcoded: [UTF8.CodeUnit] = [] | ||
| transcoded.reserveCapacity(codeUnits.underestimatedCount) | ||
| var isASCII = true | ||
| let error = transcode( | ||
| codeUnits.makeIterator(), | ||
| from: Encoding.self, | ||
| to: UTF8.self, | ||
| stoppingOnError: true, | ||
| into: { | ||
| uint8 in | ||
| transcoded.append(uint8) | ||
| if isASCII && (uint8 & 0x80) == 0x80 { isASCII = false } | ||
|
||
| } | ||
| ) | ||
| if error { return nil } | ||
| self = transcoded.withUnsafeBufferPointer{ | ||
| String._uncheckedFromUTF8($0, asciiPreScanResult: isASCII) | ||
| } | ||
| } | ||
|
|
||
| /// Creates a new string by copying and validating the sequence of | ||
| /// code units passed in, according to the specified encoding. | ||
| /// | ||
| /// This initializer does not try to repair ill-formed code unit sequences. | ||
| /// If any are found, the result of the initializer is `nil`. | ||
| /// | ||
| /// The following example calls this initializer with the contents of two | ||
| /// different arrays---first with a well-formed UTF-8 code unit sequence and | ||
| /// then with an ill-formed ASCII code unit sequence. | ||
| /// | ||
| /// let validUTF8: [Int8] = [67, 97, 0, 102, -61, -87] | ||
| /// let valid = String(validating: validUTF8, as: UTF8.self) | ||
| /// print(valid ?? "nil") | ||
| /// // Prints "Café" | ||
| /// | ||
| /// let invalidASCII: [Int8] = [67, 97, -5] | ||
| /// let invalid = String(validating: invalidASCII, as: Unicode.ASCII.self) | ||
| /// print(invalid ?? "nil") | ||
| /// // Prints "nil" | ||
| /// | ||
| /// - Parameters: | ||
| /// - codeUnits: A sequence of code units that encode a `String` | ||
| /// - encoding: A conformer to `Unicode.Encoding` that can decode | ||
| /// `codeUnits` as `UInt8` | ||
| @inlinable | ||
| @available(SwiftStdlib 5.11, *) | ||
| public init?<Encoding>( | ||
| validating codeUnits: some Sequence<Int8>, | ||
| as encoding: Encoding.Type | ||
| ) where Encoding: Unicode.Encoding, Encoding.CodeUnit == UInt8 { | ||
| let contiguousResult = codeUnits.withContiguousStorageIfAvailable { | ||
| $0.withMemoryRebound(to: UInt8.self) { | ||
| String._validate($0, as: Encoding.self) | ||
| } | ||
| } | ||
| if let validationResult = contiguousResult { | ||
| guard let validatedString = validationResult else { | ||
| return nil | ||
| } | ||
| self = validatedString | ||
| return | ||
| } | ||
|
|
||
| // slow-path | ||
| let uint8s = codeUnits.lazy.map(UInt8.init(bitPattern:)) | ||
| self.init(validating: uint8s, as: Encoding.self) | ||
| } | ||
|
|
||
| /// Creates a new string with the specified capacity in UTF-8 code units, and | ||
| /// then calls the given closure with a buffer covering the string's | ||
| /// uninitialized memory. | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This allocates an Array (which will likely require resizing as transcoded code-units are appended to it -- the
._validatefunction used by the contiguous path estimates 3-4x as many bytes of UTF8 out as code-units in), then allocates separate String storage and writes the result to it.Have you tried performing a dry-run of the transcoding and measuring the required capacity, then allocating String storage and writing directly to it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ohhh we can't do a dry run because this takes a Sequence, not a Collection 😔.
But it's pointless - we're not going to do this in a single pass; we're just going to copy internally anyway. This API should've taken a Collection, IMO.
We can dispatch based on whether or not the type is a collection, but the optimiser does a poor job specialising it (#62264). I'd still suggest we do that, though, and let the optimiser catch up. The existential wrapping and unwrapping overhead is likely less than allocating and copying everything to an array, and one day the compiler will just eliminate that overhead entirely.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is quite a bit of improvement to be done to the internal transcoding machinery, and this slowest path will then be adapted to use that. It would be better to allocate a string buffer directly and resize that, but it is not possible at the moment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My understanding is that it is fairly common in C APIs (e.g.
wcsrtombs,WideCharToMultiByte, etc) to perform a dry run by specifying the output buffer asNULL, getting the length, and converting in to an appropriately-sized buffer. That's why I suggested it.Resizing is less than ideal because it's quadratic. Array mitigates this somewhat with an over-allocation strategy that scales geometrically, but that is also wasteful, and I don't think String employs the same strategy (?).
For Sequence, where we can only make one pass, we unfortunately have to transcode in to some kind of resizing buffer. For Collection, we can just do two passes, guaranteeing no resizing and lower memory water-mark.