Skip to content

Commit 43ba3c3

Browse files
committed
Improve diffing of collections and structured values.
This PR improves the test output when comparing collections and structured values (using `#expect()`) when the comparison fails. Previously, we'd just the insertions and deletions as arrays, which was potentially hard to read and didn't show the test author _where_ these changes occurred. Now, we convert the compared values into something similar to multi-line `diff` output. Consider the following test function: ```swift @test func f() { let lhs = """ Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. """ let rhs = """ Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo potato salad. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non dentistry, sunt in culpa qui officia deserunt mollit anim id est laborum. """ #expect(lhs == rhs) } ``` The output will now be: ``` ◇ Test f() started. ✘ Test f() recorded an issue at MyTests.swift:45:5: Expectation failed: lhs == rhs ± 4 changes: "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod" "tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam," "quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo" + "potato salad. Duis aute irure dolor in reprehenderit in voluptate velit esse" - "consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse" "cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat" + "non dentistry, sunt in culpa qui officia deserunt mollit anim id est laborum." - "non proident, sunt in culpa qui officia deserunt mollit anim id est laborum." ✘ Test f() failed after 0.001 seconds with 1 issue. ``` Similarly, with this test: ```swift struct S: Equatable { var x: Int var y: String } @test func f() { let lhs = S(x: 123, y: "abc") let rhs = S(x: 123, y: "def") #expect(lhs == rhs) } ``` The output should appear similar to: ``` ◇ Test f() started. ✘ Test f() recorded an issue at MyTests.swift:36:5: Expectation failed: lhs == rhs ± 1 change: MyTests.S( x: 123 - y: "abc" + y: "def" ) ✘ Test f() failed after 0.001 seconds with 1 issue. ``` Resolves rdar://66351980.
1 parent 2cc498b commit 43ba3c3

File tree

8 files changed

+369
-43
lines changed

8 files changed

+369
-43
lines changed

Sources/Testing/Events/Event.Recorder.swift

Lines changed: 33 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -344,6 +344,37 @@ extension Event.Recorder {
344344
}
345345
}
346346

347+
// MARK: -
348+
349+
extension Difference {
350+
/// Get a description of this instance with the given recorder options.
351+
///
352+
/// - Parameters:
353+
/// - options: Options to use when writing the comments.
354+
///
355+
/// - Returns: A formatted description of `self`.
356+
fileprivate func formattedDescription(options: Set<Event.Recorder.Option>) -> String {
357+
guard options.contains(.useANSIEscapeCodes) else {
358+
return String(describing: self)
359+
}
360+
361+
return String(describing: self)
362+
.split(whereSeparator: \.isNewline)
363+
.map { line in
364+
switch line.first {
365+
case "+":
366+
"\(_ansiEscapeCodePrefix)32m\(line)\(_resetANSIEscapeCode)"
367+
case "-":
368+
"\(_ansiEscapeCodePrefix)31m\(line)\(_resetANSIEscapeCode)"
369+
default:
370+
String(line)
371+
}
372+
}.joined(separator: "\n")
373+
}
374+
}
375+
376+
// MARK: -
377+
347378
extension Tag {
348379
/// Get an ANSI escape code that sets the foreground text color to this tag's
349380
/// corresponding color, if applicable.
@@ -553,9 +584,9 @@ extension Event.Recorder {
553584
}
554585

555586
var difference = ""
556-
if case let .expectationFailed(expectation) = issue.kind, let differenceDescription = expectation.differenceDescription {
587+
if case let .expectationFailed(expectation) = issue.kind, let differenceValue = expectation.difference {
557588
let differenceSymbol = _Symbol.difference.stringValue(options: options)
558-
difference = "\n\(differenceSymbol) \(differenceDescription)"
589+
difference = "\n\(differenceSymbol) \(differenceValue.formattedDescription(options: options))"
559590
}
560591

561592
var issueComments = ""
Lines changed: 193 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,193 @@
1+
//
2+
// This source file is part of the Swift.org open source project
3+
//
4+
// Copyright (c) 2023 Apple Inc. and the Swift project authors
5+
// Licensed under Apache License v2.0 with Runtime Library Exception
6+
//
7+
// See https://swift.org/LICENSE.txt for license information
8+
// See https://swift.org/CONTRIBUTORS.txt for Swift project authors
9+
//
10+
11+
/// A type that describes the difference between two values as a sequence of
12+
/// insertions, deletions, or unmodified elements.
13+
///
14+
/// To ensure that ``Difference`` can always conform to `Sendable`, the elements
15+
/// in an instance of this type are stored as strings rather than as their
16+
/// original types. They are converted to strings using
17+
/// ``Swift/String/init(describingForTest:)``. Types can implement
18+
/// ``CustomTestStringConvertible`` to customize how they appear in the
19+
/// description of an instance of this type.
20+
@_spi(ExperimentalEventHandling)
21+
public struct Difference: Sendable {
22+
/// An enumeration representing the kinds of change that can occur during
23+
/// diffing.
24+
enum ElementKind: Sendable {
25+
/// The element was inserted.
26+
case insert
27+
28+
/// The element was removed.
29+
case remove
30+
31+
/// The element replaced a previous value.
32+
///
33+
/// - Parameters:
34+
/// - oldValue: The old value at this position.
35+
case replace(oldValue: String)
36+
}
37+
38+
/// A type representing an element of a collection that may have been changed
39+
/// after diffing was applied.
40+
///
41+
/// This type roughly approximates `CollectionDifference.Change`, however it
42+
/// is used to track _all_ elements in the collection, not just those that
43+
/// have changed, allowing for insertion of "marker" elements where removals
44+
/// occurred.
45+
typealias Element = (value: String, kind: ElementKind?)
46+
47+
/// The changed elements from the comparison.
48+
var elements = [Element]()
49+
50+
init(elements: some Sequence<Element>) {
51+
self.elements = Array(elements)
52+
}
53+
54+
/// Initialize an instance of this type by comparing two collections.
55+
///
56+
/// - Parameters:
57+
/// - lhs: The "old" state of the collection to compare.
58+
/// - rhs: The "new" state of the collection to compare.
59+
/// - describingForTest: Whether or not to convert values in `lhs` and `rhs`
60+
/// to strings using ``Swift/String/init(describingForTest:)``.
61+
init<T, U>(from lhs: T, to rhs: U, describingForTest: Bool = true)
62+
where T: BidirectionalCollection, T.Element: Equatable, U: BidirectionalCollection, T.Element == U.Element {
63+
// Compute the difference between the two elements. Sort the resulting set
64+
// of changes by their offsets, and ensure that insertions come before
65+
// removals located at the same offset. This helps to ensure that the offset
66+
// values do not drift as we walk the changeset.
67+
let difference = rhs.difference(from: lhs)
68+
69+
// Walk the initial string and slowly transform it into the final string.
70+
// Add an additional "scratch" string that is used to store a removal marker
71+
// if the last character is removed.
72+
var result: [[(value: T.Element, kind: ElementKind?)]] = lhs.map { [($0, nil)] } + CollectionOfOne([])
73+
for change in difference.removals.reversed() {
74+
// Remove the character at the specified index, then re-insert it into the
75+
// slot at the previous index (with the marker character applied.) The
76+
// previous index will then contain whatever character it already
77+
// contained after the character representing this removal.
78+
result.remove(at: change.offset)
79+
result[change.offset].insert((change.element, kind: .remove), at: 0)
80+
}
81+
for change in difference.insertions {
82+
// Insertions can occur verbatim by inserting a new substring at the
83+
// specified offset.
84+
result.insert([(change.element, kind: .insert)], at: change.offset)
85+
}
86+
87+
let describe: (T.Element) -> String = if describingForTest {
88+
String.init(describingForTest:)
89+
} else {
90+
String.init(describing:)
91+
}
92+
93+
self.init(
94+
elements: result.lazy
95+
.flatMap { $0 }
96+
.map { (describe($0.value), $0.kind) }
97+
.reduce(into: []) { (result: inout _, element: Element) in
98+
if element.kind == .remove, let previous = result.last, previous.kind == .insert {
99+
result[result.index(before: result.endIndex)] = (previous.value, .replace(oldValue: element.value))
100+
} else {
101+
result.append(element)
102+
}
103+
}
104+
)
105+
}
106+
107+
/// Get a string reflecting a value, similar to how it might have been
108+
/// initialized and suitable for display as part of a difference.
109+
///
110+
/// - Parameters:
111+
/// - value: The value to reflect.
112+
///
113+
/// - Returns: A string reflecting `value`, or `nil` if its reflection is
114+
/// trivial.
115+
///
116+
/// This function uses `Mirror`, so if the type of `value` conforms to
117+
/// `CustomReflectable`, the resulting string will be derived from the value's
118+
/// custom mirror.
119+
private static func _reflect<T>(_ value: T) -> String? {
120+
let mirrorChildren = Mirror(reflecting: value).children
121+
if mirrorChildren.isEmpty {
122+
return nil
123+
}
124+
125+
let typeName = _typeName(T.self, qualified: true)
126+
let children = mirrorChildren.lazy
127+
.map { child in
128+
if let label = child.label {
129+
" \(label): \(String(describingForTest: child.value))"
130+
} else {
131+
" \(String(describingForTest: child.value))"
132+
}
133+
}.joined(separator: "\n")
134+
return """
135+
\(typeName)(
136+
\(children)
137+
)
138+
"""
139+
}
140+
141+
/// Initialize an instance of this type by comparing the reflections of two
142+
/// values.
143+
///
144+
/// - Parameters:
145+
/// - lhs: The "old" value to compare.
146+
/// - rhs: The "new" value to compare.
147+
init?<T, U>(comparingValue lhs: T, to rhs: U) {
148+
guard let lhsDump = Self._reflect(lhs), let rhsDump = Self._reflect(rhs) else {
149+
return nil
150+
}
151+
152+
let lhsDumpLines = lhsDump.split(whereSeparator: \.isNewline)
153+
let rhsDumpLines = rhsDump.split(whereSeparator: \.isNewline)
154+
if lhsDumpLines.count > 1 || rhsDumpLines.count > 1 {
155+
self.init(from: lhsDumpLines, to: rhsDumpLines, describingForTest: false)
156+
} else {
157+
return nil
158+
}
159+
}
160+
}
161+
162+
// MARK: - Equatable
163+
164+
extension Difference.ElementKind: Equatable {}
165+
166+
// MARK: - CustomStringConvertible
167+
168+
extension Difference: CustomStringConvertible {
169+
public var description: String {
170+
// Show individual lines of the text with leading + or - characters to
171+
// indicate insertions and removals respectively.
172+
// FIXME: better descriptive output for one-line strings.
173+
let diffCount = elements.lazy
174+
.filter { $0.kind != nil }
175+
.count
176+
return "\(diffCount.counting("change")):\n" + elements.lazy
177+
.flatMap { element in
178+
switch element.kind {
179+
case nil:
180+
[" \(element.value)"]
181+
case .insert:
182+
["+ \(element.value)"]
183+
case .remove:
184+
["- \(element.value)"]
185+
case let .replace(oldValue):
186+
[
187+
"- \(oldValue)",
188+
"+ \(element.value)"
189+
]
190+
}
191+
}.joined(separator: "\n")
192+
}
193+
}

Sources/Testing/Expectations/Expectation.swift

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -31,14 +31,17 @@ public struct Expectation: Sendable {
3131
@_spi(ExperimentalEventHandling)
3232
public var expandedExpressionDescription: String?
3333

34-
/// A description of the difference between the operands in the expression
35-
/// evaluated by this expectation, if the difference could be determined.
34+
/// An array of changes that collectively describe the difference between two
35+
/// values or collections that have been compared.
3636
///
37-
/// If this expectation passed, the value of this property is `nil` because
38-
/// the difference is only computed when necessary to assist with diagnosing
39-
/// test failures.
37+
/// If this expectation did not involve the comparison of two values or
38+
/// collections, the value of this property is `nil`.
39+
///
40+
/// If this expectation passed, the value of this property will be `nil`
41+
/// because the difference is only computed when necessary to assist with
42+
/// diagnosing failures.
4043
@_spi(ExperimentalEventHandling)
41-
public var differenceDescription: String?
44+
public var difference: Difference?
4245

4346
/// Whether the expectation passed or failed.
4447
///

0 commit comments

Comments
 (0)