Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[String] Use a UTF-8 representation for native strings #20315

Merged
merged 45 commits into from
Nov 5, 2018

Conversation

milseman
Copy link
Member

@milseman milseman commented Nov 4, 2018

Original PR

Switch the native encoding of String to be UTF-8 and finalize String's ABI for Swift 5.0.

Resolves SR-7602.
rdar://problem/42339222

milseman and others added 30 commits November 4, 2018 10:42
This is a giant squashing of a lot of individual changes prototyping a
switch of String in Swift 5 to be natively encoded as UTF-8. It
includes what's necessary for a functional prototype, dropping some
history, but still leaves plenty of history available for future
commits.

My apologies to anyone trying to do code archeology between this
commit and the one prior. This was the lesser of evils.
Clean up some of the index assumptions, stick index-aware methods on
_StringGuts, and otherwise migrate code over to UnicodeHelpers.swift.
* Refactor out RRC implementation into dedicated file.

* Change our `_invariantCheck` pattern to generate efficient code in
  asserts builds and make the optimizer job's easier.

* Drop a few Bidi shims we no longer need.

* Restore View decls to String, workaround no longer needed

* Cleaner unicode helper facilities
Breadcrumbs provide us amortized O(1) access to the UTF-16 view, which
is vital for efficient Cocoa interoperability.
Add in our scalar-based fast-paths for UTF-8 and foreign strings, and
update the grapheme cache.
Clean up some of the code surrounding the normalized code unit
iterator.
swiftlang#19360)

* Add consuming/owned annotations to Collection implementations

* Update SILOptimizer tests

* Fix access_marker_verify test

* XFAIL reconstruct_type_from_mangled_name
Add inlinability annotations to restore performance parity with 4.2 String.

Take advantage of known NFC as a fast-path for comparison, and
overhaul comparison dispatch.

RRC improvements and optmizations.
this is just to be compatible with the current transcoders and can be
undone when the transcoders (based on ForwardParser/ReverseParser) are
improved too.
Add benchmarks and tests for the normalized iterator
Refactor and rename _StringGutsSlice, apply NFC-aware fast paths to a
new buffered iterator.

Also, fix bug in _typeName which used to assume ASCIIness and better
SIL optimizations on StringObject.
Replaces tests for constant-folding string concatenation with
Swift-based ones, which are easier to maintain going forwards.
Also, disable normalization benchmarks and other changes until we
merge, so we can compare with master 1-to-1.
Tweak and adjust code so that the SIL optimizer can constant-fold
small strings from literals. Also some cleanup.
@shahmishal
Copy link
Member

Looks like it started after:

Git (clang)

[CodeGen] Move emitConstant from ScalarExprEmitter to CodeGenFunction. (detail)
[CodeGen] Fix assertion on referencing constexpr Obj-C object with ARC. (detail)
[VFS] Add property 'fallthrough' that controls fallback to real file (detail)
[VFS] Add property 'fallthrough' that controls fallback to real file (detail)
[indexstore] Fix crashing issue with (detail)
[CodeGen] Fix a crash when updating a designated initializer (detail)
local test fixups: -Wframework-include-private-from-public in (detail)
[Sema] Fix bad merge of test/SemaCXX/no_destroy.cpp in commit 2b6d354. (detail)
Add an explicit -std=c++14 to this test. (detail)
[indexstore] Enhancement for IndexStoreCXX.h's getUnitNameFromOutputPath (detail)
[clang-check-codegen][cfstring] Accept either @ or % for progbits to (detail)

@gottesmm
Copy link
Contributor

gottesmm commented Nov 4, 2018

@shahmishal I think I found the problem. Trying a revert here: apple/swift-clang#230.

@milseman
Copy link
Member Author

milseman commented Nov 4, 2018

Please test with following PR:
apple/swift-lldb#1022
apple/swift-clang#230

@swift-ci please test os x platform

@milseman
Copy link
Member Author

milseman commented Nov 5, 2018

Please test with following PR:
apple/swift-lldb#1022
apple/swift-clang#230

@swift-ci please test

@swift-ci
Copy link
Contributor

swift-ci commented Nov 5, 2018

Build failed
Swift Test OS X Platform
Git Sha - 2ef299c80699e43a11503a66cdfb5f38530e9f41

@milseman
Copy link
Member Author

milseman commented Nov 5, 2018

Please test with following PR:
apple/swift-lldb#1022

@swift-ci please test

@swift-ci
Copy link
Contributor

swift-ci commented Nov 5, 2018

Build failed
Swift Test Linux Platform
Git Sha - 2ef299c80699e43a11503a66cdfb5f38530e9f41

@milseman
Copy link
Member Author

milseman commented Nov 5, 2018

Please test with following PR:
apple/swift-lldb#1022

@swift-ci please test linux platform

@swift-ci
Copy link
Contributor

swift-ci commented Nov 5, 2018

Build comment file:

Summary for master full

Unexpected test results, excluded stats for RSWeb, NIOHTTP1, NIOChatClient, Kommander, SwiftGraph, SourceKittenFramework, Yams, Starscream, swiftlint, SWXMLHash, Cub, Siesta, NIOChatServer, URLEncodedForm, SwiftLintFramework, ObjectMapper, NIOEchoClient, ReSwift, RSTree, NonEmpty, SwiftyJSON, Multipart, Example, TemplateKit, SiestaUI, RSDatabase, NIOPerformanceTester, NIOWebSocket, RxTest, Kronos, Wordy, NIOEchoServer, Result, RxSwift, Mapper, NIOHTTP1Server, NIOWebSocketServer, Socket, Then, Kingfisher, Commandant, Surge, Cryptor, CoreStore, SRP, KeychainAccess, LaunchScreenSnapshot, PinkyPromise, RxBlocking, Validation, SwifterSwift, CleanroomLogger, Tagged, PromiseKit, SwiftDate, SwiftyTextTable, SipHash, SwiftyStoreKit, PanelKit, ConsoleDevelopment, ProcedureKit, BigInt, GRDB, Alamofire, NIOTLS, Routing

Regressions found (see below)

Debug-batch

debug-batch brief

Regressed (3)
name old new delta delta_pct
Frontend.NumInstructionsExecuted 7,803,545,308,422 10,089,987,137,086 2,286,441,828,664 29.3% ⛔
LLVM.NumLLVMBytesOutput 276,456,832 373,212,504 96,755,672 35.0% ⛔
time.swift-driver.wall 754.2s 1045.3s 291.1s 38.6% ⛔
Improved (0)
name old new delta delta_pct
Unchanged (delta < 1.0% or delta < 100.0ms) (0)
name old new delta delta_pct

debug-batch detailed

Regressed (74)
name old new delta delta_pct
AST.NumASTBytesAllocated 18,234,450,532 23,209,629,047 4,975,178,515 27.28% ⛔
AST.NumDecls 17,719 26,205 8,486 47.89% ⛔
AST.NumDependencies 52,812 82,686 29,874 56.57% ⛔
AST.NumImportedExternalDefinitions 389,864 474,768 84,904 21.78% ⛔
AST.NumInfixOperators 5,915 9,791 3,876 65.53% ⛔
AST.NumLoadedModules 62,104 94,147 32,043 51.6% ⛔
AST.NumObjCMethods 7,023 7,284 261 3.72% ⛔
AST.NumPrecedenceGroups 3,249 4,576 1,327 40.84% ⛔
AST.NumReferencedDynamicNames 53 60 7 13.21% ⛔
AST.NumReferencedMemberNames 1,020,089 1,353,712 333,623 32.71% ⛔
AST.NumReferencedTopLevelNames 52,840 82,688 29,848 56.49% ⛔
AST.NumSourceBuffers 74,973 109,738 34,765 46.37% ⛔
AST.NumSourceLines 503,156 780,068 276,912 55.04% ⛔
AST.NumSourceLinesPerSecond 337,382 542,174 204,792 60.7% ⛔
AST.NumTotalClangImportedEntities 1,350,207 1,642,155 291,948 21.62% ⛔
AST.NumUsedConformances 47,222 68,977 21,755 46.07% ⛔
Driver.ChildrenMaxRSS 20,703,776,768 34,632,275,968 13,928,499,200 67.28% ⛔
Driver.NumDriverJobsRun 3,548 5,303 1,755 49.46% ⛔
Driver.NumDriverPipePolls 38,636 67,331 28,695 74.27% ⛔
Driver.NumDriverPipeReads 42,091 74,079 31,988 76.0% ⛔
Frontend.MaxMallocUsage 135,200,902,632 180,671,045,096 45,470,142,464 33.63% ⛔
Frontend.NumInstructionsExecuted 7,803,545,308,422 10,089,987,137,086 2,286,441,828,664 29.3% ⛔
IRModule.NumIRAliases 32,448 41,518 9,070 27.95% ⛔
IRModule.NumIRBasicBlocks 810,328 1,193,185 382,857 47.25% ⛔
IRModule.NumIRFunctions 482,374 675,690 193,316 40.08% ⛔
IRModule.NumIRGlobals 625,163 804,964 179,801 28.76% ⛔
IRModule.NumIRInsts 10,007,010 14,506,655 4,499,645 44.96% ⛔
IRModule.NumIRNamedMetaData 17,065 25,350 8,285 48.55% ⛔
IRModule.NumIRValueSymbols 994,290 1,338,405 344,115 34.61% ⛔
LLVM.NumLLVMBytesOutput 276,456,832 373,212,504 96,755,672 35.0% ⛔
Parse.NumFunctionsParsed 588,708 886,260 297,552 50.54% ⛔
Parse.NumIterableDeclContextParsed 229,332 349,677 120,345 52.48% ⛔
SILModule.NumSILGenFunctions 567,762 677,854 110,092 19.39% ⛔
SILModule.NumSILGenGlobalVariables 3,848 9,928 6,080 158.0% ⛔
SILModule.NumSILGenVtables 2,042 2,681 639 31.29% ⛔
SILModule.NumSILGenWitnessTables 9,013 13,541 4,528 50.24% ⛔
SILModule.NumSILOptFunctions 342,001 501,819 159,818 46.73% ⛔
SILModule.NumSILOptGlobalVariables 4,019 10,154 6,135 152.65% ⛔
SILModule.NumSILOptVtables 3,703 4,921 1,218 32.89% ⛔
SILModule.NumSILOptWitnessTables 17,552 28,556 11,004 62.69% ⛔
Sema.AccessLevelRequest 556,711 791,992 235,281 42.26% ⛔
Sema.DefaultAndMaxAccessLevelRequest 13,747 21,826 8,079 58.77% ⛔
Sema.EnumRawTypeRequest 3,763 5,457 1,694 45.02% ⛔
Sema.ExtendedNominalRequest 908,303 1,277,065 368,762 40.6% ⛔
Sema.InheritedDeclsReferencedRequest 33,698,577 38,740,318 5,041,741 14.96% ⛔
Sema.InheritedTypeRequest 224,702 252,595 27,893 12.41% ⛔
Sema.IsDynamicRequest 424,545 613,197 188,652 44.44% ⛔
Sema.IsObjCRequest 368,215 524,134 155,919 42.34% ⛔
Sema.NamedLazyMemberLoadFailureCount 7,122 8,338 1,216 17.07% ⛔
Sema.NamedLazyMemberLoadSuccessCount 4,005,136 5,940,349 1,935,213 48.32% ⛔
Sema.NominalTypeLookupDirectCount 9,091,078 11,484,805 2,393,727 26.33% ⛔
Sema.NumConformancesDeserialized 1,250,407 1,750,722 500,315 40.01% ⛔
Sema.NumConstraintScopes 3,065,545 4,274,616 1,209,071 39.44% ⛔
Sema.NumConstraintsConsideredForEdgeContraction 6,192,279 7,998,171 1,805,892 29.16% ⛔
Sema.NumDeclsDeserialized 11,163,514 15,394,448 4,230,934 37.9% ⛔
Sema.NumDeclsValidated 438,413 647,123 208,710 47.61% ⛔
Sema.NumFunctionsTypechecked 334,896 425,663 90,767 27.1% ⛔
Sema.NumGenericSignatureBuilders 339,572 442,161 102,589 30.21% ⛔
Sema.NumLazyGenericEnvironments 2,307,670 3,189,363 881,693 38.21% ⛔
Sema.NumLazyGenericEnvironmentsLoaded 43,845 62,970 19,125 43.62% ⛔
Sema.NumLazyIterableDeclContexts 1,733,760 2,477,655 743,895 42.91% ⛔
Sema.NumLeafScopes 2,149,511 2,966,950 817,439 38.03% ⛔
Sema.NumTypesDeserialized 3,845,567 5,447,862 1,602,295 41.67% ⛔
Sema.NumTypesValidated 268,416 403,627 135,211 50.37% ⛔
Sema.NumUnloadedLazyIterableDeclContexts 1,169,255 1,756,502 587,247 50.22% ⛔
Sema.OverriddenDeclsRequest 1,788,634 2,168,299 379,665 21.23% ⛔
Sema.RequirementRequest 20,579 26,006 5,427 26.37% ⛔
Sema.SelfBoundsFromWhereClauseRequest 21,261,236 25,106,420 3,845,184 18.09% ⛔
Sema.SetterAccessLevelRequest 30,908 45,937 15,029 48.62% ⛔
Sema.SuperclassDeclRequest 24,262,744 28,517,881 4,255,137 17.54% ⛔
Sema.SuperclassTypeRequest 7,760 10,138 2,378 30.64% ⛔
Sema.TypeDeclsFromWhereClauseRequest 9,708 13,433 3,725 38.37% ⛔
Sema.USRGenerationRequest 2,983,277 3,575,262 591,985 19.84% ⛔
Sema.UnderlyingTypeDeclsReferencedRequest 1,234,741 1,511,912 277,171 22.45% ⛔
Improved (0)
name old new delta delta_pct
Unchanged (delta < 1.0% or delta < 100.0ms) (21)
name old new delta delta_pct
AST.NumLinkLibraries 0 0 0 0.0%
AST.NumLocalTypeDecls 92 92 0 0.0%
AST.NumPostfixOperators 8 8 0 0.0%
AST.NumPrefixOperators 21 21 0 0.0%
Driver.DriverDepCascadingDynamic 0 0 0 0.0%
Driver.DriverDepCascadingExternal 0 0 0 0.0%
Driver.DriverDepCascadingMember 0 0 0 0.0%
Driver.DriverDepCascadingNominal 0 0 0 0.0%
Driver.DriverDepCascadingTopLevel 0 0 0 0.0%
Driver.DriverDepDynamic 0 0 0 0.0%
Driver.DriverDepExternal 0 0 0 0.0%
Driver.DriverDepMember 0 0 0 0.0%
Driver.DriverDepNominal 0 0 0 0.0%
Driver.DriverDepTopLevel 0 0 0 0.0%
Driver.NumDriverJobsSkipped 0 0 0 0.0%
Driver.NumProcessFailures 0 0 0 0.0%
Frontend.NumProcessFailures 0 0 0 0.0%
IRModule.NumIRComdatSymbols 0 0 0 0.0%
IRModule.NumIRIFuncs 0 0 0 0.0%
SILModule.NumSILGenDefaultWitnessTables 0 0 0 0.0%
SILModule.NumSILOptDefaultWitnessTables 0 0 0 0.0%

Release

release brief

Regressed (0)
name old new delta delta_pct
Improved (2)
name old new delta delta_pct
Frontend.NumInstructionsExecuted 8,529,237,865,817 8,397,490,863,912 -131,747,001,905 -1.54% ✅
time.swift-driver.wall 1681.1s 1650.9s -30.2s -1.79% ✅
Unchanged (delta < 1.0% or delta < 100.0ms) (1)
name old new delta delta_pct
LLVM.NumLLVMBytesOutput 283,344,390 281,060,548 -2,283,842 -0.81%

release detailed

Regressed (0)
name old new delta delta_pct
Improved (8)
name old new delta delta_pct
IRModule.NumIRBasicBlocks 1,088,090 1,041,713 -46,377 -4.26% ✅
IRModule.NumIRInsts 9,247,089 9,127,341 -119,748 -1.29% ✅
Sema.NumConformancesDeserialized 654,010 635,230 -18,780 -2.87% ✅
Sema.NumDeclsDeserialized 2,019,585 1,945,348 -74,237 -3.68% ✅
Sema.NumLazyGenericEnvironments 427,159 413,775 -13,384 -3.13% ✅
Sema.NumLazyGenericEnvironmentsLoaded 7,968 7,465 -503 -6.31% ✅
Sema.NumLazyIterableDeclContexts 266,079 261,292 -4,787 -1.8% ✅
Sema.NumTypesDeserialized 1,100,924 1,068,834 -32,090 -2.91% ✅
Unchanged (delta < 1.0% or delta < 100.0ms) (15)
name old new delta delta_pct
AST.NumImportedExternalDefinitions 80,803 80,803 0 0.0%
AST.NumLoadedModules 5,688 5,688 0 0.0%
AST.NumTotalClangImportedEntities 259,985 259,754 -231 -0.09%
AST.NumUsedConformances 69,388 69,390 2 0.0%
IRModule.NumIRFunctions 527,562 525,093 -2,469 -0.47%
IRModule.NumIRGlobals 608,110 602,855 -5,255 -0.86%
IRModule.NumIRValueSymbols 1,073,503 1,067,504 -5,999 -0.56%
LLVM.NumLLVMBytesOutput 283,344,390 281,060,548 -2,283,842 -0.81%
SILModule.NumSILGenFunctions 252,149 252,061 -88 -0.03%
SILModule.NumSILOptFunctions 325,372 324,341 -1,031 -0.32%
Sema.NumConstraintScopes 3,597,905 3,591,377 -6,528 -0.18%
Sema.NumDeclsValidated 345,261 345,261 0 0.0%
Sema.NumFunctionsTypechecked 190,328 190,328 0 0.0%
Sema.NumGenericSignatureBuilders 73,346 72,842 -504 -0.69%
Sema.NumTypesValidated 166,844 166,844 0 0.0%

@milseman
Copy link
Member Author

milseman commented Nov 5, 2018

@swift-ci please test

@milseman
Copy link
Member Author

milseman commented Nov 5, 2018

Please test with following PR:
apple/swift-lldb#1022

@swift-ci please test

@swift-ci
Copy link
Contributor

swift-ci commented Nov 5, 2018

Build failed
Swift Test Linux Platform
Git Sha - fee2787

@milseman
Copy link
Member Author

milseman commented Nov 5, 2018

This passed with full testing with https://ci.swift.org/job/swift-PR-osx/8588/ and https://ci.swift.org/view/Pull%20Request/job/swift-PR-Linux/8623/, only change was a conflict in how we XFAILed an unrelated test. I'll keep trying to get that green checkmark in the mean time.

@milseman
Copy link
Member Author

milseman commented Nov 5, 2018

Please test with following PR:
apple/swift-lldb#1022

@swift-ci please test linux platform

@swift-ci
Copy link
Contributor

swift-ci commented Nov 5, 2018

Build failed
Swift Test Linux Platform
Git Sha - fee2787

@milseman
Copy link
Member Author

milseman commented Nov 5, 2018

Please test with following PR:
apple/swift-lldb#1022

@swift-ci please test linux platform

@@ -217,4 +214,4 @@ extension NSRange : Codable {
try container.encode(self.location)
try container.encode(self.length)
}
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tiny nitpick: restore newline

@swift-ci
Copy link
Contributor

swift-ci commented Nov 5, 2018

Build failed
Swift Test Linux Platform
Git Sha - fee2787

@milseman
Copy link
Member Author

milseman commented Nov 5, 2018

TestSwiftFoundation issue is unrelated:

Build Command Output:
/home/docker_user/src/lldb/packages/Python/lldbsuite/test/lang/swift/foundation/main.swift:12:8: error: no such module 'Foundation'
import Foundation
       ^
make: *** [main.o] Error 1

@milseman
Copy link
Member Author

milseman commented Nov 5, 2018

As mentioned, this passed full Linux prior to an irrelevant change: https://ci.swift.org/view/Pull%20Request/job/swift-PR-Linux/8623/

@milseman
Copy link
Member Author

milseman commented Nov 5, 2018

Please test with following PR:
apple/swift-lldb#1022

@swift-ci please smoke test linux platform

1 similar comment
@milseman
Copy link
Member Author

milseman commented Nov 5, 2018

Please test with following PR:
apple/swift-lldb#1022

@swift-ci please smoke test linux platform

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants