diff --git a/Documentation/Evolution/RegexLiteralPitch.md b/Documentation/Evolution/RegexLiteralPitch.md deleted file mode 100644 index bf2a5dad3..000000000 --- a/Documentation/Evolution/RegexLiteralPitch.md +++ /dev/null @@ -1,292 +0,0 @@ -# Regular Expression Literals - -- Authors: Hamish Knight, Michael Ilseman - -## Introduction - -We propose to introduce a first-class regular expression literal into the language that can take advantage of library support to offer extensible, powerful, and familiar textual pattern matching. - -This is a component of a larger string processing picture. We would like to start a focused discussion surrounding our approach to the literal itself, while acknowledging that evaluating the utility of the literal will ultimately depend on the whole picture (e.g. supporting API). To aid this focused discussion, details such as the representation of captures in the type system, semantic details, extensions to lexing/parsing, additional API, etc., are out of scope of this pitch and thread. Feel free to continue discussion of anything related in the [overview thread][overview]. - -## Motivation - -Regular expressions are a ubiquitous, familiar, and concise syntax for matching and extracting text that satisfies a particular pattern. Syntactically, a regex literal in Swift should: - -- Support a syntax familiar to developers who have learned to use regular expressions in other tools and languages -- Allow reuse of many regular expressions not specifically designed for Swift (e.g. from Stack Overflow or popular programming books) -- Allow libraries to define custom types that can be constructed with regex literals, much like string literals -- Diagnose at compile time if a regex literal uses capabilities that aren't allowed by the type's regex dialect - -Further motivation, examples, and discussion can be found in the [overview thread][overview]. - -## Proposed Solution - -We propose the introduction of a regular expression literal that supports [the PCRE syntax][PCRE], in addition to new standard library protocols `ExpressibleByRegexLiteral` and `RegexLiteralProtocol` that allow for the customization of how the regex literal is interpreted (similar to [string interpolation][stringinterpolation]). The compiler will parse the PCRE syntax within a regex literal, and synthesize calls to corresponding builder methods. Types conforming to `ExpressibleByRegexLiteral` will be able to provide a builder type that opts into supporting various regex constructs through the use of normal function declarations and `@available`. - -_Note: This pitch concerns language syntax and compiler changes alone, it isn't stating what features the stdlib should support in the initial version or in future versions._ - -## Detailed Design - -A regular expression literal will be introduced using `/` delimiters, within which the compiler will parse [PCRE regex syntax][PCRE]: - -```swift -// Matches " = ", extracting the identifier and hex number -let regex = /([[:alpha:]]\w*) = ([0-9A-F]+)/ -``` - -The above regex literal will be inferred to be the default regex literal type `Regex`. Errors in the regex will be diagnosed by the compiler. - -_`Regex` here is a stand-in type, further details about the type such as if or how this will scale to strongly typed captures is still under investigation._ - -_How best to diagnose grapheme-semantic concerns is still under investigation and probably best discussed in their corresponding threads. For example, `Range` is not [countable][countable] and [ordering is not linguistically meaningful][ordering], so validating character class ranges may involve restricting to a semantically-meaningful range (e.g. ASCII). This is best discussed in the (upcoming) character class pitch/thread._ - -The compiler will then transform the literal into a set of builder calls that may be customized by adopting the `ExpressibleByRegexLiteral` protocol. Below is a straw-person transformation of this example: - -```swift -// let regex = /([[:alpha:]]\w*) = ([0-9A-F]+)/ -let regex = { - var builder = T.RegexLiteral() - - // __A4 = /([[:alpha:]]\w*)/ - let __A1 = builder.buildCharacterClass_POSIX_alpha() - let __A2 = builder.buildCharacterClass_w() - let __A3 = builder.buildConcatenate(__A1, __A2) - let __A4 = builder.buildCaptureGroup(__A3) - - // __B1 = / = / - let __B1 = builder.buildLiteral(" = ") - - // __C3 = /([0-9A-F]+)/ - let __C1 = builder.buildCustomCharacterClass(["0"..."9", "A"..."F"]) - let __C2 = builder.buildOneOrMore(__C1) - let __C3 = builder.buildCaptureGroup(__C2) - - let __D1 = builder.buildConcatenate(__A4, __B1, __C3) - return T(regexLiteral: builder.finalize(__D1)) -}() -``` - -In this formulation, the compiler fully parses the regex literal, calling mutating methods on a builder which constructs an AST. Here, the compiler recognizes syntax such as ranges and classifies metacharacters (`buildCharacterClass_w()`). Alternate formulations could involve less reasoning (`buildMetacharacter_w`), or more (`builderCharacterClass_word`). We'd like community feedback on this approach. - -Additionally, it may make sense for the stdlib to provide a `RegexLiteral` conformer that just constructs a string to pass off to a string-based library. Such a type might assume all features are supported unless communicated otherwise, and we'd like community feedback on mechanisms to communicate this (e.g. availability). - -### The `ExpressibleByRegexLiteral` and `RegexLiteralProtocol` protocols - -New `ExpressibleByRegexLiteral` and `RegexLiteralProtocol` protocols will be introduced to the standard library, and will serve a similar purpose to the existing literal protocols `ExpressibleByStringInterpolation` and `StringInterpolationProtocol`. - -```swift -public protocol ExpressibleByRegexLiteral { - associatedtype RegexLiteral : RegexLiteralProtocol = DefaultRegexLiteral - init(regexLiteral: RegexLiteral) -} - -public protocol RegexLiteralProtocol { - init() - - // Informal builder requirements for building a regex literal - // will be specified here. -} -``` - -Types conforming to `ExpressibleByRegexLiteral` will be able to provide a custom type that conforms to `RegexLiteralProtocol`, which will be used to build the resulting regex value. A default conforming type will be provided by the standard library (`DefaultRegexLiteral` here). - -Libraries can extend regex handling logic for their domains. For example, a higher-level library could provide linguistically richer regular expressions by incorporating locale, collation, language dictionaries, and fuzzier matching. Similarly, libraries wrapping different regex engines (e.g. `NSRegularExpression`) can support custom regex literals. - -### Opting into certain regex features - -We intend for the compiler to completely parse [the PCRE syntax][PCRE]. However, types conforming to `RegexLiteralProtocol` might not be able to handle the full feature set. The compiler will look for corresponding function declarations inside `RegexLiteralProtocol` and will emit a compilation error if missing. Conforming types can use `@available` on these function declarations to communicate versioning and add more support in the future. - -This approach of lookup combined with availability allows the stdlib to support more features over time. - -### Impact of using `/` as the delimiter - -#### On comment syntax - -Single line comments use the syntax `//`, which would conflict with the spelling for an empty regex literal. As such, an empty regex literal would be forbidden. - -While not conflicting with the syntax proposed in this pitch, it's also worth noting that the `//` comment syntax (in particular documentation comments that use `///`) would likely preclude the ability to use `///` as a delimiter if we ever wanted to support multi-line regex literals. It's possible though that future multi-line support could be provided through raw regex literals. Alternatively, it could be inferred from the regex options provided. For example, a regex that uses the multi-line option `/(?m)/` could be allowed to span multiple lines. - -Multi-line comments use the `/*` delimiter. As such, a regex literal starting with `*` wouldn't be parsed. This however isn't a major issue as an unqualified `*` is already invalid regex syntax. An escaped `/\*/` regex literal wouldn't be impacted. - -#### On custom infix operators using the `/` character - -Choosing `/` as the delimiter means there will be a conflict for infix operators containing `/` in cases where whitespace isn't used, for example: - -```swift -x+/y/+z -``` - -Should the operators be parsed as `+/` and `/+` respectively, or should this be parsed as `x + /y/ + z`? - -In this case, things can be disambiguated by the user inserting additional whitespace. We therefore could continue to parse `x+/y/+z` as a binary operator chain, and require additional whitespace to interpret `/y/` as a regex literal. - -#### On custom prefix and postfix operators using the `/` character - -There will also be parsing ambiguity with any user-defined prefix and postfix operators containing the `/` character. For example, code such as the following poses an issue: - -```swift -let x = /0; let y = 1/ -``` - -Should this be considered to be two `let` bindings, with each initialization expression using prefix and postfix `/` operators, or is it a single regex literal? - -This also extends more generally to prefix and postfix operators containing the `/` character, e.g: - -```swift -let x = Int { 0 } -} - -let x = 0 -/ 1 / .foo() -``` - -Today, this is parsed as a single binary operator chain `0 / 1 / .foo()`, with `.foo()` becoming an argument to the `/` operator. This is because while Swift does have some parser behavior that is affected by newlines, generally newlines are treated as whitespace, and expressions therefore may span multiple lines. However the user may well be expecting the second line to be parsed as a regex literal. - -This is also potentially an issue for result builders, for example: - -```swift -SomeBuilder { - x - / y / - z -} -``` - -Today this is parsed as `SomeBuilder { x / y / z }`, however it's likely the user was expecting this to become a result builder with 3 elements, the second of which being a regex literal. - -There is currently no source compatibility impact as both cases will continue to parse as binary operations. The user may insert a `;` on the prior line to get the desired regex literal parsing. However this may not be sufficient we may need to change parsing rules (under a version check) to favor parsing regex literals in these cases. We'd like to discuss this further with the community. - -It's worth noting that this is similar to an ambiguity that already exists today with trailing closures, for example: - -```swift -SomeBuilder { - SomeType() - { print("hello") } - AnotherType() -} -``` - -`{ print("hello") }` will be parsed as a trailing closure to `SomeType()` rather than as a separate element to the result builder. - -It can also currently arise with leading dot syntax in a result builder, e.g: - -```swift -SomeBuilder { - SomeType() - .member -} -``` - -`.member` will be parsed as a member access on `SomeType()` rather than as a separate element that may have its base type inferred by the parameter of a `buildExpression` method on the result builder. - - -## Future Directions - -### Typed captures - -Typed captures would statically represent how many captures and of what kind are present in a regex literals. They could produce a `Substring` for a regular capture, `Substring?` for a zero-or-one capture, and `Array` (or a lazy collection) for a zero(or one)-or-more capture. These are worth exploring, especially in the context of the [start of variadic generics][variadics] support, but we'd like to keep this pitch and discussion focused to the details presented. - -### Other regex literals - -Multi-line extensions to regex literals is considered future work. Generally, we'd like to encourage refactoring into `Pattern` when the regex gets to that degree of complexity. - -User-specified [choice of quote delimiters][perlquotes] is considered future work. A related approach to this could be a "raw" regex literal analogous to [raw strings][rawstrings]. For example (total strawperson), an approach where `n` `#`s before the opening delimiter would requires `n` `#` at the end of the trailing delimiter as well as requiring `n-1` `#`s to access metacharacters. - -```txt -// All of the below are trying to match a path like "/tmp/foo/bar/File.app/file.txt" - -/\/tmp\/.*\/File\.app\/file\.txt/ -#//tmp/.*/File\.app/file\.txt/# -##//tmp/#.#*/File.app/file.txt/## -``` - -"Swiftier" literals, such as with non-semantic whitespace (e.g. [Raku's][rakuregex]), is future work. We'd want to strongly consider using a different backing technology for Swifty matching literals, such as PEGs. - -Fully-custom literal support, that is literals whose bodies are not parsed and there is no default type available, is orthogonal to this work. It would require support for compilation-time Swift libraries in addition to Swift APIs for the compiler and type system. - - -### Further extension to Swift language constructs - -Other language constructs, such as raw-valued enums, might benefit from further regex enhancements. - -```swift -enum CalculatorToken: Regex { - case wholeNumber = /\d+/ - case identifier = /\w+/ - case symbol = /\p{Math}/ - ... -} -``` - -As mentioned in the overview, general purpose extensions to Swift (syntactic) pattern matching could benefit regex - -```swift -func parseField(_ field: String) -> ParsedField { - switch field { - case let text <- /#\s?(.*)/: - return .comment(text) - case let (l, u) <- /([0-9A-F]+)(?:\.\.([0-9A-F]+))?/: - return .scalars(Unicode.Scalar(hex: l) ... Unicode.Scalar(hex: u ?? l)) - case let prop <- GraphemeBreakProperty.init: - return .property(prop) - } -} -``` - -### Other semantic details - -Further details about the semantics of regex literals, such as what definition we give to character classes, the initial supported feature set, and how to switch between grapheme-semantic and scalar-semantic usage, is still under investigation and outside the scope of this discussion. - -## Alternatives considered - -### Using a different delimiter to `/` - -As explored above, using `/` as the delimiter has the potential to conflict with existing operators using that character, and may necessitate: - -- Changing of parsing rules around chained `/` over multiple lines -- Deprecating prefix and postfix operators containing the `/` character -- Requiring additional whitespace to disambiguate from infix operators containing `/` -- Requiring a new language version mode to parse the literal with `/` delimiters - -However one of the main goals of this pitch is to introduce a familiar syntax for regular expression literals, which has been the motivation behind choices such as using the PCRE regex syntax. Given the fact that `/` is an existing term of art for regular expressions, we feel that if the aforementioned parsing issues can be solved in a satisfactory manner, we should prefer it as the delimiter. - - -### Reusing string literal syntax - -Instead of supporting a first-class literal kind for regular expressions, we could instead allow users to write a regular expression in a string literal, and parse, diagnose, and generate the appropriate code when it's coerced to an `ExpressibleByRegexLiteral` conforming type. - -```swift -let regex: Regex = "([[:alpha:]]\w*) = ([0-9A-F]+)" -``` - -However we decided against this because: - -- We would not be able to easily apply custom syntax highlighting for the regex syntax -- It would require an `ExpressibleByRegexLiteral` contextual type to be treated as a regex, otherwise it would be defaulted to `String`, which may be undesired -- In an overloaded context it may be ambiguous whether a string literal is meant to be interpreted as a literal string or regex -- Regex escape sequences aren't currently compatible with string literal escape sequence rules, e.g `\w` is currently illegal in a string literal -- It wouldn't be compatible with other string literal features such as interpolations - -[PCRE]: http://pcre.org/current/doc/html/pcre2syntax.html -[overview]: https://forums.swift.org/t/declarative-string-processing-overview/52459 -[variadics]: https://forums.swift.org/t/pitching-the-start-of-variadic-generics/51467 -[stringinterpolation]: https://github.com/apple/swift-evolution/blob/master/proposals/0228-fix-expressiblebystringinterpolation.md -[countable]: https://en.wikipedia.org/wiki/Countable_set -[ordering]: https://forums.swift.org/t/min-function-doesnt-work-on-values-greater-than-9-999-any-idea-why/52004/16 -[perlquotes]: https://perldoc.perl.org/perlop#Quote-and-Quote-like-Operators -[rawstrings]: https://github.com/apple/swift-evolution/blob/main/proposals/0200-raw-string-escaping.md -[rakuregex]: https://docs.raku.org/language/regexes diff --git a/Documentation/Evolution/RegexLiterals.md b/Documentation/Evolution/RegexLiterals.md new file mode 100644 index 000000000..3a9d02a8b --- /dev/null +++ b/Documentation/Evolution/RegexLiterals.md @@ -0,0 +1,389 @@ +# Regex Literals + +- Authors: [Hamish Knight](https://github.com/hamishknight), [Michael Ilseman](https://github.com/milseman), [David Ewing](https://github.com/DaveEwing) + +## Introduction + +We propose the introduction of regex literals to Swift source code, providing compile-time checks and typed-capture inference. Regex literals help complete the story told in *[Regex Type and Overview][regex-type]*. + +## Motivation + +In *[Regex Type and Overview][regex-type]* we introduced the `Regex` type, which is able to dynamically compile a regex pattern: + +```swift +let pattern = #"(\w+)\s\s+(\S+)\s\s+((?:(?!\s\s).)*)\s\s+(.*)"# +let regex = try! Regex(compiling: pattern) +// regex: Regex +``` + +The ability to compile regex patterns at run time is useful for cases where it is e.g provided as user input, however it is suboptimal when the pattern is statically known for a number of reasons: + +- Regex syntax errors aren't detected until run time, and explicit error handling (e.g `try!`) is required to deal with these errors. +- No special source tooling support, such as syntactic highlighting, code completion, and refactoring support, is available. +- Capture types aren't known until run time, and as such a dynamic `AnyRegexOutput` capture type must be used. +- The syntax is overly verbose, especially for e.g an argument to a matching function. + +## Proposed solution + +A regex literal may be written using `/.../` delimiters: + +```swift +// Matches " = ", extracting the identifier and hex number +let regex = /(?[[:alpha:]]\w*) = (?[0-9A-F]+)/ +// regex: Regex<(Substring, identifier: Substring, hex: Substring)> +``` + +Forward slashes are a regex term of art. They are used as the delimiters for regex literals in, e.g., Perl, JavaScript and Ruby. Perl and Ruby additionally allow for [user-selected delimiters](https://perldoc.perl.org/perlop#Quote-and-Quote-like-Operators) to avoid having to escape any slashes inside a regex. For that purpose, we propose the extended literal `#/.../#`. + +An extended literal, `#/.../#`, avoids the need to escape forward slashes within the regex. It allows an arbitrary number of balanced `#` characters around the literal and escape. When the opening delimiter is followed by a new line, it supports a multi-line literal where whitespace is non-semantic and line-ending comments are ignored. + +The compiler will parse the contents of a regex literal using regex syntax outlined in *[Regex Construction][internal-syntax]*, diagnosing any errors at compile time. The capture types and labels are automatically inferred based on the capture groups present in the regex. Regex literals allows editors and source tools to support features such as syntax coloring inside the literal, highlighting sub-structure of the regex, and conversion of the literal to an equivalent result builder DSL (see *[Regex builder DSL][regex-dsl]*). + +A regex literal also allows for seamless composition with the Regex DSL, enabling lightweight intermixing of a regex syntax with other elements of the builder: + +```swift +// A regex for extracting a currency (dollars or pounds) and amount from input +// with precisely the form /[$£]\d+\.\d{2}/ +let regex = Regex { + Capture { /[$£]/ } + TryCapture { + /\d+/ + "." + /\d{2}/ + } transform: { + Amount(twoDecimalPlaces: $0) + } +} +``` + +This flexibility allows for terse matching syntax to be used when it's suitable, and more explicit syntax where clarity and strong types are required. + +Due to the existing use of `/` in comment syntax and operators, there are some syntactic ambiguities to consider. While there are quite a few cases to consider, we do not feel that the impact of any individual case is sufficient to disqualify the syntax. Some of these ambiguities require a couple of source breaking language changes, and as such the `/.../` syntax requires upgrading to a new language mode in order to use. + +## Detailed design + +### Named typed captures + +Regex literals have their capture types statically determined by the capture groups present. This follows the same inference behavior as [the DSL][regex-dsl], and is explored in more detail in *[Strongly Typed Captures][strongly-typed-captures]*. One aspect of this that is currently unique to the literal is the ability to infer labeled tuple elements for named capture groups. For example: + +```swift +func matchHexAssignment(_ input: String) -> (String, Int)? { + let regex = /(?[[:alpha:]]\w*) = (?[0-9A-F]+)/ + // regex: Regex<(Substring, identifier: Substring, hex: Substring)> + + guard let match = regex.matchWhole(input), + let hex = Int(match.hex, radix: 16) + else { return nil } + + return (match.identifier, hex) +} +``` + +This allows the captures to be referenced as `match.identifier` and `match.hex`, in addition to numerically (like unnamed capture groups) as `match.1` and `match.2`. This label inference behavior is not available in the DSL, however users are able to [bind captures to named variables instead][dsl-captures]. + +### Extended delimiters `#/.../#`, `##/.../##` + +Backslashes may be used to write forward slashes within the regex literal, e.g `/foo\/bar/`. However, this can be quite syntactically noisy and confusing. To avoid this, a regex literal may be surrounded by an arbitrary number of balanced number signs. This changes the delimiter of the literal, and therefore allows the use of forward slashes without escaping. For example: + +```swift +let regex = #/usr/lib/modules/([^/]+)/vmlinuz/# +// regex: Regex<(Substring, Substring)> +``` + +The number of `#` characters may be further increased to allow the use of e.g `/#` within the literal. This is similar in style to the raw string literal syntax introduced by [SE-0200], however it has a couple of key differences. Backslashes do not become literal characters. Additionally, a multi-line mode, where whitespace and line-ending comments are ignored, is entered when the opening delimiter is followed by a newline. + +```swift +let regex = #/ + usr/lib/modules/ # Prefix + (? [^/]+) + /vmlinuz # The kernel +#/ +// regex: Regex<(Substring, subpath: Substring)> +``` + +#### Escaping of backslashes + +This syntax differs from raw string literals `#"..."#` in that it does not treat backslashes as literal within the regex. A string literal `#"\n"#` represents the literal characters `\n`. However a regex literal `#/\n/#` remains a newline escape sequence. + +One of the primary motivations behind this escaping behavior in raw string literals is that it allows the contents to be easily transportable to/from e.g external files where escaping is unnecessary. For string literals, this suggests that backslashes be treated as literal by default. For regex literals however, it instead suggests that backslashes should retain their semantic meaning. This enables interoperability with regexes taken from outside your code without having to adjust escape sequences to match the delimiters used. + +With string literals, escaping can be tricky without the use of raw syntax, as backslashes may have semantic meaning to the consumer, rather than the compiler. For example: + +```swift +// Matches '\' * '=' * + +let regex = try NSRegularExpression(pattern: "\\\\w\\s*=\\s*\\d+", options: []) +``` + +In this case, the intent is not for the compiler to recognize any of these sequences as string literal escapes, it is instead for `NSRegularExpression` to interpret them as regex escape sequences. However this is not an issue for regex literals, as the regex parser is the only possible consumer of such escape sequences. Such a regex would instead be spelled as: + +```swift +let regex = /\\\w\s*=\s*\d+/ +// regex: Regex +``` + +Backslashes still require escaping to be treated as literal, however we don't expect this to be as common of an occurrence as needing to write a regex escape sequence such as `\s`, `\w`, or `\p{...}`, within a regex literal with extended delimiters `#/.../#`. + +#### Multi-line mode + +Extended regex delimiters additionally support a multi-line mode when the opening delimiter is followed by a new line. For example: + +```swift +let regex = #/ + # Match a line of the format e.g "DEBIT 03/03/2022 Totally Legit Shell Corp $2,000,000.00" + (? \w+) \s\s+ + (? \S+) \s\s+ + (? (?: (?!\s\s) . )+) \s\s+ # Note that account names may contain spaces. + (? .*) + /# +``` + +In this mode, [extended regex syntax][extended-regex-syntax] `(?x)` is enabled by default. This means that whitespace becomes non-semantic, and end-of-line comments are supported with `# comment` syntax. + +This mode is supported with any (non-zero) number of `#` characters in the delimiter. Similar to multi-line strings introduced by [SE-0168], the closing delimiter must appear on a new line. To avoid parsing confusion, such a literal will not be parsed if a closing delimiter is not present. This avoids inadvertently treating the rest of the file as regex if you only type the opening. + +### Ambiguities with comment syntax + +Line comment syntax `//` and block comment syntax `/*` will continue to be parsed as comments. An empty regex literal is not a particularly useful thing to express, but can be written as `#//#` if desired. `*` would be an invalid starting character of a regex, and therefore does not pose an issue. + +A parsing conflict does however arise when a block comment surrounds a regex literal ending with `*`, for example: + + ```swift + /* + let regex = /[0-9]*/ + */ + ``` + +In this case, the block comment prematurely ends on the second line, rather than extending all the way to the third line as the user would expect. This is already an issue today with `*/` in a string literal, though it is more likely to occur in a regex given the prevalence of the `*` quantifier. This issue can be avoided in many cases by using line comment syntax `//` instead, which it should be noted is the syntax that Xcode uses when commenting out multiple lines. + + +### Ambiguity with infix operators + +There is a minor ambiguity when infix operators are used with regex literals. When used without whitespace, e.g `x+/y/`, the expression will be treated as using an infix operator `+/`. Whitespace is therefore required for regex literal interpretation, e.g `x + /y/`. Alternatively, extended literals may be used, e.g `x+#/y/#`. + +### Regex syntax limitations + +In order to help avoid further parsing ambiguities, a `/.../` regex literal will not be parsed if it starts with a space, tab, or `)` character. Though the latter is already invalid regex syntax. This restriction may be avoided by using the extended `#/.../#` literal. + +#### Rationale + +This is due to 2 main parsing ambiguities. The first of which arises when a `/.../` regex literal starts a new line. This is particularly problematic for result builders, where we expect it to be frequently used, in particular within a `Regex` builder: + +```swift +let digit = Regex { + TryCapture(OneOrMore(.digit)) { Int($0) } +} +// Matches against + (' + ' | ' - ') + +let regex = Regex { + digit + / [+-] / + digit +} +``` + +Instead of being parsed as 3 result builder elements, the second of which being a regex literal, this is instead parsed as a single operator chain with the operands `digit`, `[+-]`, and `digit`. This will therefore be diagnosed as semantically invalid. + +To avoid this issue, a regex literal may not start with a space or tab character. This takes advantage of the fact that infix operators require consistent spacing on either side. + +If a space or tab is needed as the first character, it must be either escaped, e.g: + +```swift +let regex = Regex { + digit + /\ [+-] / + digit +} +``` + +or extended literal must be used, e.g: + +```swift +let regex = Regex { + digit + #/ [+-] /# + digit +} +``` + +The second ambiguity arises with Swift's ability to pass an unapplied operator reference as an argument to a function or subscript, for example: + +```swift +let arr: [Double] = [2, 3, 4] +let x = arr.reduce(1, /) / 5 +``` + +The `/` in the call to `reduce` is in a valid expression context, and as such could be parsed as a regex literal. This is also applicable to operators in tuples and parentheses. To help mitigate this ambiguity, a regex literal will not be parsed if the first character is `)`. This should have minimal impact, as this would not be valid regex syntax anyway. + +It should be noted that this only mitigates the issue, as it does not handle the case where the next character is a comma or right square bracket. These cases are explored further in the following section. + +### Language changes required + +In addition to ambiguities listed above, there are also some parsing ambiguities that require the following language changes in a new language mode: + +- Deprecation of prefix operators containing the `/` character. +- Parsing `/,` and `/]` as the start of a regex literal if a closing `/` is found, rather than an unapplied operator in an argument list. For example, `fn(/, /)` becomes a regex literal rather than 2 unapplied operator arguments. + +#### Prefix operators containing `/` + +We need to ban prefix operators starting with `/`, to avoid ambiguity with cases such as: + +```swift +let x = /0; let y = 1/ +let z = /^x^/ +``` + +Prefix operators containing `/` more generally also need banning, in order to allow prefix operators to be used with regex literals in an unambiguous way, e.g: + +```swift +let x = !/y / .foo() +``` + +Today, this is interpreted as the prefix operator `!/` on `y`. With the banning of prefix operators containing `/`, it becomes prefix `!` on a regex literal, with a member access `.foo`. + +Postfix `/` operators do not require banning, as they'd only be treated as regex literal delimiters if we are already trying to lex as a regex literal. + +#### `/,` and `/]` as regex literal openings + +As stated previously, there is a parsing ambiguity with unapplied operators in argument lists, tuples, and parentheses. Some of these cases can be mitigated by not parsing a regex literal if the starting character is `)`. However it does not solve the issue when the next character is `,` or `]`. Both of these are valid regex starting characters, and comma in particular may be a fairly common case for a regex. + +For example: + +```swift +// Ambiguity with comma: +func foo(_ x: (Int, Int) -> Int, _ y: (Int, Int) -> Int) {} +foo(/, /) + +// Also affects cases where the closing '/' is outside the argument list. +func bar(_ fn: (Int, Int) -> Int, _ x: Int) -> Int { 0 } +bar(/, 2) + bar(/, 3) + +// Ambiguity with right square bracket: +struct S { + subscript(_ fn: (Int, Int) -> Int) -> Int { 0 } +} +func baz(_ x: S) -> Int { + x[/] + x[/] +} +``` + +`foo(/, /)` is currently parsed as 2 unapplied operator arguments. `bar(/, 2) + bar(/, 3)` is currently parsed as two independent calls that each take an unapplied `/` operator reference. Both of these will become regex literals arguments, `/, /` and `/, 2) + bar(/` respectively (though the latter will produce a regex error). + +To disambiguate these cases, users will need to surround at least the opening `/` with parentheses, e.g: + +```swift +foo((/), /) +bar((/), 2) + bar(/, 3) + +func baz(_ x: S) -> Int { + x[(/)] + x[/] +} +``` + +This takes advantage of the fact that a regex literal will not be parsed if the first character is `)`. + + + +## Source Compatibility + +As explored above, two source breaking changes are needed for `/.../` syntax: + +- Deprecation of prefix operators containing the `/` character. +- Parsing `/,` and `/]` as the start of a regex literal if a closing `/` is found, rather than an unapplied operator in an argument list. For example, `fn(/, /)` becomes a regex literal rather than two unapplied operator arguments. + +As such, both these changes and the `/.../` syntax will be introduced in Swift 6 mode. However, projects will be able to adopt the syntax earlier by passing the compiler flag `-enable-bare-regex-syntax`. Note this does not affect the extended delimiter syntax `#/.../#`, which will be usable immediately. + +## Future Directions + +### Modern literal syntax + +We could support a more modern Swift-like syntax in regex literals. For example, comments could be done with `//` and `/* ... */`, and quoted sequences could be done with `"..."`. This would however be incompatible with the syntactic superset of regex syntax we intend to parse, and as such may need to be introduced using a new literal kind, with no obvious choice of delimiter. + +However, such a syntax would lose out on the familiarity benefits of standard regex, and as such may lead to an "uncanny valley" effect. It's also possible that the ability to use regex literals in the DSL lessens the benefit that this syntax would bring. + +## Alternatives Considered + +Given the fact that `/.../` is an existing term of art for regular expressions, we feel it should be the preferred delimiter syntax. It should be noted that the syntax has become less popular in some communities such as Perl, however we still feel that it is a compelling choice, especially with extended delimiters `#/.../#`. Additionally, while there has some syntactic ambiguities, we do not feel that they are sufficient to disqualify the syntax. To evaluate this trade-off, below is a list of alternative delimiters that would not have the same ambiguities. + +### Prefixed quote `re'...'` + +We could choose to use `re'...'` delimiters, for example: + +```swift +// Matches " = ", extracting the identifier and hex number +let regex = re'([[:alpha:]]\w*) = ([0-9A-F]+)' +``` + +The use of two letter prefix could potentially be used as a namespace for future literal types. It would also have obvious extensions to extended and multi-line literals using `re#'...'#` and `re'''...'''` respectively. However, it is unusual for a Swift literal to be prefixed in this way. We also feel that its similarity to a string literal might have users confuse it with a raw string literal. + +Also, there are a few items of regex grammar that use the single quote character as a metacharacter. These include named group definitions and references such as `(?'name')`, `(?('name'))`, `\g'name'`, `\k'name'`, as well as callout syntax `(?C'arg')`. The use of a single quote conflicts with the `re'...'` delimiter as it will be considered the end of the literal. However, alternative syntax exists for all of these constructs, e.g `(?)`, `\k`, and `(?C"arg")`. Those could be required instead. An extended regex literal syntax e.g `re#'...'#` would also avoid this issue. + +### Prefixed double quote `re"...."` + +This would be a double quoted version of `re'...'`, more similar to string literal syntax. This has the advantage that single quote regex syntax e.g `(?'name')` would continue to work without requiring the use of the alternative syntax or extended literal syntax. However it could be argued that regex literals are distinct from string literals in that they introduce their own specific language to parse. As such, regex literals are more like "program literals" than "data literals", and the use of single quote instead of double quote may be useful in expressing this difference. + +### Single letter prefixed quote `r'...'` + +This would be a slightly shorter version of `re'...'`. While it's more concise, it could potentially be confused to mean "raw", especially as Python uses this syntax for raw strings. + +### Single quotes `'...'` + +This would be an even more concise version of `re'...'` that drops the prefix entirely. However, given how close it is to string literal syntax, it may not be entirely clear to users that `'...'` denotes a regex as opposed to some different form of string literal (e.g some form of character literal, or a string literal with different escaping rules). + +We could help distinguish it from a string literal by requiring e.g `'/.../'`, though it may not be clear that the `/` characters are part of the delimiters rather than part of the literal. Additionally, this would potentially rule out the use of `'...'` as a future literal kind. + +### Magic literal `#regex(...)` + +We could opt for for a more explicitly spelled out literal syntax such as `#regex(...)`. This is a more heavyweight option, similar to `#selector(...)`. As such, it may be considered syntactically noisy as e.g a function argument `str.match(#regex([abc]+))` vs `str.match(/[abc]+/)`. + +Such a syntax would require the containing regex to correctly balance parentheses for groups, otherwise the rest of the line might be incorrectly considered a regex. This could place additional cognitive burden on the user, and may lead to an awkward typing experience. For example, if the user is editing a previously written regex, the syntax highlighting for the rest of the line may change, and unhelpful spurious errors may be reported. With a different delimiter, the compiler would be able to detect and better diagnose unbalanced parentheses in the regex. + +We could avoid the parenthesis balancing issue by requiring an additional internal delimiter such as `#regex(/.../)`. However this is even more heavyweight, and it may be unclear that `/` is part of the delimiter rather than part of an argument. Alternatively, we could replace the internal delimiter with another character such as ```#regex`...` ```, `#regex{...}`, or `#regex/.../`. However those would be inconsistent with the existing `#literal(...)` syntax and the first two would overload the existing meanings for the ``` `` ``` and `{}` delimiters. + +It should also be noted that `#regex(...)` would introduce a syntactic inconsistency where the argument of a `#literal(...)` is no longer necessarily valid Swift syntax, despite being written in the form of an argument. + +### Shortened magic literal `#(...)` + +We could reduce the visual weight of `#regex(...)` by only requiring `#(...)`. However it would still retain the same issues, such as still looking potentially visually noisy as an argument, and having suboptimal behavior for parenthesis balancing. It is also not clear why regex literals would deserve such privileged syntax. + +### Using a different delimiter for multi-line + +Instead of re-using the extended delimiter syntax `#/.../#` for multi-line regex literals, we could choose a different delimiter for it. Unfortunately, the obvious choice for a multi-line regex literal would be to use `///` delimiters, in accordance with the precedent set by multi-line string literals `"""`. This signifies a (documentation) comment, and as such would not be viable. + +### Reusing string literal syntax + +Instead of supporting a first-class literal kind for regex, we could instead allow users to write a regex in a string literal, and parse, diagnose, and generate the appropriate code when it's coerced to the `Regex` type. + +```swift +let regex: Regex = #"([[:alpha:]]\w*) = ([0-9A-F]+)"# +``` + +However we decided against this because: + +- We would not be able to easily apply custom syntax highlighting and other editor features for the regex syntax. +- It would require a `Regex` contextual type to be treated as a regex, otherwise it would be defaulted to `String`, which may be undesired. +- In an overloaded context it may be ambiguous or unclear whether a string literal is meant to be interpreted as a literal string or regex. +- Regex-specific escape sequences such as `\w` would likely require the use of raw string syntax `#"..."#`, as they are otherwise invalid in a string literal. +- It wouldn't be compatible with other string literal features such as interpolations. + +### No custom literal + +Instead of adding a custom regex literal, we could require users to explicitly write `try! Regex(compiling: "[abc]+")`. This would be similar to `NSRegularExpression`, and loses all the benefits of parsing the literal at compile time. This would mean: + +- No source tooling support (e.g syntax highlighting, refactoring actions) would be available. +- Parse errors would be diagnosed at run time rather than at compile time. +- We would lose the type safety of typed captures. +- More verbose syntax is required. + +We therefore feel this would be a much less compelling feature without first class literal support. + +[SE-0168]: https://github.com/apple/swift-evolution/blob/main/proposals/0168-multi-line-string-literals.md +[SE-0200]: https://github.com/apple/swift-evolution/blob/main/proposals/0200-raw-string-escaping.md + +[pitch-status]: https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/ProposalOverview.md +[regex-type]: https://github.com/apple/swift-evolution/blob/main/proposals/0350-regex-type-overview.md +[strongly-typed-captures]: https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/StronglyTypedCaptures.md + +[internal-syntax]: https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/RegexSyntax.md +[extended-regex-syntax]: https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/RegexSyntax.md#extended-syntax-modes + +[regex-dsl]: https://github.com/apple/swift-evolution/blob/main/proposals/0351-regex-builder.md +[dsl-captures]: https://github.com/apple/swift-evolution/blob/main/proposals/0351-regex-builder.md#capture-and-reference