Split apart the grammar for statements and items

[CodaFi]

TSPL has referred to declarations and expressions as "statements", while
the Swift Compiler refers to this general class of syntax
as "items". Split the statement and item productions apart.

[CodaFi]

cf

New Parser -> https://github.com/apple/swift-syntax/blob/main/Sources/SwiftParser/TopLevel.swift#L126

Old Parser -> https://github.com/apple/swift/blob/bde58e7d584873367fe549f27068cbe4e7387492/lib/Parse/ParseStmt.cpp#L274

[krilnon]

This might benefit from a more detailed discussion of some sort; especially given how we structure the different parts of the reference portions of the book. Are folks over in the compiler world generally happy with the word "item" as a useful top-level term in grammar here? In contrast to the familiar PL terms of art statement and expression, item feels like it doesn't carry much explanatory weight for readers. But maybe that's just me. Let's talk about this more, either here in the issue or elsewhere.

[CodaFi]

There's several questions here so

Are folks over in the compiler world generally happy with the word "item" as a useful top-level term in grammar here?

It is a term of art. Perhaps block-item would be more descriptive, but it's also a bit odd to use "block" as a qualifier because of script-mode and globals in general. It's an artifact of the C++ parser that items at the top level are wrapped (or not) in top level code declaration ASTs that function as block-like scopes.

It's worth noting that we're not alone here in our usage of this term

Rust refers to crate and module-level syntactic constructs as items.
OCaml refers to compilation-unit level syntax as "module items"

In contrast to the familiar PL terms of art statement and expression

It's worth understanding why languages that e.g. choose to make declarations and statements a subordinate production of their statement grammars do so, and why Swift is different. Take Kotlin, which is a near neighbor to us in terms of syntax. They also have a script-mode equivalent, and they choose to structure their grammar in this manner. To my knowledge, Kotlin does not have a notion of compile-time configuration akin to Swift's #if construct. This means they truly can get away with representing the contents of their braced blocks as simply a list of statements, embedding declarations and expressions as necessary.

For Swift, the matter is not so simple. The context a #if occurs in changes the set of valid constructs that such a production's body can refer to. In item context, a #if body can contain items.

#if Foo
func foo() {}
bar()
baz * quux
#endif

In declaration context, this production may only contain declarations

class Foo {
#if Foo
  func foo() {} // OK
  bar() // parse error
  baz * quux // parse error
#endif
}

In expression context, this production may only involve postfix expressions

class Foo {
  func foo() {
    bar
#if Foo
  .baz() // OK
  func foo() {} // parse error
  bar() // parse error
  baz * quux // parse error
#endif
  }
}

In the context of switch statements, this production may only involve case statements and their bodies, not any kind of statement, and not any kind of item

class Foo {
  func foo() {
    switch bar {
#if Foo
  func foo() {} // parse error
  bar() // parse error
  baz * quux // parse error
  case .quux: // OK
    break
#endif
  }
}

We must be able to distinguish these three kinds of productions from one another, and also reason about when all three are allowed. For that, we cannot conflate declarations, expressions, and statements so freely.

item feels like it doesn't carry much explanatory weight for readers

This is a good point. It kind of feels like a cop-out from a descriptive point of view. Programming language grammars are not really designed for the kind of comprehensibility that you describe here as a goal. We have a chance to amend that, certainly, if we can come up with another kind of syntactic category here.

[amartini51]

Marking this as a draft while we continue these discussions.