what is the top-level scope in a source file, and what names are found there?

[zygoloid]

Originally raised in a Discord discussion.

Our status quo rule is that the top-level scope in a source file is the package scope -- that is the scope in which new unqualified declarations introduce names -- but that only names declared in the same source file are found by unqualified lookups in that scope. This leads to some surprising behavior where a (re)declaration in that scope can make information from a different library or source file visible:

package MyPackage library "X" api;
namespace MyNamespace;
namespace OurNamespace;
fn Foo() {}
fn MyNamespace.Bar() {}
fn OurNamespace.Baz() {}

package MyPackage impl;
import MyPackage library "X";
namespace OurNamespace;
fn DoStuff() {
  // Error, `Foo` not declared, did you mean `MyPackage.Foo`?
  Foo();
  // Error, `MyNamespace` not declared, did you mean `MyPackage.MyNamespace.Bar`?
  MyNamespace.Bar();
  // OK! Finds `OurNamespace` declared above, and finds `Baz` from library "X".
  OurNamespace.Baz();
}

And:

package MyPackage library "X" api;
class C {};

package MyPackage impl;
import MyPackage library "X";
// Error, what C?
let error: C;
class C;
// But after only a forward declaration it's now a complete type.
let ok: C;

It's been suggested that we should move away from this rule, and towards a model where unqualified declarations introduce names into the same scope that unqualified lookup looks into. Three models have been proposed:

1. The file is in its own file scope, and the package is a scope within that, just like a namespace. Exported declarations are written as fn MyPackage.Foo(), and anything written at file scope without package name qualification is file local. This has the advantage of giving a default that's never silently the wrong thing (you don't accidentally export things / declare an external symbol), but the disadvantage of making the interface of a package harder to read by repeating the package name a lot.

2. The file scope is the package scope. Imports consistently make new names appear in their declared scopes. So after an import of the same package, you may have new top-level names, because the top level is the package scope.

3. The file scope is library scope. An impl file starts with the names from its api file visible. Imports don't change the set of names visible because you don't import your own library. The package scope is a scope within the library scope, just like a namespace. An exported declaration in an api file also injects a corresponding name into the package scope (as if by a namespace declaration for a namespace, and as if by an alias for anything else). This means that a type can only be declared in a single library within a package, never forward-declared in one library and defined in another, and that an overload set can't be split across libraries.

From discussion on Discord, we didn't like (1) due to the significant ceremony of mentioning your own package name whenever declaring any part of an exported entity. (3) is more complex than (2), and has more ceremony than (1) when mentioning parts of other libraries in your package; however, it gives us a library scope that permits a library to define its own private names that are shared within the library, and it gives us the property that every unqualified name finds a library-local result (unlike (1) and the status quo which give a file-local result).

[josh11b]

My description of model 3 is: "unqualified lookup finds names in this library; to find exported names from other libraries in this package, qualify them with the name of the package". I do think we want a rule that can be explained clearly and succinctly, do you think my description qualifies?

[zygoloid]

I think one of the biggest risks of confusion with model 3 is that each namespace declaration observably introduces two different namespaces -- one in the library and one in the package. For me, your description doesn't sufficiently capture that. Example:

package Foo library "bar" api;
// Library "baz" might also introduce namespace X.
import Foo library "baz";
namespace X;
// A and B (might) have different contents, so are not the same namespace.
// Contents of X from library "baz" are only visible in B, not A.
alias A = X;
alias B = Foo.X;
// OK, exported declaration is added to both A and B.
fn X.F() { A.F(); B.F(); }
// Presumably OK, name added to both A and B.
fn A.G() {}
// OK or not? If this is OK, should A.H() be valid?
fn B.H() {}
// Similarly these: OK or not? And can they be found by J() or X.K() or A.K()?
fn Foo.J() {}
fn Foo.X.K() {}

[jonmeow]

Just to be sure, under #3 you mention implicit imports of api by impl, and that's presently the design. I think that's separate from how name lookup occurs though -- are you just trying to emphasize that the impl sees names in library scope declared by the api?

What would you think of somewhere between (2) and (3), where public names are in the package scope, but there's a subtle library scope for private things? In particular, like (2) names appear in their declared scopes, and like (3) there's a library scope. Unlike (3), the library scope is somewhat separated from the package scope. Name lookup order could be library scope then package scope. Because an api file is implicitly public, it defaults to package scope; because an impl file is implicitly private, it defaults to library scope. For a name in library scope, it could canonically be library.<name> with library keyword reuse.

To try to explain how that may work:

package Foo library "bang" api;

// This name is canonically `Foo.Print()`.
fn Print() { ... }

// This name is canonically `library.Lookup()`.
private fn Lookup() { ... }

package Foo library "bang" impl;

// Each of these are valid.
Print();
Foo.Print();
Lookup();
library.Lookup();

// These are invalid.
library.Print();
Foo.Lookup();

In your example for rule 3, namespace X; would canonically be Foo.X because it's in the api file, which defaults public.

Regarding the class C example, the forward declaration would still be incomplete because it's in a different scope. (also, separately, maybe we can disallow forward declarations that come after a matching definition in the same scope?)

One concern might be that, due to the difference in defaults, a name in the api file might not match the scope of the impl file. However, the impl file sees the api, and it would be an error to mismatch regardless. Additionally, I think it may be weird if the impl could independently add things to the package scope anyways, as those things wouldn't be visible outside the impl file.

For example:

package Foo library "bang" api;

// Forward declaration.
fn Foo();

package Foo library "bang" impl;

// Three possibilities:
// 1. This is detected as the definition for the `api` file, and is package scope instead
//   of the library scope it may look like at a glance. (but either that's how things kind
//   of work anyways, or we actually need `private` for things in the impl that aren't
//   definitions for the `api` anyways.
// 2. A keyword is required to mark this public and/or an implementation of the `api` fn.
//   (or the `impl` file requires `private` on non-public things)
// 3. We error in compile because there's no definition for the API file's `Foo()`.
fn Foo() {
  // implementation
}

[zygoloid]

@jonmeow Can you describe how your amended rule would treat namespaces? Specifically for examples like the one in my reply to @josh11b I'd like to be clear on how the package versus library split interacts with the identity of namespaces and how we look for and declare names within them.

[jonmeow]

@jonmeow Can you describe how your amended rule would treat namespaces? Specifically for examples like the one in my reply to @josh11b I'd like to be clear on how the package versus library split interacts with the identity of namespaces and how we look for and declare names within them.

I think there are few options routes...

Option (A) would be that namespace X actually declares library.X and Foo.X as distinct namespaces. Non-specific name lookup (e.g., X.Bar(), or just Bar() from something within X) performs lookup in library then Foo, mirroring top-level lookups.

i.e.:

package Foo library "bar" api;
// Library "baz" might also introduce namespace X.
import Foo library "baz";

// This is an API file, so re-declares `Foo.X` and also defines `library.X`.
namespace X;

// This has the contents of `library.X`, and name lookup should continue to `Foo.X`.
alias A = X;
// This is the more awkward case, wherein it skips `library.X` and only sees `Foo.X` due to name lookup ordering.
alias B = Foo.X;

// OK, exported declaration is visible to both A and B.
fn X.F() { A.F(); B.F(); }

// Should probably be OK, name is public and added to Foo.
fn A.G() {}

// OK, name is public and added to Foo.
fn B.H() {}

// OK, name is public and added to Foo.
fn Foo.J() {}
fn Foo.X.K() {}

// Declares `library.X.Baz`.
private fn X.Baz() {}

// Declares `Foo.X.Bang`
fn X.Bang() {
  // Should probably be made to find `library.X.Baz()` for ergonomics --
  // but this is weird, so see option (B).
  Baz();
}

Option (B), which I would advocate for, is that library namespaces are disallowed. A namespace only exists as a concept for a package. Libraries are already very limited in scope/conflicts (they are a single api and impl, whereas a namespace can be multi-library), and they are by definition private, and so there is no benefit to offering a split semantic.

i.e.:

I think of the namespaces as distinct: i.e., library.X and Foo.X are separate namespaces. I grant this builds a little weirdness, but equally, I might try to justify it that the library scope shouldn't be namespaced (perhaps we can disallow library-private namespaces entirely, since libraries exist as a scope on their own?)

To be explicit in your example:

package Foo library "bar" api;
// Library "baz" might also introduce namespace X.
import Foo library "baz";

// Declares `Foo.X` unambiguously.
namespace X;

// References to `Foo.X`.
alias A = X;
alias B = Foo.X;

// OK and simple.
fn X.F() { A.F(); B.F(); }
fn A.G() {}
fn B.H() {}
fn Foo.J() {}
fn Foo.X.K() {}

// Invalid because private functions can't be in a namespace.
private fn X.Baz() {}

// Privates must be top-level.
private fn Baz() {}

// Declares `Foo.X.Bang`
fn X.Bang() {
  // Again, not really ambiguous for name lookup.
  Baz();
}

[josh11b]

I am totally fine with @jonmeow 's idea that the library scope and private declarations would not be allowed to have namespaces. My main question with this latest proposal is how it handles names in the same package in a different library. My reading of what you wrote is that lookup would find names in the package scope without any qualification. That worries me a bit, just because it is a bit less explicit and a bit more context / places to look to understand what a name means. I do think these are somewhat separable questions:

• NamespacePrivate: Are private symbols allowed to be in a namespace?
• PackageQualifier: Do you have to add a package qualifier for names in another library in the same package?

Our current proposed rules are:

0. new unqualified declarations introduce names in package scope, lookup is into file scope; PackageQualifier = Yes, but inconsistently when there are namespaces involved
1. the file is in its own file scope, and the package is a scope within that; PackageQualifier = Yes
2. file scope is the package scope; PackageQualifier = No
3. file scope is library scope, public symbols also in package scope; PackageQualifier = Yes; some concerns about the complexity of semantics with namespaces might be addressed by saying NamespacePrivate = No
4. file scope is first library scope then package scope, public symbols go in package scope, private symbols go in library scope; PackageQualifier = No, and @jonmeow 's preference (B) is for NamespacePrivate = No

[jonmeow]

My reading of what you wrote is that lookup would find names in the package scope without any qualification.

Yes. This is me looking between options #2 and #3 and thinking that, ergonomically, maybe we should do PackageQualifier = No. However, we could do PackageQualifier = Yes and still do NamespacePrivate = No in order to address zygoloid's problematic namespace example.

NamespacePrivate: Are private symbols allowed to be in a namespace?

I'd suggest instead "LibraryPrivateInNamespace": Are library-private symbols allowed to be in a namespace? (renaming "NamespacePrivate", including in my above response)

The reason for this is a separate concept of namespace-private, e.g.:

package Pkg api;
namespace Fn;

// Declares a symbol `library.Baz()` that is only visible to other members of this library.
private fn Baz();

// Declares a symbol `Pkg.NS.Bar()` that is only visible to other members of `Pkg.NS`.
// This may remind people of class `private`.
private fn NS.Bar() {}

I'm only mentioning this to note the related concept that may be desirable, but I don't think it should be decided here (although a decision of LibraryPrivateInNamespace = Yes would probably make namespace-private overlap syntactically in bad ways, ruling out the latter feature).

[zygoloid]

we could do PackageQualifier = Yes and still do NamespacePrivate = No in order to address zygoloid's problematic namespace example.

Hm, I'm not sure we have a great option here, even with NamespacePrivate = No. I can see four possible choices:

1. OurNamespace is the same as MyPackage.OurNamespace, which results in one problematic example where declaring the namespace makes things from other libraries visible within it. (I think this is PackageQualifier = No.)
2. OurNamespace is a different namespace from MyPackage.OurNamespace, which results in another problematic example where it's confusing that we have two observably-different namespaces. (I think this is PackageQualifier = Yes.)
3. OurNamespace is invalid and one must write MyPackage.OurNamespace, which has bad ergonomics.
4. Don't allow a namespace to be spread across multiple libraries. That doesn't seem feasible given how we intend code to be organized.

[jonmeow]

I assume in your third choice, you're suggesting that all references write MyPackage.OurNamespace in order to avoid ambiguity. If so, a a variation with slightly better ergonomics (particularly if the namespace is only in one file) would be to require a namespace be defined in a single place (much like other entities), and everything else should alias that central definition. i.e., one file does namespace OurNamespace;. In every other file, the same would be a name conflict. They should do alias OurNamespace = MyPackage.OurNamespace;.

Note though, I think it's really the namespace example that pushes me to want to do PackageQualifier = no -- I agree choices aren't too great, and I think developers will generally prefer the ergonomics of not writing MyPackage for everything.

[josh11b]

Hm, I'm not sure we have a great option here, even with NamespacePrivate = No. I can see four possible choices:

1. OurNamespace is the same as MyPackage.OurNamespace, which results in one problematic example where declaring the namespace makes things from other libraries visible within it. (I think this is PackageQualifier = No.)

I don't think @jonmeow 's proposal has this problem, because things from other libraries in the same package are visible independent of whether the namespace was declared locally. That is also the downside of PackageQualifier = No.

[zygoloid]

Ah, right, yes. If we give up on trying to provide the property that all unqualified names have a local (same file or at least same library) declaration, we can avoid these problems. That property doesn't seem to mesh very well with also wanting a package scope that contains a mixture of local and non-local names.

If we want to go that way, I think it'd be simpler to also stop trying to distinguish package, library, and file scopes, and say they're all the same thing: that the top level of a file is its package scope. And separately, private names in a library are private to that library and not visible outside, but that this is accomplished by something more like a linkage and visibility rule rather than by having a distinct scope.

With that approach, a MyPackage. qualifier would always be redundant except perhaps to avoid a name conflict. In fact, I think we should consider dropping the injection of a name MyPackage entirely, in favor of using something like package.Foo as syntax for explicitly naming the top level.

[jonmeow]

FWIW, an argument towards injecting MyPackage (regardless of package.Foo syntax) would be that import MyPackage has the same consequences as import OtherPackage for name insertion. package.Foo may still be preferred for explicitness (and does allow for code to be moved more easily cross-package), but some generic way of specifying the top-level might be better because OtherPackage probably also needs an explicit way to reference it, and package.OtherPackage seems awkward in that context.

Note, a counter-argument for name insertion is that imports from the same package could use some special syntax as well, e.g. import package;, making it clear that it behaves differently versus import OtherPackage;. But then it sort of gets into how much we want references to the current package to be special.

[chandlerc]

After reading all of this, I lean toward (2) and as @zygoloid says stopping distinguishing package / library / file scopes. And I somewhat like the idea from @jonmeow of making same-package imports visibly different. I wonder about just omitting the package entirely: import library ... -> import of a library in the current package, and has different name-introduction properties compared to importing from another package.

[zygoloid]

I find myself also leaning towards rule (2). Specifically and concretely, the rule set I would propose is:

• The top-level scope in a package is the scope of the package.
  • Within this scope (and its sub-namespaces), all visible names from the same package appear. This includes names from the same file, names from the api file of a library when inside an impl file, and names from imported libraries of the same package.
  • private in an api file means "private to this library". private in an impl file means "private to this file". Private names don't conflict with names outside the region they're private to: two different libraries can have different private names foo without conflict.
  • Private names in namespaces are permissible. These seem important: otherwise, the key purpose of namespaces as distinct from name prefixes, namely that you can name their contents without qualification, is lost for private entities.
  • The hiding of private names is separate from scoping, and behaves consistently within the current package and in other packages: in both cases there is logically exactly one scope for a package and one scope for each namespace within the package, and different subsets of those scopes might be visible depending on what is imported.
• The name of the current package is not inserted within the file scope (that is, as effectively a member of itself) -- there is no "injected package name".
  • The syntax package PackageName ... does not introduce the name PackageName.
  • In scopes where package members might be shadowed by something else, the syntax package.Foo can be used to name the Foo member of the current package. (Note that PackageName.Foo is not a complete solution for this problem, specifically if the package name itself is shadowed.)
  • The syntax import PackageName library ... introduces the name PackageName as a private name naming the given package. It cannot be used to import libraries of the current package.
  • The syntax import library ... imports a library of the current package, and consequently adds all the public top-level names within the given library to the top-level scope of the current file, and similarly for names in namespaces.
  • The two distinct forms of import are important to emphasize that import Blah consistently makes the name Blah visible (and doesn't affect any other names), and import library ... makes some (not explicitly stated) set of names visible.
• Unqualified name lookup walks the semantically-enclosing scopes, not only the lexically-enclosing ones. So when a lookup is performed within fn MyNamespace.MyClass.MyNestedClass.MyFunction(), we will look in MyNestedClass, MyClass, MyNamespace, and the package scope, even when the lexically-enclosing scope is the package scope.
  • As above, this appears to be necessary in order for namespaces to have utility beyond being fancy name prefixes: when one namespace member wants to mention another, it should not need to specify the namespace name as a qualifier. But especially because namespaces don't have lexical scopes in our current conception of them, a lexical rule would require such qualification.

Compared to the above discussion, this is model (2), with NamespacePrivate=Yes, PackageQualifier=No.

[chandlerc]

This seems to be a good fit for the direction all the discussion has leaned (including @jonmeow I think), and I'm very happy with it. With two of the leads and no real strong concerns, I'll mark this as decided. If I've misunderstood your position @jonmeow and you have concerns here, feel free to re-open.

[jonmeow]

Filed #2001 to track the need for a proposal.