RFC: Simplify rules on when traits can be used as objects

[nikomatsakis]

The current rules for when traits can be used as object types are a bit complex and hard to explain and understand. I propose the following:

• A trait is said to be "object-compatible" if it meets the following requirements:
  • No method uses the Self type more than once, including the receiver and return type positions.
  • No use of by-value Self
• For every object type @T, &T, or ~T, the trait T must be an object-compatible trait
• No "static" functions (see extension below)

I am not sure what's the best place to report errors. We can enforce this in various possible ways:

1. Report an error for every object type with an incompatible trait T.
2. Report an error whenever an object is created with an incompatible trait T.
3. Report an error when a method is called on an incompatible trait T.

Enforcing the rule at point (1) may result in a lot of errors but is probably the clearest thing. No matter what, I think we should definitely enforce the rule at point (2). It guarantees that no instances of illegal traits actually exist at runtime. We'll have to be careful around point (3) anyhow because users may try to call illegal methods; I think right now we ICE (see #5085).

Justifications

Why these rules? We need to prevent objects from being used in scenarios like this:

trait Eq {
    fn eq(&self, b: &Self) -> bool;
}

since we can't enforce this rule if we don't know what the Self type is.

Note that it's not enough to say "don't call eq()" because of generic code:

fn foo<A: Eq>(a: &A, b: &A) -> bool { a.eq(b) }

Unless we're careful, one could call foo() on an instance of @Eq since @Eq implements Eq.

What we do today

Today we do not permit eq() to be called and we say that if a trait is non-object-like, then @Eq does not implement Eq, thus preventing a generic method like foo() from being invoked. This is maximally flexible and sound but (I think) overly complex and hard to explain.

Possible Extensions

We could in principle allow return types of Self and "auto-box" them. This is an extension.

In some cases, we could also allow static functions, but we would need (a) auto-boxing for return values and (b) auto-unboxing on call. We then have to make sure that Self does not appear in a by-value position or as part of another type, like here:

trait Foo {
  fn foo(v: &Self) { ... } // OK, we can "auto-unwrap" in monomorphization
  fn foo(v: Self) { ... } // Not so good
  fn foo(v: Option<Self>) { ... } // Uh-oh
}

[nikomatsakis]

Closing in favor of #5087