- Type: Design proposal
- Author: Raul Raja
- Contributors: Tomás Ruiz López, Jorge Castillo, Francesco Vasco, Claire Neveu
- Status: New
- Prototype: initial implementation
The goal of this proposal is to enable compile time dependency resolution through extension syntax. Overall, we want to enable extension contract interfaces to be declared as constraints in function or class constructor arguments and enable the compiler to automatically resolve and inject those instances that must be provided with evidence in one of a given set of scopes. We'll cover these later in the Extension resolution order section. In the case of not having evidence of a required interface (program constraints), the compiler would fail and provide the proper error messages.
This would bring first-class named extension families to Kotlin. Extension families allow us to guarantee that a given data type (class, interface, etc.) satisfies behaviors (a group of functions) that are decoupled from the type's inheritance hierarchy.
Unlike the traditional subtype style composition where users are forced to extend and implement classes and interfaces, extension families favor horizontal composition based on compile time resolution between types and their extensions.
- Support compile-time verification of program dependencies (extensions).
- Enable nested extension resolution.
- Support compile-time verification of the correctness of a program give that behavioral constraints are raised to the interface types.
- Enable the definition of polymorphic functions whose constraints can be verified at compile time in call sites.
We propose to use the existing interface semantics, allowing for a generic definition of behaviors and their instances in the same style that's used to define interfaces.
package com.data
interface Repository<A> {
fun loadAll(): Map<Int, A>
fun loadById(id: Int): A?
fun A.save(): Unit
}The above declaration can serve as a target for implementations for any arbitrary type passed for A.
In the implementation below, we provide evidence that there is a Repository<User> extension available in scope, enabling both methods defined for the given behavior to work over the User type. As you can see, we're enabling a new keyword here: extension.
package com.data.instances
import com.data.Repository
import com.domain.User
extension object UserRepository : Repository<User> {
val storedUsers: MutableMap<Int, User> = mutableMapOf() // e.g. users stored in a DB
override fun loadAll(): Map<Int, User> {
return storedUsers
}
override fun loadById(id: Int): User? {
return storedUsers[id]
}
override fun User.save() {
storedUsers[this.id] = this
}
}You can also provide evidence of an extension using classes:
package com.data.instances
import com.data.Repository
import com.domain.User
extension class UserRepository: Repository<User> {
val storedUsers: MutableMap<Int, User> = mutableMapOf() // e.g. users stored in a DB
override fun loadAll(): Map<Int, User> {
return storedUsers
}
override fun loadById(id: Int): User? {
return storedUsers[id]
}
override fun User.save() {
storedUsers[this.id] = this
}
}In KEEP, as it’s coded now, extensions are named primarily with the purpose of supporting Java. We’d be fine with this narrower approach that we’re illustrating here, but would be open to iterating towards allowing definition through properties and anonymous classes if there’s a need for it.
Now that we've got the constraint definition (interface) and a way to provide evidence of an extension for it, we just need to connect the dots. Interfaces can be used to define constraints of a function or a class constructor. We use the with keyword for that.
fun <A> fetchById(id: Int, with repository: Repository<A>): A? {
return loadById(id) // Repository syntax is automatically activated inside the function scope!
}As you can see, the constraint syntax is automatically active inside the function scope, so we can call its functions at will. That's because we consider Repository a constraint of our program at this point. In other words, the program cannot work without it, it's a requirement. That means the following two functions would be equivalent:
// Kotlin + KEEP-87
fun <A> fetchById(id: Int, with repository: Repository<A>): A? {
return loadById(id)
}
// Regular Kotlin
fun <A> fetchById(id: Int, repository: Repository<A>): A? =
with (repository) {
return loadById(id)
}On the call site, we could use it as follows:
fetchById<User>(11829) // compiles since we got evidence of a `Repository<User>` in scope.
fetchById<Coin>(12398) // does not compile: No `Repository<Coin>` evidence defined in scope!All values can be passed to functions with extension parameters, or we can omit extension parameters and let the compiler resolve the suitable extensions for them. This makes the approach really similar to the way default arguments work in Kotlin.
fetchById<User>(11829) // compiles since we got evidence of a `Repository<User>` in scope.
fetchById<User>(11829, UserRepository()) // you can provide it manually.
fetchById<User>(11829, repository=UserRepository()) // you can provide it manually with named application.When with is used in class constructors, it is important to add val to extension class fields to make sure they are accessible in the scope of the class. Here, the with keyword adds the value to the scope of every method in the class. To showcase this, let's say we have a Validator<A>, like:
interface Validator<A> {
fun A.isValid(): Boolean
}In this scenario, the following classes would be equivalent:
// Kotlin + KEEP-87
extension class ValidatedRepository<A>(with val V: Validator<A>) : Repository<A> {
val storedUsers: MutableMap<Int, A> = mutableMapOf() // e.g. users stored in a DB
override fun loadAll(): Map<Int, A> {
return storedUsers.filter { it.value.isValid() }
}
override fun loadById(id: Int): A? {
return storedUsers[id]?.let { if (it.isValid()) it else null }
}
override fun A.save() {
storedUsers[generateKey(this)] = this
}
}
// Regular Kotlin
class ValidatedRepository<A>(val V: Validator<A>) : Repository<A> {
val storedUsers: MutableMap<Int, A> = mutableMapOf() // e.g. users stored in a DB
override fun loadAll(): Map<Int, A> {
with (V) {
return storedUsers.filter { it.value.isValid() }
}
}
override fun loadById(id: Int): A? {
with (V) {
return storedUsers[id]?.let { if (it.isValid()) it else null }
}
}
override fun A.save() {
storedUsers[generateKey(this)] = this
}
}As you can see in the first example, A.isValid() becomes available inside the method’s scope automatically. The equivalent version of doing this without KEEP-87 is to manually add with (V) inside each method, as you can see in the second example.
Constraint interface declarations and extension evidences can encode further constraints on their type parameters so that they can be composed nicely:
package com.data.instances
import com.data.Repository
import com.domain.User
data class Group<A>(val values: List<A>)
extension class GroupRepository<A>(with val repoA: Repository<A>) : Repository<Group<A>> {
override fun loadAll(): Map<Int, Group<A>> {
return repoA.loadAll().mapValues { Group(it.value) }
}
override fun loadById(id: Int): Group<A>? {
return repoA.loadById(id)?.let { Group(it) }
}
override fun Group<A>.save() {
this.items.map { repoA.run { it.save() } }
}
}The above extension provides evidence of a Repository<Group<A>> as long as there is a Repository<A> in scope. The Call site would look like:
fun <A> fetchGroup(with repo: GroupRepository<A>) = loadAll()
fun main() {
fetchGroup<User>() // Succeeds! There's evidence of Repository<Group<A>> and Repository<User> provided in scope.
fetchGroup<Coin>() // Fails! There's evidence of Repository<Group<A>> but no evidence of `Repository<Coin>` available.
}We believe that the encoding proposed above fits nicely with Kotlin's philosophy of extensions and will reduce the boilerplate compared to other languages that also support compile-time dependency resolution.
- Add
withto require evidence of extensions in both function and class/object declarations. - Add
extensionto provide instance evidence for a given interface.
Usage of these language changes are demonstrated by the previous and below examples:
extension class GroupRepository<A>(with R: Repository<A>) : Repository<Group<A>> {
/* ... */
}fun <A> fetch(id: String, with R: Repository<A>): A = loadById(id) // function position using parameter "R"extension object UserRepository : Repository<User> {
val storedUsers: MutableMap<Int, User> = mutableMapOf() // e.g. users stored in a DB
override fun loadAll(): Map<Int, User> {
return storedUsers
}
override fun loadById(id: Int): User? {
return storedUsers[id]
}
override fun User.save() {
storedUsers[this.id] = this
}
}extension class UserRepository: Repository<User> {
val storedUsers: MutableMap<Int, User> = mutableMapOf() // e.g. users stored in a DB
override fun loadAll(): Map<Int, User> {
return storedUsers
}
override fun loadById(id: Int): User? {
return storedUsers[id]
}
override fun User.save() {
storedUsers[this.id] = this
}
}Classical interfaces only permit their implementation at the site of a type definition. Compiler extension resolution patterns typically relax this rule and allow extension evidences to be declared outside of the type definition. When relaxing this rule, it is important to preserve the coherency we take for granted with classical interfaces. For those reasons, constraint interfaces must be provided in one of the following scopes (in strict resolution order):
- Arguments of the caller function.
- Companion object for the target type (User).
- Companion object for the constraint interface we're looking for (Repository).
- Subpackages of the package where the target type (User) to resolve is defined, under the same gradle module. The extension needs to be marked as
internal. - Subpackages of the package where the constraint interface (Repository) is defined, under the same gradle module. The extension needs to be marked as
internal.
All other instances are considered orphan instances and are not allowed. See Appendix A for a modification to this proposal that allows for orphan instances.
Additionally, a constraint extension must not conflict with any other pre-existing extension for the same constraint interface; for the purpose of checking this, we use the normal resolution rules. That's what we refer to as compiler "coherence".
It looks into the caller function argument list for an evidence of the required extension. Here, bothValid() gets a Validator<A> passed in so, whenever it needs to resolve it for the inner calls to validate(), it'll be able to retrieve it from its own argument list.
fun <A> validate(a: A, with validator: Validator<A>): Boolean = a.isValid()
fun <A> bothValid(x: A, y: A, with validator: Validator<A>): Boolean = validate(x) && validate(y)In case there's no evidence at the caller function level, we'll look into the companion of the target type. Let's say we have an extension of Validator<User>:
package com.domain
data class User(val id: Int, val name: String) {
companion object {
extension class UserValidator(): Validator<User> {
override fun User.isValid(): Boolean {
return id > 0 && name.length > 0
}
}
}
}That'll be enough for resolving the extension.
In case neither evidence exists in the companion of the target type, we'll look in the companion of the constraint interface:
interface Validator<A> {
fun A.isValid(): Boolean
companion object {
extension class GroupValidator<A>(with val userValidator: Validator<User>) : Validator<Group> {
override fun Group.isValid(): Boolean {
for (x in users) {
if (!x.isValid()) return false
}
return true
}
}
}
}The next step would be to look into the subpackages of the package where the target type (User) is declared. It will simply look in subpackages under the current gradle module, it doesn't support cross-module definitions. These extensions must be flagged as internal.
package com.domain.repository
import com.domain.User
internal extension object UserRepository : Repository<User> {
val storedUsers: MutableMap<Int, User> = mutableMapOf() // e.g. users stored in a DB
override fun loadAll(): Map<Int, User> {
return storedUsers
}
override fun loadById(id: Int): User? {
return storedUsers[id]
}
override fun User.save() {
storedUsers[this.id] = this
}
}Here we got a Repository<User> defined in a subpackage of com.domain, where the User type is defined.
The last place to look would be the subpackages of the package where the constraint interface is defined. It will simply look in subpackages under the current gradle module, it doesn't support cross-module definitions. These extensions must be flagged as internal.
package com.data.instances
import com.data.Repository
import com.domain.User
internal extension object UserRepository : Repository<User> {
val storedUsers: MutableMap<Int, User> = mutableMapOf() // e.g. users stored in a DB
override fun loadAll(): Map<Int, User> {
return storedUsers
}
override fun loadById(id: Int): User? {
return storedUsers[id]
}
override fun User.save() {
storedUsers[this.id] = this
}
}Here, the constraint is resolved by finding a valid evidence for it in the com.data.instances, which a subpackage of com.data, where our constraint Repository is defined.
We've got a CallChecker in place to report inlined errors using the context trace. That allows us to report as many errors as possible in a single compiler pass. Also provide them in two different formats:
The checker is running whenever you’re coding and proper unresolvable extension errors can be reported within IDEA inspections.
You will also get a report of those errors once you hit the "compile" button or run any compile command.
KEEP-87 is currently deployed to our own Idea plugin repository over Amazon s3. To use it:
- Download the latest version of IntelliJ IDEA 2018.2.4 from JetBrains
- Go to
preferences->pluginssection. - Click on "Manage Plugin Repositories".
- Add our Amazon s3 plugin repository as seen in the image below.
- Now browse for the "keep87" plugin.
- Install it.
- Download and run The Keep87Sample project on that IntellIJ instance.
- Clone Our Kotlin fork and checkout the keep-87 branch.
- Follow the instructions on the README to configure the necessary JVMs.
- Follow the instructions on the README to open the project in IntelliJ IDEA.
- Once you have everything working, you can run a new instance of IntelliJ IDEA with the new modifications to the language by executing
./gradlew runIde. There is also a pre-configured run configuration titled IDEA that does this. - It will open a new instance of the IDE where you can create a new project and experiment with the new features of the language. You can also download The Keep87Sample project ) that includes some sample code that you can try out.
Some scenarios aren’t covered yet, given we’re lacking some knowledge about how subtyping resolution rules are coded in the Kotlin compiler. These scenarios would be required for a fully working compile-time extension resolution feature and are described in detail here.
Inlined functions get into trouble when it comes to capturing resolved extensions. The problem is described here.
Ideally, we'd enable users to provide extensions also using val and fun. They'd look similar to:
// Simple fun extension provisioning
extension fun userRepository(): Repository<User> = object : Repository<User>() {
/* ... */
}
// Nested fun extension provisioning (would require both to resolve).
extension fun userValidator(): Validator<User> = UserValidator()
extension fun userRepository(with validator: Validator<User>) : Repository<User> = UserRepository(validator)
// Simple extension provisioning
extension val userRepository: Repository<User> = UserRepository()
// Nested extension provisioning (would require both to resolve).
extension val userValidator: Validator<User> = UserValidator()
extension val userRepository: Repository<User> = UserRepository(userValidator)In nested extension provisioning, all nested extensions would be required from the caller scope, but not necessarily all provided in the same resolution scope. E.g: Repository<User> could be provided in a different resolution scope than Validator<User>, and the program would still compile successfully as long as both are available.
We have additional complications when you consider multi-parameter constraint interfaces.
package foo.repo
// I stands for the index type, A for the stored type.
interface Repository<I, A> {
...
}package data.foo
data class Id(...)
extension class RepoIndexedById<A> : Repository<Id, A> {
...
}package data.foo.user
data class User(...)
extension class UserRepository<I> : Repository<I, User> {
...
}The above instances are defined alongside their respective type definitions and yet they clearly conflict with each other. We will also run into quandaries once we consider generic types. We can crib some prior art from Rust1 to help us out here.
To determine whether an extension definition is a valid type-side implementation, we'd need to perform the following check:
- A "local type" is any type (but not type alias) defined in the current file (e.g. everything defined in
data.foo.userif we're evaluatingdata.foo.user). - A generic type parameter is "covered" by a type if it occurs within that type (e.g.
MyTypecoversTinMyType<T>but notPair<T, MyType>). - Write out the parameters to the constraint interface in order.
- The parameters must include a type defined in this file.
- Any generic type parameters must occur after the first instance of a local type or be covered by a local type.
If an extension meets these rules it is a valid type-side implementation.
Orphan implementations are a subject of controversy. Combining two libraries - one defining a target type (User), the other defining an interface (Repository) - is a feature that many programmers have longed for. However, implementing this feature in a way that doesn't break other features of interfaces is difficult and drastically complicates how the compiler works with those interfaces.
Orphan implementations are the reason that other implementations of this approach have often been described as "anti-modular", as the most common way of dealing with them is through global coherence checks. This is necessary to ensure that two libraries have not defined incompatible extensions of a given constraint interface.
Relaxing the orphan rules is a backwards-compatible change. If this proposal is accepted without permitting orphans, it will be useful to consider how they could be added in the future.
Ideally, we want to ban orphan implementations in libraries but not in executables; this allows a programmer to manually deal with coherence in their own code but prevents situations where adding a new library breaks code.
A simple way to allow orphan extensions is to replace the file-based restrictions with package-based restrictions. Because there are no restrictions on packages, it is possible to do the following:
// In some library foo
package foo.collections
extension class Repository<A> {
...
}// In some application that uses the foo library
package foo.collections
extension object : Repository<Int> {
...
}This approach wouldn't forbid orphan extensions in libraries but, it would highly discourage libraries from providing them, as this would involve writing code in the package namespace of another library.
An alternate approach is to require that orphan extensions be marked internal. The full rules would be as follows:
- All orphan extensions must be marked
internal. - All code which closes over an internal extension must be marked internal. Code closes over a constraint interface extension if it contains a static reference to such an extension.
- Internal extensions defined in the same module are in scope for the current module.
- Internal extensions defined in other modules are not valid for constraint interface resolution.
This approach works well but it has a few problems.
- It forces applications that use orphan extensions to mark all their code as internal, which is a lot of syntactic noise.
- It complicates the compiler's resolution mechanism since it's not as easy to enumerate definition sites.
The first problem actually leads us to a better solution.
Currently, Kotlin doesn't make use of Java 9 modules but, it's easy to see how they could eventually replace Kotlin's internal modifier. The rules for this approach would be the same as the internal-based approach; code which uses orphans is not allowed to be exported.