Semantic API

[xeno-by]

Scala.meta 1.5 supports inspections of the structure of Scala code (syntax and tokens), but it doesn't provide any facilities to inspect semantics (what's the type of this fragment of code? what does this name point to? etc). We had this in scala.reflect, and we need to have this in scala.meta, too, because that's one of the most basic features one expects in a metaprogramming framework.

In the past, we've made two unsuccessful attempts at implementing the semantic API for the Scala compiler (first and second). Both of them were based on the concept of a scala.reflect -> scala.meta tree converter, i.e. a module that would convert attributed compiler trees into equivalent scala.meta trees with semantic information encoded in a platform-independent way. Long story short, eventually we realized that it is prohibitively hard to implement a useful converter, so we had to start looking for other ways to build a semantic API (see #148 for a detailed explanation).

Around Scala eXchange 2016, we had an idea to give up on converters altogether and obtain semantic information about scala.meta trees using positions. Instead of eagerly converting attributed scala.reflect trees into attributed scala.meta trees, we handle semantic requests dynamically. For every request to a scala.meta tree, we use its position to identify a corresponding attributed scala.reflect tree and then use that tree's attributes to complete the request. After several hacking sessions in December and January, I'm now convinced that this approach is solid enough to become the basis of the semantic API.

After some consideration, we have also decided to cut the scope of the initial vision of the semantic API as designed ca. 2013. Concretely, in the first iteration of the API (scala.meta.semantic.v1), we're going to temporarily introduce the notion of symbols, and we're going to only support fully attributed and fully unattributed ASTs. We are still planning to eventually implement the initial vision (scala.meta.semantic.v2, see #623 for more information), but we'd like to ship something partially useful as soon as possible.

(April 2017). After more deliberation, we have further cut the scope of the semantic API. For the time being, we're going to give up on supporting unattributed trees and on the unification of symbols and trees. Therefore, scala.meta.semantic.v2 bites the dust and scala.meta.semantic.v1 becomes scala.meta.semantic.

Implementation plan:

• Support for semantic databases (#605)
• Minimal scala.meta.semantic.v1:
  • Online mirror with support for Ref.symbol (#606)
  • Rudimentary functionality to work with symbols (#609)
  • Ref.symbol for unattributed trees (#608)
  • Offline mirror with support for Ref.symbol (#607)
  • Term.tpe (#611)
  • Functionality to work with types (#612)
• Full-blown scala.meta.semantic.v1:
  • Term.desugaring and other issues labelled with Semantic (list)

[smarter]

For every request to a scala.meta tree, we use its position to identify a corresponding attributed scala.reflect tree and then use that tree's attributes to complete the request.

Interesting! This seems similar to the kind of things IDEs do to handle features like type-at-point/symbol-at-point (which incidentally should soon be very easy to do in dotty :))

[xeno-by]

@smarter Sounds very nice! Can you provide more details?

Upd. Here are some links from our hangout call with @smarter:

• Semester project report: http://guillaume.martres.me/project1_report.pdf
• LSP support in Dotty: https://github.com/dotty-staging/dotty/commits/add/ide-support
• VS Code integration: https://github.com/smarter/vscode-dotty

[xeno-by]

Recently, we decided to limit the surface of our work to actionable items only - either bugs in existing code or feature requests that are warranted by the roadmap.

While semantic API is definitely on our roadmap, we're going to figure out desired APIs from use cases, not from reflecting on scala.reflect and pondering what functionality from there we should add to Scalameta. That's why I'll be closing this issue and letting feature requests be submitted naturally by our users.