Add Scala Syntax Highlight

[BalmungSan]

Probably don't add too much value (therefore not so priority). But is somewhat sad not having it.

By looking at this. It seems that is not so difficult to add it, and probably scalariform already does all the hard work for us.

[mwz]

Yes, that's a good idea! 👍

[BalmungSan]

Hi @mwz, I was giving a look to this hoping it was trivial to implement. But, sadly, I encountered a big problem.
As you can see here, in order to register a new highlight in a file, you need the start and end position of the token (in terms of lines and columns).

However, the scalariform tokens don't provide that information. Instead, they use an offset range.
(Note that there is a highlight method that can work with that, but it is deprecated).

Do you have any ideas?

[mwz]

I think that the offsets together with newline tokens should allow you to determine the position/range of the highlights. You might have to go through the list of tokens one by one and potentially keep track of some state like e.g. line number and column as you traverse the tokens.

[BalmungSan]

Hi, I though that too, the problem is that multi-line comments absorb all new-line tokens, thus we will end with the wrong linenum.

Example

val code = """/*
* Sonar Scala Plugin
* Copyright (C) 2011-2016 SonarSource SA   
*/"""

val tokens =  Scala.tokenize(code, ScalaVersions.Scala_2_11)

// tokes List[scalariform.lexer.Token] =
// List(Token(MULTILINE_COMMENT,/*
// * Sonar Scala Plugin
// *  Copyright (C) 2011-2016 SonarSource SA
// */,0,/*
// * Sonar Scala Plugin
// * Copyright (C) 2011-2016 SonarSource SA
// */), Token(EOF,,70,))
// */

tokens.length
// res0: Int = 2

[mwz]

I think it is still possible to do this, you just need to handle the EOL characters yourself, which makes it a little bit more work...

I've come up with this naive implementation - it probably doesn't cover some edge cases and XML tokens, but it looks like it works for this simple example and should give you a rough idea about what I meant.

import scalariform.lexer.{ScalaLexer, Tokens}

import scala.util.matching.Regex

val code ="""/*
* Sonar Scala Plugin
* Copyright (C) 2011-2016 SonarSource SA
*/

// testing
case class Hello(world: String) {
  def ok(): Unit = ()
}
"""

case class Range(
  startLine: Long,
  startLineOffset: Long,
  endLine: Long,
  endLineOffset: Long,
  length: Long
)

val newLineRegex: Regex = "(\r\n)|\r|\n".r
val eolRegex: Regex = "(\r\n)|\r|\n$".r

def countEols(s: String): Long =
  newLineRegex.findAllIn(s).size

def endsWithEol(s: String): Boolean =
  eolRegex.findFirstIn(s).nonEmpty

def lastLine(s: String): String =
  s.linesWithSeparators.toSeq.lastOption.getOrElse("")

val initialRange = Range(1, 0, 1, 0, 0)

val tokens = ScalaLexer.rawTokenise(
  code,
  forgiveErrors = false,
  "2.12"
)

// tokens: List[scalariform.lexer.Token] = List(Token(MULTILINE_COMMENT,/*
// * Sonar Scala Plugin
// * Copyright (C) 2011-2016 SonarSource SA
// */,0,/*
// * Sonar Scala Plugin
// * Copyright (C) 2011-2016 SonarSource SA
// */), Token(WS,
//
//   ,67,
//
// ), Token(LINE_COMMENT,// testing
//   ,69,// testing
// ), Token(CASE,case,80,case), Token(WS, ,84, ), Token(CLASS,class,85,class),
// Token(WS, ,90, ), Token(VARID,Hello,91,Hello), Token(LPAREN,(,96,(),
// Token(VARID,world,97,world), Token(COLON,:,102,:), Token(WS, ,103, ), 
// Token(VARID,String,104,String), Token(RPAREN,),110,)), Token(WS, ,111, ),
// Token(LBRACE,{,112,{), Token(WS,
//  ,113,
// ), Token(DEF,def,116,def), Token(WS, ,119, ), Token(VARID,ok,120,ok), 
// Token(LPAREN,(,122,(), Token(RPAREN,),123,)), Token(COLON,:,124,:),
// Token(WS, ,125, ), Token(VARID,Unit,126,Unit), Token(WS, ,130, ), 
// Token(EQUALS,=,131,=), Token(WS, ,132, ), Token(LPAREN,(,133,(), 
// Token(RPAREN,),134,)), Token(WS,
//  ,135,
// ), Token(RBRACE,},136,}), Token(WS,
//  ,137,
// ), Token(EOF,,138,))

tokens.scanLeft(None: Option[Range]) {
  case (acc, token) =>
    val a = acc.getOrElse(initialRange)
    token.tokenType match {
      case Tokens.LINE_COMMENT | Tokens.WS =>
        val eols = countEols(token.rawText)
        val ll = lastLine(token.rawText)
        val eol = endsWithEol(token.rawText)
        Some(Range(
          a.endLine,
          a.endLineOffset,
          a.endLine + eols,
          if (eol) 0 else a.endLineOffset + ll.length,
          token.length
        ))
      case Tokens.MULTILINE_COMMENT =>
        val eols = countEols(token.rawText)
        val ll = lastLine(token.rawText)
        Some(Range(a.endLine,
          a.endLineOffset,
          a.endLine + eols,
          ll.length,
          token.length
        ))
      case _ =>
        Some(Range(
          a.endLine,
          a.endLineOffset,
          a.endLine,
          a.endLineOffset + token.length,
          token.length
        ))
    }
}

// res0: List[Option[Range]] = List(
// None, Some(Range(1,0,4,2,67)), Some(Range(4,2,6,0,2)),
// Some(Range(6,0,7,0,11)), Some(Range(7,0,7,4,4)), Some(Range(7,4,7,5,1)),
// Some(Range(7,5,7,10,5)), Some(Range(7,10,7,11,1)), Some(Range(7,11,7,16,5)),
// Some(Range(7,16,7,17,1)), Some(Range(7,17,7,22,5)), Some(Range(7,22,7,23,1)),
// Some(Range(7,23,7,24,1)), Some(Range(7,24,7,30,6)), Some(Range(7,30,7,31,1)),
// Some(Range(7,31,7,32,1)), Some(Range(7,32,7,33,1)), Some(Range(7,33,8,35,3)),
// Some(Range(8,35,8,38,3)), Some(Range(8,38,8,39,1)), Some(Range(8,39,8,41,2)),
// Some(Range(8,41,8,42,1)), Some(Range(8,42,8,43,1)), Some(Range(8,43,8,44,1)),
// Some(Range(8,44,8,45,1)), Some(Range(8,45,8,49,4)), Some(Range(8,49,8,50,1)),
// Some(Range(8,50,8,51,1)), Some(Range(8,51,8,52,1)), Some(Range(8,52,8,53,1)),
// Some(Range(8,53,8,54,1)), Some(Range(8,54,9,0,1)), Some(Range(9,0,9,1,1)),
// Some(Range(9,1,10,0,1)), Some(Range(10,0,10,0,0))
// )