Create a Syntax highlighter [VSCode]

[georgelemon]

https://code.visualstudio.com/api/language-extensions/syntax-highlight-guide

[Uzo2005]

hi, I started work on this here. Basic syntax highlighting works, but I still have to make sure I dont have any regex blindspots. Am planning to add full language support that will enable vscode users to see the output of their templates on hover(just like tailwindcss is doing). Let me know if you need me to add or remove other things.

To use the extension just clone the repo, and press F5 when you open the cloned folder on vscode.

[Uzo2005]

Hi, the tokenization method (textmate grammar) which vscode uses makes it very hard to match contents that span multiple lines. The discussion of this problem can be found here microsoft/vscode-textmate#41.
A very good example is this block of valid tim code:

div.row > div.col-12.text-center
  div.my-3#clickable
    a.btn.btn-primary.btn-lg.rounded-pill.px-4.py-2 href="https://github.com/openpeeps/tim" target="_blank":
      svg viewBox="0 0 24 24" width="24" height="24" stroke="currentColor" stroke-width="2" fill="none" stroke-linecap="round" stroke-linejoin="round" class="css-i6dzq1"
        path d="M9 19c-5 1.5-5-2.5-7-3m14 6v-3.87a3.37 3.37 0 0 0-.94-2.61c3.14-.35
                6.44-1.54 6.44-7A5.44 5.44 0 0 0 20 4.77 5.07 5.07 0 0 0
                19.91 1S18.73.65 16 2.48a13.38 13.38 0 0 0-7 0C6.27.65
                5.09 1 5.09 1A5.07 5.07 0 0 0 5 4.77a5.44 5.44 0 0 0-1.5
                3.78c0 5.42 3.3 6.61 6.44 7A3.37 3.37 0 0 0 9 18.13V22"
      span.fw-bold.ms-2: "Check it on GitHub"
    if $this.path == "/about":
      a href="/" .btn.btn-link.text-light.btn-lg.rounded-pill.px-4.py-2:
        "Go back to Homepage"
div.text-center
  p.mb-0: "© " & $app.year & " — Made by Humans from OpenPeeps"
  p: "Open Source | LGPL-3.0 license"

@client target="#clickable"
  // transpile tim code to javascript for client-side rendering
  div.mt-3 > a.text-secondary.text-decoration-none href="https://hetzner.cloud/?ref=Hm0mYGM9NxZ4"
    small
      span: "👉 Create a VPS using our link and 👇 "
      br
      span: "Get €20 in cloud credits from Hetzner"
@end

using the current state of Uzo2005/tim-vscode-extension@622fe94 to highlight this shows that syntax highlight breaks for the value of the d atttribute in the path element. I understand that this problem arises because textmate works one line at a time and forgets the context on the next line. So I am going to use vscode's sematic highlight features to do the highlighting (https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide)

[georgelemon]

Hey, thanks for your effort! There is an unfinished .sublime-syntax available that you can use for inspiration.

There will be 3 types of strings, single, double and triple quoted. Also, backticks literals will have a similar functionality with what we have in JavaScript.

Single-quoted strings are one-liners

var x = 'It\'s crazy!'
var y = 'This "weird thing" is really nice, right!'

Double-quoted strings

Allow multi-line strings but require double quote be escaped using a backslash

var myobj = {
  title: "Town \"Crier\""
  description: "With booming voices and ringing bells,
    they delivered news and announcements in the days
    before mass media"
}

Triple-quoted string literals

Similar with Nim's triple quoted strings, may span multiple lines that don't need escaping. This may be helpful for injecting quick JS code and other dirty string-based data.

head
  title: "Awesome!"

body
  script: """
function say(x) {
  return x + "  hello, hello!"
}
console.log(say("Hello,"))
  """

  h1: "Tim is Awesome!"

Template literals

For this we can take a look at JavaScript.tmLanguage

var y = 123
var x = `backticks as template literals ${y}`
var z = `${y + 1} likes`

A summary of Tim keywords

Keywords

if, elif else, for, in, case, of, while, var, const, and, or, echo, return, void

Also, there will be a special dot expression .type (or typeof, not sure). that will return the type of the identifier.

if $myVar.type == string:
  echo "for sure it's a string"

Other chars and aliases

& (string concat)
&& (alias of and)
|| (alias of or)

^ circumflex accent char to handle index ranges

var x = [1, 2, 3, 4, 5]
echo $x[1..^1]

? conditional ternary operator and | for handling the else body

var x = true
div ($x == false ? disabled="disabled" | contenteditable="true"): "Some content"

Comparison operators

<, <=, >, >=, ==, !=

Math Operators

+, -, *, /

Data types

string, int, float, bool, array, object

Special @ keywords

@client, @js, @yaml, @css, @end, @view, @include, @import, @placeholder

Storage identifiers

$this and $app are special storage identifiers

Embed Code Snippets

Should use the built-in highlight extensions.

@css
.btn {
  color: blue;
}
@end

[georgelemon]

? conditional ternary operator and | for handling the else body

better || to determine the else body

var x = 123
echo ($x == 2 ? "that's two" || "yes, yes, yes!")

[Uzo2005]

Thanks a lot for clarifying all the possible tokens, it makes everything clearer now. I will take hints from the sublime syntax extension and improve the vscode extension.
I have a question about the last comment: if || is used as ternary else, doesnt it clash with the token for logical or(||), or is this overloading by intentional design? Would it not be more intuitive to copy javascript's syntax for ternary expressions:

var x = 123
echo ($x == 2 ? "that's two" : "yes, yes, yes!")

which also allows chaining of else-ifs:

var x = 123
echo ($x == 2 ? "that's two" : $x == 3 ? "yes, yes, its three!" : "definitely something else")

which means : will now be used to determine conditionals

[georgelemon]

I agree. let's use : token to determine else branch

[Uzo2005]

Hi, I am done with everything involving syntax highlighting and so am ready to publish version 0.0.1 of the extension. You can preview the extension by:

1. cloning this repo
2. copying the cloned folder into your <user home>/.vscode/extensions folder and restarting Vscode.

I have some questions on how to go about the publication:

• Is the extension going to be published under the openpeeps organisation?
• Is the main branch for development still going to be https://github.com/Uzo2005/tim-vscode-extension or will we create a new repo for the extension under openpeeps?

Thanks

PS: sorry for taking so long, we had lots of tests to write this week

[georgelemon]

Nice! Thank you, Cletus! You can use your repo to publish the extension. If you want, use this banner in README.md and give a link to openpeeps/tim repo. Also, you can add some tags to your repo

Thanks for using Tim and helping with this <3 Hopefully soon I will finish the docs and release the first version.

Cheers!

[Uzo2005]

thanks, I really believe that Tim will solve all the pain points I experience with html templating languages, so I really want to see it go mainstream.

I just published the extension here