[Go] syntax rework #1662

mitranim · 2018-07-14T15:04:47Z

This PR is a complete rework of the Go syntax.

Improvements:

Contextual types. Types are scoped where types are expected. There's no whitelist of "known" types.
Better understands the language and its syntax. Better at handling whitespace and comments inside complex forms. Better at handling other nuances. Unlikely to be confused by unusual code style.
Vars and consts in the root scope are added to the symbol index and searchable by ⌘R, ⌘⇪R, and goto_definition.
Supports parenthesized type (...) groups, not just const (...) and var (...).
The first argument to new and make is scoped as a type.
Methods declared inside an interface are added to the symbol index, reflecting the fact that they actually exist as functions.
Supports embedded structs.
Supports inherited interfaces.
Supports imaginary number literals.
Variable declarations are scoped differently from other variable occurrences.
iota is scoped only in constant initialization expressions.
Symbols in ⌘R are annotated with their origin: var, const, type or func.
Auto-pairing of backticks.
Much more comprehensive tests.
All keywords and predeclared identifiers are added to auto-completions by default.
Better snippets that don't mess with auto-completion.
Slightly better indentation rules that more closely match gofmt.

Current shortcomings (?):

Fewer meta scopes: needs feedback.
No support for block labels: needs feedback.

While this syntax is ready-to-use and fairly complete, I expect the current maintainers, particularly @wbond who's responsible for the current version of the Go syntax, to have many qualms and request many changes. Arguably, this version has some "regressions", particularly where meta scopes are concerned. This needs feedback and discussion, as I don't fully understand what role many of those meta scopes were intended to serve.

While the PR is pending, the syntax is available standalone: https://github.com/Mitranim/sublime-gox

Screenshot before-after:

Screenshot of symbol index before-after:

(Many other differences didn't fit into the screenshots. This uses my Cloud color scheme, available at https://github.com/Mitranim/sublime-themes.)

* Contextual types. Types are scoped where types are expected. There's no whitelist of "known" types. * Better understands the language and its syntax. Better at handling whitespace and comments inside complex forms. Better at handling other nuances. Unlikely to be confused by unusual code style. * Vars and consts in the root scope are added to the symbol index and searchable by ⌘R, ⌘⇪R, and `goto_definition`. * Supports parenthesized `type (...)` groups, not just `const (...)` and `var (...)`. * The first argument to `new` and `make` is scoped as a type. * Methods declared inside an `interface` are added to the symbol index, reflecting the fact that they actually exist as functions. * Supports embedded structs. * Supports inherited interfaces. * Supports imaginary number literals. * Variable declarations are scoped differently from other variable occurrences. * `iota` is scoped only in constant initialization expressions. * Symbols in ⌘R are annotated with their origin: `var`, `const`, `type` or `func`. * Auto-pairing of backticks. * Much more comprehensive tests. * All keywords and predeclared identifiers are added to auto-completions by default. * Better snippets that don't mess with auto-completion. * Slightly better indentation rules that more closely match `gofmt`. Current shortcomings (?): * Fewer meta scopes: needs feedback. * No support for block labels: needs feedback.

FichteFoll

Did a small review of everything that's not the actual syntax definition or test file(s).

FichteFoll · 2018-07-15T14:58:35Z

Go/Default.sublime-keymap

+  },
+  { "keys": ["backspace"], "command": "run_macro_file", "args": {"file": "res://Packages/Default/Delete Left Right.sublime-macro"}, "context":
+    [
+      { "key": "setting.auto_match_enabled", "operator": "equal", "operand": true },


Doesn't have a selector context specifier.

Thanks, added the missing selector.

FichteFoll · 2018-07-15T14:58:53Z

Go/Default.sublime-keymap

+
+  { "keys": ["enter"], "command": "run_macro_file", "args": {"file": "res://Packages/Default/Add Line in Braces.sublime-macro"}, "context":
+    [
+      { "key": "setting.auto_indent", "operator": "equal", "operand": true },


Also doesn't have a selector context specifier.

Thanks, added the missing selector.

FichteFoll · 2018-07-15T14:59:48Z

Go/Go.sublime-completions

+
+  "completions": [
+    // https://golang.org/ref/spec#Keywords
+    "break",


I suggest adding a description for these completions, i.e. keyword. See http://docs.sublimetext.info/en/latest/reference/completions.html.

Hmm the docs don't mention any descriptions.

Apparently, some completions in the standard packages use tab-separated triggers: word\tdescription. Tried this; it looks cool in the completion list, but suppresses the normal buffer completions, which is exactly what you don't want, especially in a standard package. If we could add descriptions without messing things up, I'd love to.

Played with generating sublime-snippet files for these completions. Snippets have descriptions and don't suppress normal buffer completions. They also don't appear when you just hit ^Space or equivalent to bring up the completion list. The downside is that they clog up the Command Palette.

Really the problem is that even one sublime-completion with a tab-separated description completely suppresses buffer completion. It's ridiculous and wrong.

Meh, sounds like sublimehq/sublime_text#1061 is striking again. I thought tab triggers wouldn't affect it.

Or maybe it is actually a different issue, but that would be surprising.

FichteFoll · 2018-07-15T15:00:49Z

Go/Go.sublime-settings

@@ -0,0 +1,11 @@
+{
+  // Prefer tabs because gofmt enforces tabs.
+  "translate_tabs_to_spaces": false,


This is certainly debatable, but I don't have insight on whether people write Go without gofmt.

FichteFoll · 2018-07-15T15:01:40Z

Go/Go.sublime-settings

+  // Prefer tabs because gofmt enforces tabs.
+  "translate_tabs_to_spaces": false,
+  // Trigger autocompletion on dot access.
+  "auto_complete_triggers": [


Many packages use dots for attribute access, however none of them defines auto_complete_triggers. I'm not sure whether they should or should not, but this is something we should discuss, imo.

This certainly could be removed. I'll run without this setting for a few days to get a feel.

Removed the trigger setting for now.

FichteFoll · 2018-07-15T15:04:58Z

Go/Symbols/GoConstSymbolIndex.tmPreferences

+<dict>
+  <key>scope</key>
+  <string>
+    variable.other.constant.declaration.go - meta.block.go


entity.name.constant.go would be more appropriate, assuming the syntax definition is updated as well.

Agreed. The problem is that applying different const/var scopes depending on whether we're inside a block seems to require lots of duplication in the syntax; not sure it's worth it.

FichteFoll · 2018-07-15T15:07:05Z

Go/Symbols/GoConstSymbols.tmPreferences

@@ -0,0 +1,16 @@
+<?xml version="1.0" encoding="UTF-8"?>


You can define both, the indexed and non-indexed symbols, in the same file, assuming their scope selectors are equal. This applies to the other files as well.

These are defined in separate files to keep symbolTransformation from applying to the indexed symbols and messing up "goto_definition". I'd love to be able to merge the files, but it doesn't seem possible while keeping the current rule logic.

Can you confirm that symbolTransformation does indeed affect the symbol that is used for the index if showInIndexedSymbolList is set? Previous experience doesn't indicate so (and neither to the docs derived from that).

Even with two different files, since the scope selectors are the same it would have the exact same effect as all the meta settings will be set on any token that is matched.

Interesting! I double checked, and indeed, it looks like symbolTransformation doesn't affect indexing. Must have mixed it up with something else when I was writing this.

Merged some of the symbol files. Thanks!

* consistent with other syntaxes for C-style languages

FichteFoll · 2018-07-26T19:11:12Z

Something I did for my YAML rework was to have a sample file containing lots of examples with bad highlighting in the original and then providing a Screenshot of that file being rendered with and without the pr to give a better before-after comparison (#90).

mitranim · 2018-07-28T08:03:41Z

@FichteFoll Nice idea. Added comparison screenshots.

* support negative exponents * support fractional notation in complex numbers * punctuation scopes for dot and exponent * storage scopes for octal and hex prefixes

cskujbus · 2018-10-25T08:12:03Z

Hi,
I'd like to ask why type, struct, func, chan etc. don't have keyword scope? They have storage.type which means they are styled differently than keywords.

mitranim · 2018-10-25T08:16:12Z

@cskujbus An excellent point. I followed the established convention of scoping type keywords the same as type names, but frankly, it makes no sense whatsoever. Perhaps we should break with the convention, and ask the rest of the syntax developers to follow suit.

@wbond @keith-hall

FichteFoll · 2018-10-25T10:16:57Z

#582

mitranim · 2018-10-25T10:31:52Z

As an experiment, I've scoped these as keywords, and the keyword coloring started "popping out" in the middle of type signatures. It feels weird when you're used to the "old" way. I've come to associate keyword coloring with control flow.

Thoughts?

wbond · 2018-10-25T13:45:59Z

I’m not inclined to revamp the scope names for function and class keywords. Most languages that use those have the ability to treat such as first class citizens, so I think it makes sense the keywords are scoped as storage.type. The keyword is describing the type of the identifier, in this case a function, class, etc.

FichteFoll · 2018-10-25T17:18:33Z

For now, I suppose the best action forward is to scope them as storage.type.class and storage.type.function (and not func!) and storage.type.struct, so that people may colorize them differently if they wish.

mitranim · 2018-10-26T09:12:40Z

My inner language-author/syntax-author/Lisper feels that all language keywords should have a unified scope, so that a simplified color scheme can consistently mark them as keywords, without special-casing the various storage. variants and without affecting the actual types names. However, the screenshots above convince me that for type keywords that may occur inside type signatures, giving them a coloring similar or identical to type names is a nice default. This is particularly useful for languages with anonymous inline types, such as Go, where a type signature may mix type names with keywords.

Satisfying both use cases requires the scope on keywords like struct or interface to be distinct from storage.type so we can distinguish type keywords from type names. The current convention is to scope them identically, which is really dubious.

I would propose keyword.storage if we had the luxury of wide breaking changes. Otherwise, storage.keyword appears to be the most sensible option. Probably not storage.modifier since it more commonly refers to mode of mutation rather than the data layout itself.

Long story short: what if we scope all type keywords, such as func, struct, interface, as storage.keyword?

Addendum for consistency with the argument above. I don't see a reason why type, var, const and the like shouldn't also be scoped as storage.keyword.X. The same logic applies: they're still keywords; a simpler color scheme may want to mark them as such; a plugin may want to use this scope to find all keywords, regardless of their fiddly little differences.

cskujbus · 2018-10-26T10:25:37Z

@mitranim thank you for this detailed explanation, I just wanted to know the reasoning behind classifying some keywords as storage types (VS Code for example seems classifying all Go keywords in the same scope).
I tested also my color scheme (with keywords in bold) with a very simple syntax with all Go keywords in the same scope and, I have to admit, the text looks a bit heavier than before but also keywords are more distinguishable from simple types like int, string etc.

Go/Go.sublime-syntax

Co-Authored-By: Mitranim <me@mitranim.com>

Type keywords now use "storage.keyword" rather than "storage.type". This distinguishes them from type names while keeping the "storage" prefix. This is consistent with the default coloring for most other languages, but a color scheme can also choose to treat them like regular keywords. This commit also changes "var" and "const" from "storage.modifier" to "storage.keyword", for logical consistency with the change above.

@deathaxe

Added punctuation scopes for comment start and comment end. Added more detailed annotation meta scopes. Also by @deathaxe, see mitranim/sublime-gox#1

wbond · 2018-11-27T23:17:14Z

I really wanted to merge this for the next dev build, but it is extensively using storage.keyword, which is not part of the guidelines, and has chosen to change them from scopes that are currently the standard.

I don't want to break a bunch of syntax coloring for users, and thus am not inclined to merge this since it takes scope names in a new direction that is different (in a major way) from all of the syntaxes that have been working on over the past couple of years. For consistency sake, I want new syntax work to be consistent with other actively maintained syntaxes. So to merge this, we would have to revamp scope names in a bunch of syntaxes, change the standard and revamp color schemes.

mitranim · 2018-11-28T06:44:58Z

Glad to see you back in action, @wbond!

I've updated the storage.keyword.X scopes to storage.type.keyword.X. Also pushed a few other minor changes. Let me know if there are any other issues.

FichteFoll · 2018-11-28T13:52:04Z

I suppose replacing storage.type.keyword.function with storage.type.function would be appropriate, which is used across multiple syntaxes iirc. And similar occurances.

Not sure about interface and const, however.

wbond · 2018-11-28T19:09:47Z

I am more inclined for that format @FichteFoll. The impression I get is that @mitranim is trying to make it possible to color storage.type.keyword the same as keyword, and so having to enumerate all of the different possible storage.type. specializations would not be viable.

wbond · 2018-11-29T04:38:30Z

I'd rather get this merged and we can discuss subtle changes to scope later.

ccampbell · 2018-11-30T23:29:10Z

Just updated my Sublime build. One thing I have noticed is the new highlighting used for types within a package.

For example:

From what I can tell, both the native type string and the Time from time.Time are using storage.type.go as the scope. Would it be possible to use something like storage.type.go.package or storage.type.go.imported to be able to differentiate the types for theming? It is also misleading to call it storage.type.go since it is not a native go type.

It may just be that I am not used to the change, but I think I would prefer to be able to highlight them differently. Previously time.Time was not highlighted at all (IIRC).

deathaxe · 2018-11-30T23:34:56Z

Time would need to be scoped using support.type.go if it is part of a "basic" framework library. storage.type.go is meant for language types only.

(This is what I know from other syntaxes and is how I understand the scope names, which may be wrong)

mitranim · 2018-12-03T08:09:32Z

Thanks for asking. This is an important difference between my syntaxes and most other ST built-ins. I've been meaning to raise this topic for a while now.

For the language user, there's no functional difference between predefined and user-defined types. Standard library types like time.Time are user-defined as far as the language is concerned. A type is just a type.

The idea of treating types differently is so weird, I can't even comprehend where it's coming from. Maybe it's inertia from syntaxes that wanted but couldn't scope types properly?

Modern general-purpose languages are explicitly designed to be extensible. Complicated programs tend to develop their own DSLs, speaking mostly in their own "language", often supplanting built-in types and idioms. Treating all types equally, exactly the way the language does, supports this notion.

A syntax's job is to reflect the language. When the language treats all types equally, so does the syntax.

Yes, many built-in ST syntaxes have these weird ideas about "built-in" and "support" types. This doesn't make them any less wrong. I plan to convince other syntax developers to revise this notion across all standard packages.

deathaxe · 2018-12-03T17:48:09Z

Not sure about the origin, but this is just the existing way to achieve @ccampbell 's request.

I honestly find it helpful to see language defined functions (support.function) in another color than user defined (variable.function) ones. Same might be helpful with basic/builtin/atomic language types like ints/floats/... and complex types like classes/structs/... which are a collection of the atomic types extended by functions/methods/... .

You are right with support. describing basic framework functions/types being less suitable for use with user defined types. This is indeed weird. It's even not possible to distinguish them in most languages.

But the possibility to distinguish between atomic and complex types should be applied anyway. Maybe something like storage.type.go for simple types and storage.class.go for classes (instead of support.class.go) or all in the 3rd level (storage.type.simple.go vs. storage.type.class.go)?

Thom1729 · 2018-12-05T15:00:25Z

I'm in favor of scoping builtin types and functions.

It must be admitted that scoping builtins is not a truly syntactic concern. In most languages, a built-in function or type is parsed exactly the same as any other type. You can have a completely accurate syntax definition that doesn't scope any builtins at all.

On the other hand, to the extent that builtins are defined by the language itself, I think that it's useful to use scopes to distinguish them. One reason is that builtins generally don't have lexical declarations, so when I see something highlighted as a builtin, I know not to look for one. In the absence of full language-specific intellisense, builtin highlighting also serves as a first line of defense against typos.

I think that support scopes should be purely additive. That is, a builtin function should be scoped just like any other function, plus whatever extra scope builtins get. The core syntaxes don't all do this, and I think they should. For one thing, this makes it easier for users who don't want builtin highlighting to modify their color scheme to disable it.

What builtins deserve highlighting? The answer surely depends on the language. In JavaScript, there is a fixed set of global types and functions that do not need to be included and are available in any environment. These, I think, should be scoped. There are also many other global builtins available in other environments, such as web browsers. I'm not convinced that these should be scoped in the core language. For JavaScript, I'm planning to write an extension for JS Custom to allow users to choose between sets of builtins (like browser/node) and to add their own.

FichteFoll · 2018-12-06T12:39:44Z

This is becoming somewhat off-topic and approaching RFC levels. I suggest to summarize a few options and formulate those into a proposal in a several issue. I also have a thing or two to say about this, later.

After all, this doesn't just affect Go.

mitranim · 2018-12-07T08:29:20Z

@Thom1729 thanks for the explanation. These points make a lot of sense.

I like the idea of built-ins scoping being purely additive. Reserving it for a small, rarely-expanded set of language built-ins, rather than any "library" APIs like the DOM, also seems like a no-brainer.

Getting back to Go. Unlike most languages, it lets you tell apart, on a glance, local/external and private/public definitions:

builtin
privateLocal
PublicLocal
external.Public

Syntactically, built-ins overlap only with private local definitions. In addition, the number of built-in types and functions is fairly small and easy to remember. Since I still don't understand the benefit of additional "builtin" or "support" scoping, my default preference would be to stick with storage.type for simplicity and consistency. Obviously, this is up to consensus.

mitranim force-pushed the go branch from f543171 to 984f814 Compare July 14, 2018 15:16

FichteFoll reviewed Jul 15, 2018

View reviewed changes

mitranim added 7 commits July 16, 2018 11:27

[Go] consistent tmPreferences indentation

31abd0d

[Go] added missing scope selectors to keymap

6e733ba

[Go] minor cosmetics

adffe48

[Go] removed dot autocompletion trigger

2938d85

[Go] merged symbol settings files where possible

8d63f72

[Go] cosmetics in symbol preferences

e1007b3

[Go] removed spaces from block comments

caa2fd4

* consistent with other syntaxes for C-style languages

mitranim added 10 commits August 24, 2018 17:20

[Go] meta scope for //go: annotations

ed3c3bd

[Go] number corrections

50997bd

* support negative exponents * support fractional notation in complex numbers * punctuation scopes for dot and exponent * storage scopes for octal and hex prefixes

[Go] detect embedded struct fields when tagged

980645e

[Go] corrected comma-grouped var and const declarations

9fa51d2

[Go] removed stale symbol file

982dcbc

[Go] consistent escape sequences in char and string literals

0392749

[Go] removed unnecessary (?) syntax name

eb519b4

[Go] detect embedded type followed by line comment

06e203c

[Go] cosmetic tests and comments

23993dc

[Go] don't unindent block comment terminator

2825db4

deathaxe reviewed Oct 28, 2018

View reviewed changes

Go/Go.sublime-syntax Outdated Show resolved Hide resolved

deathaxe and others added 6 commits October 29, 2018 10:51

[Go] cosmetic cleanup in noise regexp

a06dd39

Co-Authored-By: Mitranim <me@mitranim.com>

[Go] comment and whitespace cosmetics

a182359

[Go] better comment scoping

7f44c59

Added punctuation scopes for comment start and comment end. Added more detailed annotation meta scopes. Also by @deathaxe, see mitranim/sublime-gox#1

[Go] cosmetic renaming for consistency

8614e81

[Go] corrected comment EOL rule

7bb6f90

mitranim added 6 commits November 28, 2018 09:40

[Go] whitespace cosmetics in tests

8c7bf1b

[Go] special scope for blank variable

5dbd19d

[Go] "fori" snippet produces valid syntax

c40f286

[Go] slightly improved indent rules

c19aaec

[Go] misc changes in comments

02634d3

[Go] storage.keyword -> storage.type.keyword

a7f273d

wbond merged commit 14a35f6 into sublimehq:master Nov 29, 2018

ccampbell mentioned this pull request Dec 6, 2018

[Go] Built in functions no longer highlighted #1795

Closed

kortschak mentioned this pull request Dec 8, 2018

[Go] Add back builtin type matching #1803

Open

mitranim mentioned this pull request Jan 23, 2019

Sublime Syntax compatibility tree-sitter/tree-sitter#139

Closed

deathaxe pushed a commit to deathaxe/sublime-packages that referenced this pull request Jun 9, 2019

[Go] syntax rework (sublimehq#1662)

adeb37f

[Go] syntax rework #1662

[Go] syntax rework #1662

Conversation

mitranim commented Jul 14, 2018 • edited Loading

FichteFoll left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FichteFoll commented Jul 26, 2018 • edited Loading

mitranim commented Jul 28, 2018

cskujbus commented Oct 25, 2018

mitranim commented Oct 25, 2018

FichteFoll commented Oct 25, 2018

mitranim commented Oct 25, 2018 • edited Loading

wbond commented Oct 25, 2018

FichteFoll commented Oct 25, 2018

mitranim commented Oct 26, 2018 • edited Loading

cskujbus commented Oct 26, 2018

wbond commented Nov 27, 2018

mitranim commented Nov 28, 2018 • edited Loading

FichteFoll commented Nov 28, 2018 • edited Loading

wbond commented Nov 28, 2018

wbond commented Nov 29, 2018

ccampbell commented Nov 30, 2018 • edited Loading

deathaxe commented Nov 30, 2018

mitranim commented Dec 3, 2018

deathaxe commented Dec 3, 2018

Thom1729 commented Dec 5, 2018

FichteFoll commented Dec 6, 2018

mitranim commented Dec 7, 2018

mitranim commented Jul 14, 2018 •

edited

Loading

FichteFoll commented Jul 26, 2018 •

edited

Loading

mitranim commented Oct 25, 2018 •

edited

Loading

mitranim commented Oct 26, 2018 •

edited

Loading

mitranim commented Nov 28, 2018 •

edited

Loading

FichteFoll commented Nov 28, 2018 •

edited

Loading

ccampbell commented Nov 30, 2018 •

edited

Loading