Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove Incompatible Regex Patterns #481

Open
14 of 17 tasks
wbond opened this issue Jun 17, 2016 · 4 comments
Open
14 of 17 tasks

Remove Incompatible Regex Patterns #481

wbond opened this issue Jun 17, 2016 · 4 comments
Labels
C: Syntax T: enhancement Improvement of existing language features

Comments

@wbond
Copy link
Member

wbond commented Jun 17, 2016

The following languages need to be improved to remove regex patterns that cause us to hit the Oniguruma library:

As noted above, some are minor tweaks, while others probably need a major overhaul.

@wbond wbond added the T: enhancement Improvement of existing language features label Jun 17, 2016
deathaxe added a commit to deathaxe/sublime-packages that referenced this issue May 11, 2018
Fixes sublimehq#541, sublimehq#1029, sublimehq#1325, sublimehq#1326, sublimehq#1495

Supersedes sublimehq#1556

According to sublimehq#481 this commit provides an Oniguruma free Perl syntax
definition written from scratch.

OUTLINE

  1. fixes all currently open issues
  2. much more feature complete
  3. completely compatible with new regex engine
  4. uses embed to include custom syntaxes within
     POD and HEREDOCS (CSS,JS,JSON,HTML,SQL,XML)
  5. provides a syntax test file
  6. uses as much atomic rules from Perl.sublime-syntax and
     ModernPerl package as possible (with some fixes/modifications)
  7. follows the doc http://perldoc.perl.org as close as possible
  8. add some snippets with some files renamed to match naming style

REMARKS

1. The benchmark tests on a 100k file with all kinds of statements
   output about 360ms for evaluation on a Windows box.

2. The main focus is to create a robust base for syntax highlighting
   rather than scoping each language entity/construct in detail.

   a) Not all meta scoping RFCs are implemented in detail due to
      some limitations caused by HEREDOC. Especially pushing in/out
      of `meta.block` or `meta.braces` breaks some HEREDOC constructs.

   b) Scope names were changed to follow the scope naming guidelines
      as close as possible and try to use as general names as possible.
      -> Maybe needs some refines?

3. At the current point the implementation is a work in progress and
   needs testing by perl power users to

   a) check for highlighting to work in all situations

   b) constants/variables/functions are scoped correctly

4. This commit adds some snippets but does not include any completions
   in favor of https://packagecontrol.io/packages/Perl%20Completions
@michaelblyons
Copy link
Collaborator

With the latest batch of merges, Perl and Clojure are done, yes?

deathaxe pushed a commit to deathaxe/sublime-packages that referenced this issue Feb 10, 2019
Fixes sublimehq#1609
Fixes sublimehq#1724

According to sublimehq#481 this PR provides an Oniguruma free Erlang syntax
definition written from scratch.

Outline
=======

The syntax definition is based on the http://erlang.org/doc and
includes all sematic features of Erlang/OTP 21.2 which are required
for highlighting.

It follows https://www.sublimetext.com/docs/3/syntax.html and includes
concepts from https://github.com/sublimehq/Packages.

Nearly every aspect and detail of the syntax definition was designed
from ground up making use of the latest features of ST's syntax engine
and with the intent to apply the most recent best practice examples
found in other recently updated syntax definitions.

Changes
=======

* Add support for Erlang ShellScript (escript)
* Add support for Erlang Type Language (`-type`, `-spec` directive)
* Add lots of detailed test cases
* Add some invalid illegal matching where appropriate
* Add builtin constants and macros highlighting
* Improve the list of builtin functions
  ([BIFs](http://erlang.org/doc/man/erlang.html#exports))
* Improve indention settings
* Improve shebang/editorconfig detection in `first_line_match`
* Improve support for _Goto Definition_ and _Goto Reference_
* Improve parsing performance by round about 50%
* Fix quoted atom highlighting (function names, module names, field names, ...)
* Fix bitstring data type highlighting
* Fix string escape highlighting
* Fix string placeholders highlighting
* Fix identifier break handling (`@` is no word-break)
* Fix lots of context pop offs
* ...

Benchmarks
==========

The whole design and development process included benchmarking of
different solutions to find the best possible performance for the
different sets of rules. The following examples provide an impression
of the performance improvements achieved with the new design.

**10k binary**

  Bin  = << << (X*2) >> || <<X>> <= << 1,2,3 >> >>.

**10k maps**

  Expr#{name=>"adam",{age,24}:=fct(),4.0=>{july,29}}

**10k records**

  #record{field1="val1", Field2=3, field3, 'Field-4'={}, _=atom}

**10k fun types**

  -type fun() :: fun(() -> int()).

**10k spec**

  -spec is_foo(Pid) -> boolean() when Pid :: pid() | if(); (Pid) -> ok.

**ErlangOTP files**

https://github.com/erlang/otp/blob/master/lib/wx/include/gl.hrl
https://github.com/erlang/otp/blob/master/lib/stdlib/test/re_testoutput1_split_test.erl
https://github.com/erlang/otp/blob/master/lib/xmerl/test/xmerl_sax_std_SUITE.erl
https://github.com/erlang/otp/blob/master/lib/crypto/test/crypto_SUITE.erl

Results
=======

Type                           | before     | after       | difference
-------------------------------|------------|-------------|------------
10k binaries                   |     303ms  |      140ms  |   -53.8%
10k maps                       |     326ms  |      156ms  |   -52.1%
10k records                    |     399ms  |      130ms  |   -67.4%
10k fun type                   |      n.a.  |       79ms  |
10k spec                       |      n.a.  |      176ms  |
gl.hrl                         |      47ms  |       24ms  |   -48.9%
re_testoutput1_split_test.erl  |     448ms  |      305ms  |   -31.9%
xmerl_sax_std_SUITE.erl        |     389ms  |      127ms  |   -67.4%
crypto_SUITE.erl               |      79ms  |       27ms  |   -65.8%
wbond pushed a commit that referenced this issue Oct 16, 2019
* [Erlang] Rewrite Syntax

Fixes #1609
Fixes #1724

According to #481 this PR provides an Oniguruma free Erlang syntax
definition written from scratch.

Outline
=======

The syntax definition is based on the http://erlang.org/doc and
includes all sematic features of Erlang/OTP 21.2 which are required
for highlighting.

It follows https://www.sublimetext.com/docs/3/syntax.html and includes
concepts from https://github.com/sublimehq/Packages.

Nearly every aspect and detail of the syntax definition was designed
from ground up making use of the latest features of ST's syntax engine
and with the intent to apply the most recent best practice examples
found in other recently updated syntax definitions.

Changes
=======

* Add support for Erlang ShellScript (escript)
* Add support for Erlang Type Language (`-type`, `-spec` directive)
* Add lots of detailed test cases
* Add some invalid illegal matching where appropriate
* Add builtin constants and macros highlighting
* Improve the list of builtin functions
  ([BIFs](http://erlang.org/doc/man/erlang.html#exports))
* Improve indention settings
* Improve shebang/editorconfig detection in `first_line_match`
* Improve support for _Goto Definition_ and _Goto Reference_
* Improve parsing performance by round about 50%
* Fix quoted atom highlighting (function names, module names, field names, ...)
* Fix bitstring data type highlighting
* Fix string escape highlighting
* Fix string placeholders highlighting
* Fix identifier break handling (`@` is no word-break)
* Fix lots of context pop offs
* ...

Benchmarks
==========

The whole design and development process included benchmarking of
different solutions to find the best possible performance for the
different sets of rules. The following examples provide an impression
of the performance improvements achieved with the new design.

**10k binary**

  Bin  = << << (X*2) >> || <<X>> <= << 1,2,3 >> >>.

**10k maps**

  Expr#{name=>"adam",{age,24}:=fct(),4.0=>{july,29}}

**10k records**

  #record{field1="val1", Field2=3, field3, 'Field-4'={}, _=atom}

**10k fun types**

  -type fun() :: fun(() -> int()).

**10k spec**

  -spec is_foo(Pid) -> boolean() when Pid :: pid() | if(); (Pid) -> ok.

**ErlangOTP files**

https://github.com/erlang/otp/blob/master/lib/wx/include/gl.hrl
https://github.com/erlang/otp/blob/master/lib/stdlib/test/re_testoutput1_split_test.erl
https://github.com/erlang/otp/blob/master/lib/xmerl/test/xmerl_sax_std_SUITE.erl
https://github.com/erlang/otp/blob/master/lib/crypto/test/crypto_SUITE.erl

Results
=======

Type                           | before     | after       | difference
-------------------------------|------------|-------------|------------
10k binaries                   |     303ms  |      140ms  |   -53.8%
10k maps                       |     326ms  |      156ms  |   -52.1%
10k records                    |     399ms  |      130ms  |   -67.4%
10k fun type                   |      n.a.  |       79ms  |
10k spec                       |      n.a.  |      176ms  |
gl.hrl                         |      47ms  |       24ms  |   -48.9%
re_testoutput1_split_test.erl  |     448ms  |      305ms  |   -31.9%
xmerl_sax_std_SUITE.erl        |     389ms  |      127ms  |   -67.4%
crypto_SUITE.erl               |      79ms  |       27ms  |   -65.8%

* [Erlang] Apply numeric scope guideline

This commit applies numeric constant scope names with regards to
the updated guideline which is a result of a proposal at #1877.

* [Erlang] Fix anonymous function definitions

This commit applies suggestions from the review process.

Including anonymous function definitions into goto definition features
makes not much sense on the one hand and brings too much noise on the
other. The anonymous function maybe assigned to a variable, but this is
a dynamic reference to the function then, which can be redefined at any
point.

Hence scoping the l-value of an anonymous function definition as
`entity.name.function` is removed. It's scoped as ordinary
`variable.other` instead.

* [Erlang] Fix record scopes

This commit applies suggestions from the review process.

A record is defined only by `-define(record_name, {...})` preprocessor.
Everything else can be considered usage. This simplifies the `record`
context a little bit.

* [Erlang] Remove anonymous functions from goto definition

* [Erlang] Add missing MACRO to symbol list

This commit adds macro definitions to the symbol list.

* [Erlang] Fix record field scopes

As the local creation/copy of records is no longer scoped as definition
the logical result is to handle the field access within the creation
expression as normal field-write-access instead of a field definition
as well.

To keep the field definition within preprocessor expressions intact,
the `record-body` is now assigned to record definitions only. A copy
and modified `record-fields` context is used within record creation
expressions.

Example:

    Expr#rec{field=1}
             ^^^^^ variable.other.field

Note:

  We might probably think about more general scopes for fields and
  records?

  Maybe:

   `variable.other.field` -> `variable.other.member`
   `variable.other.record` -> `variable.other.struct` or `variable.qualifier`??

* [Erlang] Fix spec/type scopes

This commit fixes an issue with clearing too many scopes in the
-spec() and -type() preprocessor expressions, which caused the main
scope `source.erlang` to be removed by accident.

* [Erlang] Remove obsolete symbol list override

As anonymous function definitions were removed recently, the
corresponding and therefore obsolete symbol list definition can be
removed as well.

* [Erlang] Update indexed symbol and reference lists

This commit...

1. removes `entity.name.record` from the references this scope was
   recently removed from all "usage" use-cases.
2. cleans the indexed symbol list a bit.

* [Erlang] Add meta.atom.erlang

This commit adds the `meta.atom.erlang` scope to all atoms. 

Note:
It's handled optional and therefore no tests are added.

* [Erlang] Fix fun scopes

This commit fixes an issue with clearing too many scopes in `fun()`
expressions.

* [Erlang] Update type test cases

This commit extends the test cases of `-type` expressions to verify not
to clear too many scopes in any situation.

* [Erlang] Fix macro in indexed symbol list

* [Erlang] Tweak symbol list

This commit applies review suggestions to the symbol list.

1. As all definitions are made in the preprocessor statements, the
   corresponding keywords are used as prefix for the symbol list item
   labels.

   <   ->  -export
   >   ->  -import
   #   ->  -record
   ::  ->  -spec, -type
   Ξ   ->  removed
   ...

2. Merges the "Function Macro" list into "Macro".
3. Fixes an issue with exports not being listed.
4. Fixes an issue with function definitions not being listed, which
   use macros as name (e.g.: `?macro(args) -> ok.`).

* [Erlang] Fix some emergency bailouts

This commit adds (some) missing emergency bailouts to pop off from
scopes not after a clause's end.

Effects:
 - function parameters
 - function call arguments
 - type preprocessor statements
 - illegal parens/brackets/braces

* [Erlang] Fix more emergency bailouts

This commit adds (some) missing emergency bailouts to pop off from
scopes not after a clause's end.

Effects:
 - groups
 - lists
 - maps
 - record fields
 - tuples

It also adds the `illegal-stray` context to scope the last closing
parentheses/bracket/brace `invalid.illegal` in order to indicate an
syntax error before the end of the clause. The idea behind is any
nested unclosed context to with the `.`.

* [Erlang] Add bailout from hex numbers

This commit adds a bailout to pop off from a hexadecimal number
context not after a clause's end.

* [Erlang] Add bailout from binary strings

This commit adds a bailout to pop off from a binary string context not
after a clause's end.

* [Erlang] Remove repetitions

This commit creates an `clause-end-or-stray` context to avoid including
the two `clause-end-pop` and `illegal-stray` one after each other
repetitively.

* [Erlang] Add bailout from function parameters

This commit makes sure to pop off from parameter lists of
function/spec/type definitions even with unbalanced parentheses/... .

* [Erlang] Add ... in list typespecs

The `...` is used in Erlang's typespec language to indicate an
arbitrary number of list items, parameters or tuple items.

This commit makes sure not to interfere with emergency bailouts.

The variable is used in ErlangOTP\erts\preloaded\src\erlang.erl

* [Erlang] Remove meta.function from macro definitions

Even though macro definitions may look like function definitions, they
are not related with them in any manner. The parameter list is just
used to create multi-purpose macros.

This commit therefore removes the `meta.function` scopes from
expressions:

   -define(MACRO_NAME(ARG1,ARG2) , ...).

* [Erlang] Rename _ scope

The `_` variable is called "anonymous variable" by Erlang.

Hence the scope is renamed accordingly.

* [Erlang] Add variable-anonymous-pop context

This commit is to reduce duplicated matches.

* [Erlang] Add variable-other-pop context

This commit is to reduce duplicated matches.

* [Erlang] Reorganize function-call context

This commit splits the function-call context into smaller pieces and
adds a `function-call-pop` for later use.

* [Erlang] Reorganize type-call context

This commit splits the type-call context into smaller pieces and adds
a `type-call-pop` for later use. The identifier context is moved to the
other "atom" contexts.

* [Erlang] Reorganize type-fun context

This commit splits the type-fun context into smaller pieces and adds
a `type-fun-pop` for later use.

* [Erlang] Merge fun-type into type-call context

Basically the fun-type context contains a special type of type-call.
Hence those contexts are merged.

This does not change the behavior in any way.

* [Erlang] Add meta.path

This commit applies the scope naming guideline which says:

Complete identifiers, including namespace names, should use the following scope. Such identifiers are the fully-qualified forms of variable, function and class names.

* [Erlang] Fix fun call identifier

The `variable.function` scope is generally used to scope an function
identifier in all default syntaxes. A normal function identifier in
Erlang is an atom. Means a normal function name must start with lower
case or underscore.

  my_func_name()

On the other hand, Erlang supports the definition of closures via fun
keyword, which binds a "function" to a real variable.

  // define the function
  MyLocalFunc = fun() ... end,

  // call the funciton
  MyLocalFunc(),

Scoping both, an ordinary function identifier and a closure as
`variable.function` is therefore confusing. As we already support
constants or macros to be used as identifiers

  %MY_FUNC_MACRO()

it is logical to scope `MyLocalFunc` as `variable.other` because it
is a real variable, which holds a function.

* [Erlang] Fix fun definition identifier

This commit applies the philosophy of the previous commit to
`fun` definitions and renames the `entity.name.function` to
`variable.other` for local fun identifiers.

  Fun1 = fun FunName(X) -> X+ FunName(X) end.
              ^ rename entity -> variable

* [Erlang] Fix fun definition identifier part 2

A local fun identifier really needs to be a variable.

Invalid: `fun funcName() -> ... end.`
Valid:   `fun FuncName() -> ... end.`

* [Erlang] Fix record field access in macros

* [Erlang] Fix mapping separator

This commit fixes an issue reported during the review process, which
caused the `:=` key-value separator not to be scoped correctly.
@jrappen
Copy link
Contributor

jrappen commented Nov 22, 2021

image

@deathaxe
Copy link
Collaborator

Current JSON syntax doesn't suffer from sregex incompatible patterns.

@jrappen
Copy link
Contributor

jrappen commented Nov 22, 2021

@deathaxe I am aware. I just posted this for the answers to my question, as Will didn't mention what triggers a fallback here.

jrappen added a commit to jrappen/Packages that referenced this issue Dec 2, 2021
- Using inheritance split up `JSON.sublime-syntax` into:
    - `JSON (Basic).sublime-syntax` with `scope:source.json.basic`
    - `JSON.sublime-syntax` with `scope:source.json`
    - `JSONC.sublime-syntax` with `scope:source.json.jsonc`
    - `JSON5.sublime-syntax` with `scope:source.json.json5`
- Add many more file extensions for `JSON` & `JSONC`:
    - add doc links to extensions where applicable as a reference to be
      able to more quickly verify that they (still) use said syntax
      flavor
- JSON:
    - (correctly formatted) JSON code can now be prettified or minified
      via the context menu or the command palette
    - highlight leading, trailing & multiple commas as invalid
    - only allow exactly one structure (object, array) or value
      (constant, number, string) at top level (thanks to Keith)
- JSONC:
    - highlight some files by default as `JSONC` (as decided by Jon
      in sublimehq#285)
    - highlight leading & multiple commas as invalid, trailing as valid
- JSON5:
    - explicitly pos numbers, hexadecimal ints, Infinity and NaN
    - single quoted strings
    - more escape chars for strings
    - ECMA identifierName as object keys (thanks to Thomas)
        - scoped as plain unquoted strings
    - line continuation in strings (with tests thanks to Keith)
- Objects:
    - Add `meta.toc-list` scope to top level object keys to add them to
      the symbol list (also add tests, see below)
    - Highlighting speed improvements for empty objects (thanks to
      FichteFoll)
    - Make `mapping.*` contexts more modular
- Arrays:
    - Highlighting speed improvements for empty arrays (thanks to
      FichteFoll)
- Numbers:
    - Correctly scope number signs with `constant.numeric.sign` instead
      of `keyword.operator.arithmetic`
    - Significantly improve number highlighting (thanks to deathaxe)
- Syntaxes:
    - Make use of newer syntax features including those only available
      in `version: 2` syntaxes
    - Make use of `variables` (with optimizations provided by deathaxe
      and regex patterns provided by Thomas)
- Tests:
    - Significantly extend tests to cover more parts of the syntaxes
      defined.
    - Split original test file into logical parts
    - Add indentation tests for:
        - `json`, `jsonc` & `json5`
        - `mapping` (objects), `sequence` (arrays)
    - Add symbols tests for:
        - scope: `meta.toc-list.json | meta.toc-list.json5`
        - languages: `json`, `jsonc` & `json5`
    - Fix tests for `meta.mapping meta.mapping.*`
- Leave `JSON` headers in `Markdown` as `json` only, but split up fenced
  code blocks into `json`, `jsonc` & `json5` to behave similarly to
  `GitHub Flavored Markdown`


BREAKING CHANGES:

- JSON does not have values that can be set via an inline calculation
  with the help of operators, but only simple number values. Scopes for
  number signs have changed from being `keyword.operator.arithmetic` to
  `constant.numeric.sign`. Color scheme authors should add this, should
  it be missing.
- The `JSON.sublime-syntax` now marks comments as `invalid`, third party
  plugin authors should target `JSONC.sublime-syntax` instead to have
  the same user experience as before.


- fix sublimehq#285
- address sublimehq#481 to remove incompatible regex patterns according to Will
- address sublimehq#757 to fix line comments for `JSONC` (thanks to Keith)
- address sublimehq#2430 using sort-order (as requested by deathaxe)
- address sublimehq#2852 to fix scopes of curly braces & square brackets in
  `JSON` (thanks to Thomas)
- address sublimehq/sublime_text#3154 and add symbol tests


Co-authored-by: Ashwin Shenoy <Ultra-Instinct-05@users.noreply.github.com>
Co-authored-by: Jack Cherng <jfcherng@users.noreply.github.com>
Co-authored-by: Janos Wortmann <jwortmann@users.noreply.github.com>
Co-authored-by: Jon Skinner <jps@sublimetext.com>
Co-authored-by: FichteFoll <FichteFoll@users.noreply.github.com>
Co-authored-by: Keith Hall <keith-hall@users.noreply.github.com>
Co-authored-by: Michael B. Lyons <michaelblyons@users.noreply.github.com>
Co-authored-by: Rafał Chłodnicki <rchl@users.noreply.github.com>
Co-authored-by: Thomas Smith <Thom1729@users.noreply.github.com>
Co-authored-by: Will Bond <wbond@users.noreply.github.com>
Co-authored-by: deathaxe <deathaxe@users.noreply.github.com>
mitranim pushed a commit to mitranim/Packages that referenced this issue Mar 25, 2022
* [Erlang] Rewrite Syntax

Fixes sublimehq#1609
Fixes sublimehq#1724

According to sublimehq#481 this PR provides an Oniguruma free Erlang syntax
definition written from scratch.

Outline
=======

The syntax definition is based on the http://erlang.org/doc and
includes all sematic features of Erlang/OTP 21.2 which are required
for highlighting.

It follows https://www.sublimetext.com/docs/3/syntax.html and includes
concepts from https://github.com/sublimehq/Packages.

Nearly every aspect and detail of the syntax definition was designed
from ground up making use of the latest features of ST's syntax engine
and with the intent to apply the most recent best practice examples
found in other recently updated syntax definitions.

Changes
=======

* Add support for Erlang ShellScript (escript)
* Add support for Erlang Type Language (`-type`, `-spec` directive)
* Add lots of detailed test cases
* Add some invalid illegal matching where appropriate
* Add builtin constants and macros highlighting
* Improve the list of builtin functions
  ([BIFs](http://erlang.org/doc/man/erlang.html#exports))
* Improve indention settings
* Improve shebang/editorconfig detection in `first_line_match`
* Improve support for _Goto Definition_ and _Goto Reference_
* Improve parsing performance by round about 50%
* Fix quoted atom highlighting (function names, module names, field names, ...)
* Fix bitstring data type highlighting
* Fix string escape highlighting
* Fix string placeholders highlighting
* Fix identifier break handling (`@` is no word-break)
* Fix lots of context pop offs
* ...

Benchmarks
==========

The whole design and development process included benchmarking of
different solutions to find the best possible performance for the
different sets of rules. The following examples provide an impression
of the performance improvements achieved with the new design.

**10k binary**

  Bin  = << << (X*2) >> || <<X>> <= << 1,2,3 >> >>.

**10k maps**

  Expr#{name=>"adam",{age,24}:=fct(),4.0=>{july,29}}

**10k records**

  #record{field1="val1", Field2=3, field3, 'Field-4'={}, _=atom}

**10k fun types**

  -type fun() :: fun(() -> int()).

**10k spec**

  -spec is_foo(Pid) -> boolean() when Pid :: pid() | if(); (Pid) -> ok.

**ErlangOTP files**

https://github.com/erlang/otp/blob/master/lib/wx/include/gl.hrl
https://github.com/erlang/otp/blob/master/lib/stdlib/test/re_testoutput1_split_test.erl
https://github.com/erlang/otp/blob/master/lib/xmerl/test/xmerl_sax_std_SUITE.erl
https://github.com/erlang/otp/blob/master/lib/crypto/test/crypto_SUITE.erl

Results
=======

Type                           | before     | after       | difference
-------------------------------|------------|-------------|------------
10k binaries                   |     303ms  |      140ms  |   -53.8%
10k maps                       |     326ms  |      156ms  |   -52.1%
10k records                    |     399ms  |      130ms  |   -67.4%
10k fun type                   |      n.a.  |       79ms  |
10k spec                       |      n.a.  |      176ms  |
gl.hrl                         |      47ms  |       24ms  |   -48.9%
re_testoutput1_split_test.erl  |     448ms  |      305ms  |   -31.9%
xmerl_sax_std_SUITE.erl        |     389ms  |      127ms  |   -67.4%
crypto_SUITE.erl               |      79ms  |       27ms  |   -65.8%

* [Erlang] Apply numeric scope guideline

This commit applies numeric constant scope names with regards to
the updated guideline which is a result of a proposal at sublimehq#1877.

* [Erlang] Fix anonymous function definitions

This commit applies suggestions from the review process.

Including anonymous function definitions into goto definition features
makes not much sense on the one hand and brings too much noise on the
other. The anonymous function maybe assigned to a variable, but this is
a dynamic reference to the function then, which can be redefined at any
point.

Hence scoping the l-value of an anonymous function definition as
`entity.name.function` is removed. It's scoped as ordinary
`variable.other` instead.

* [Erlang] Fix record scopes

This commit applies suggestions from the review process.

A record is defined only by `-define(record_name, {...})` preprocessor.
Everything else can be considered usage. This simplifies the `record`
context a little bit.

* [Erlang] Remove anonymous functions from goto definition

* [Erlang] Add missing MACRO to symbol list

This commit adds macro definitions to the symbol list.

* [Erlang] Fix record field scopes

As the local creation/copy of records is no longer scoped as definition
the logical result is to handle the field access within the creation
expression as normal field-write-access instead of a field definition
as well.

To keep the field definition within preprocessor expressions intact,
the `record-body` is now assigned to record definitions only. A copy
and modified `record-fields` context is used within record creation
expressions.

Example:

    Expr#rec{field=1}
             ^^^^^ variable.other.field

Note:

  We might probably think about more general scopes for fields and
  records?

  Maybe:

   `variable.other.field` -> `variable.other.member`
   `variable.other.record` -> `variable.other.struct` or `variable.qualifier`??

* [Erlang] Fix spec/type scopes

This commit fixes an issue with clearing too many scopes in the
-spec() and -type() preprocessor expressions, which caused the main
scope `source.erlang` to be removed by accident.

* [Erlang] Remove obsolete symbol list override

As anonymous function definitions were removed recently, the
corresponding and therefore obsolete symbol list definition can be
removed as well.

* [Erlang] Update indexed symbol and reference lists

This commit...

1. removes `entity.name.record` from the references this scope was
   recently removed from all "usage" use-cases.
2. cleans the indexed symbol list a bit.

* [Erlang] Add meta.atom.erlang

This commit adds the `meta.atom.erlang` scope to all atoms. 

Note:
It's handled optional and therefore no tests are added.

* [Erlang] Fix fun scopes

This commit fixes an issue with clearing too many scopes in `fun()`
expressions.

* [Erlang] Update type test cases

This commit extends the test cases of `-type` expressions to verify not
to clear too many scopes in any situation.

* [Erlang] Fix macro in indexed symbol list

* [Erlang] Tweak symbol list

This commit applies review suggestions to the symbol list.

1. As all definitions are made in the preprocessor statements, the
   corresponding keywords are used as prefix for the symbol list item
   labels.

   <   ->  -export
   >   ->  -import
   #   ->  -record
   ::  ->  -spec, -type
   Ξ   ->  removed
   ...

2. Merges the "Function Macro" list into "Macro".
3. Fixes an issue with exports not being listed.
4. Fixes an issue with function definitions not being listed, which
   use macros as name (e.g.: `?macro(args) -> ok.`).

* [Erlang] Fix some emergency bailouts

This commit adds (some) missing emergency bailouts to pop off from
scopes not after a clause's end.

Effects:
 - function parameters
 - function call arguments
 - type preprocessor statements
 - illegal parens/brackets/braces

* [Erlang] Fix more emergency bailouts

This commit adds (some) missing emergency bailouts to pop off from
scopes not after a clause's end.

Effects:
 - groups
 - lists
 - maps
 - record fields
 - tuples

It also adds the `illegal-stray` context to scope the last closing
parentheses/bracket/brace `invalid.illegal` in order to indicate an
syntax error before the end of the clause. The idea behind is any
nested unclosed context to with the `.`.

* [Erlang] Add bailout from hex numbers

This commit adds a bailout to pop off from a hexadecimal number
context not after a clause's end.

* [Erlang] Add bailout from binary strings

This commit adds a bailout to pop off from a binary string context not
after a clause's end.

* [Erlang] Remove repetitions

This commit creates an `clause-end-or-stray` context to avoid including
the two `clause-end-pop` and `illegal-stray` one after each other
repetitively.

* [Erlang] Add bailout from function parameters

This commit makes sure to pop off from parameter lists of
function/spec/type definitions even with unbalanced parentheses/... .

* [Erlang] Add ... in list typespecs

The `...` is used in Erlang's typespec language to indicate an
arbitrary number of list items, parameters or tuple items.

This commit makes sure not to interfere with emergency bailouts.

The variable is used in ErlangOTP\erts\preloaded\src\erlang.erl

* [Erlang] Remove meta.function from macro definitions

Even though macro definitions may look like function definitions, they
are not related with them in any manner. The parameter list is just
used to create multi-purpose macros.

This commit therefore removes the `meta.function` scopes from
expressions:

   -define(MACRO_NAME(ARG1,ARG2) , ...).

* [Erlang] Rename _ scope

The `_` variable is called "anonymous variable" by Erlang.

Hence the scope is renamed accordingly.

* [Erlang] Add variable-anonymous-pop context

This commit is to reduce duplicated matches.

* [Erlang] Add variable-other-pop context

This commit is to reduce duplicated matches.

* [Erlang] Reorganize function-call context

This commit splits the function-call context into smaller pieces and
adds a `function-call-pop` for later use.

* [Erlang] Reorganize type-call context

This commit splits the type-call context into smaller pieces and adds
a `type-call-pop` for later use. The identifier context is moved to the
other "atom" contexts.

* [Erlang] Reorganize type-fun context

This commit splits the type-fun context into smaller pieces and adds
a `type-fun-pop` for later use.

* [Erlang] Merge fun-type into type-call context

Basically the fun-type context contains a special type of type-call.
Hence those contexts are merged.

This does not change the behavior in any way.

* [Erlang] Add meta.path

This commit applies the scope naming guideline which says:

Complete identifiers, including namespace names, should use the following scope. Such identifiers are the fully-qualified forms of variable, function and class names.

* [Erlang] Fix fun call identifier

The `variable.function` scope is generally used to scope an function
identifier in all default syntaxes. A normal function identifier in
Erlang is an atom. Means a normal function name must start with lower
case or underscore.

  my_func_name()

On the other hand, Erlang supports the definition of closures via fun
keyword, which binds a "function" to a real variable.

  // define the function
  MyLocalFunc = fun() ... end,

  // call the funciton
  MyLocalFunc(),

Scoping both, an ordinary function identifier and a closure as
`variable.function` is therefore confusing. As we already support
constants or macros to be used as identifiers

  %MY_FUNC_MACRO()

it is logical to scope `MyLocalFunc` as `variable.other` because it
is a real variable, which holds a function.

* [Erlang] Fix fun definition identifier

This commit applies the philosophy of the previous commit to
`fun` definitions and renames the `entity.name.function` to
`variable.other` for local fun identifiers.

  Fun1 = fun FunName(X) -> X+ FunName(X) end.
              ^ rename entity -> variable

* [Erlang] Fix fun definition identifier part 2

A local fun identifier really needs to be a variable.

Invalid: `fun funcName() -> ... end.`
Valid:   `fun FuncName() -> ... end.`

* [Erlang] Fix record field access in macros

* [Erlang] Fix mapping separator

This commit fixes an issue reported during the review process, which
caused the `:=` key-value separator not to be scoped correctly.
jrappen added a commit to jrappen/Packages that referenced this issue May 31, 2022
- Using inheritance split up `JSON.sublime-syntax` into:
    - `JSON.sublime-syntax` with `scope:source.json`
    - `JSONC.sublime-syntax` with `scope:source.json.jsonc`
    - `JSON5.sublime-syntax` with `scope:source.json.json5`
    - `JSON_dotNET.sublime-syntax` with `scope:source.json.json-dotnet`
- Add many more file extensions for `JSON` & `JSONC`:
    - add doc links to extensions where applicable as a reference to be
      able to more quickly verify that they (still) use said syntax
      flavor
- JSON:
    - Make use of newer syntax features including those only available
      in `version: 2` syntaxes
    - Make use of `variables` (with optimizations provided by @deathaxe
      and regex patterns provided by @Thom1729)
    - Context names now more closely match the naming scheme of other
      (recently re-written) default syntaxes
    - (correctly formatted) JSON code can now be prettified or minified
      via the context menu or the command palette. JSON code can
      optionally be auto-prettified on pre save events.
    - highlight leading, trailing & multiple commas as invalid
    - only allow exactly one structure (object, array) or value
      (constant, number, string) at top level (thanks to @keith-hall)
    - links (`meta.link.inet`) and email addresses (`meta.link.email`) are
      scoped the same as in Markdown (thanks to @deathaxe)
- JSONC:
    - highlight some files by default as `JSONC` (as decided by
      @jskinner in sublimehq#285)
    - highlight leading & multiple commas as invalid, trailing as valid
    - scope empty block comments as such
    - support syntax based folding of ST4131+,
      compare sublimehq#3291
- JSON5:
    - explicitly pos numbers, hexadecimal ints, Infinity and NaN
    - single quoted strings
    - more escape chars for strings
    - ECMA identifierName as object keys (regexes thanks to @Thom1729)
        - scoped as plain unquoted strings (thanks to @Thom1729)
        - support string interpolation (thanks to @deathaxe)
    - line continuation in strings (with tests thanks to @keith-hall)
- JSON.NET:
    - support requested by @keith-hall,
      built with feedback from @michaelblyons
- Objects:
    - Highlighting speed improvements for empty objects (thanks to
      @FichteFoll)
    - Make `mapping.*` contexts more modular
- Arrays:
    - Highlighting speed improvements for empty arrays (thanks to
      @FichteFoll)
- Numbers:
    - Correctly scope number signs with `constant.numeric.sign` instead
      of `keyword.operator.arithmetic`
    - Significantly improve number highlighting (thanks to @deathaxe)
- Completions:
    - completions have been added for language constants, including kind info
      and details (with links to docs)
        - `null`, `false`, `true` for JSON
        - `Infinity` and `NaN` for JSON5
- Settings:
    - a `default_extension` is now set for all JSON flavors
- Symbol index:
    - with an object structure at the top-level, only top-level keys
      within now show up in the index (including tests for symbols and
      syntax)
- Tests:
    - test files now test the base scope
    - Significantly extend tests to cover more parts of the syntaxes
    - Split original test file into logical parts
    - Add indentation tests for:
        - `json`, `json5` & `jsonc`
        - `mapping` (objects), `sequence` (arrays)
    - Add symbols tests for:
        - top-level keys of object structures (thanks to deathaxe)
        - languages: `json`, `json5` & `jsonc`
    - Fix tests for `meta.mapping meta.mapping.*`
- Leave `JSON` headers in `Markdown` as `json` only, but split up fenced
  code blocks into `json`, `json5` & `jsonc` to behave similarly to
  `GitHub Flavored Markdown`

BREAKING CHANGES:

- JSON does not have values that can be set via an inline calculation
  with the help of operators, but only simple number values. Scopes for
  number signs have changed from being `keyword.operator.arithmetic` to
  `constant.numeric.sign`. Color scheme authors should add this, should
  it be missing.
- The `JSON.sublime-syntax` now marks comments as `invalid`, third party
  plugin authors should instead target `JSONC.sublime-syntax` to keep
  the user experience as-is.
- Indexed symbols (i.e. top-level keys in JSON object structures) are
  scoped as `source.json meta.mapping.key - (meta.mapping.value meta.mapping.key | meta.sequence.list meta.mapping.key)`.
  Color scheme authors should add special highlighting to differentiate
  them from other keys.

- fix sublimehq#285
- address sublimehq#421 (thanks to @FichteFoll)
- address sublimehq#481 to remove incompatible regex patterns according
  to @wbond
- address sublimehq#757 to fix line comments for `JSONC` (thanks to
  @keith-hall)
- address sublimehq#2430 using sort-order (as requested by @deathaxe)
- address sublimehq#2711 with regards to `constant.language.null`
  vs. `constant.language.empty` (thanks to @FichteFoll)
- address sublimehq#2852 to fix scopes of curly braces & square
  brackets in `JSON` (thanks to @Thom1729)
- address sublimehq#3228 to fix `punctuation.separator` scopes,
  compare sublimehq#3270
- address sublimehq/sublime_text#3154 and add symbol tests

Co-authored-by: Ashwin Shenoy <Ultra-Instinct-05@users.noreply.github.com>
Co-authored-by: Jack Cherng <jfcherng@users.noreply.github.com>
Co-authored-by: Janos Wortmann <jwortmann@users.noreply.github.com>
Co-authored-by: Jon Skinner <jps@sublimetext.com>
Co-authored-by: FichteFoll <FichteFoll@users.noreply.github.com>
Co-authored-by: Keith Hall <keith-hall@users.noreply.github.com>
Co-authored-by: Michael B. Lyons <michaelblyons@users.noreply.github.com>
Co-authored-by: Rafał Chłodnicki <rchl@users.noreply.github.com>
Co-authored-by: Thomas Smith <Thom1729@users.noreply.github.com>
Co-authored-by: Will Bond <wbond@users.noreply.github.com>
Co-authored-by: deathaxe <deathaxe@users.noreply.github.com>
jrappen added a commit to jrappen/Packages that referenced this issue May 31, 2022
- Using inheritance split up `JSON.sublime-syntax` into:
    - `JSON.sublime-syntax` with `scope:source.json`
    - `JSONC.sublime-syntax` with `scope:source.json.jsonc`
    - `JSON5.sublime-syntax` with `scope:source.json.json5`
    - `JSON_dotNET.sublime-syntax` with `scope:source.json.json-dotnet`
- Add many more file extensions for `JSON` & `JSONC`:
    - add doc links to extensions where applicable as a reference to be
      able to more quickly verify that they (still) use said syntax
      flavor
- JSON:
    - Make use of newer syntax features including those only available
      in `version: 2` syntaxes
    - Make use of `variables` (with optimizations provided by @deathaxe
      and regex patterns provided by @Thom1729)
    - Context names now more closely match the naming scheme of other
      (recently re-written) default syntaxes
    - (correctly formatted) JSON code can now be prettified or minified
      via the context menu or the command palette. JSON code can
      optionally be auto-prettified on pre save events.
    - highlight leading, trailing & multiple commas as invalid
    - only allow exactly one structure (object, array) or value
      (constant, number, string) at top level (thanks to @keith-hall)
    - links (`meta.link.inet`) and email addresses (`meta.link.email`) are
      scoped the same as in Markdown (thanks to @deathaxe)
- JSONC:
    - highlight some files by default as `JSONC` (as decided by
      @jskinner in sublimehq#285)
    - highlight leading & multiple commas as invalid, trailing as valid
    - scope empty block comments as such
    - support syntax based folding of ST4131+,
      compare sublimehq#3291
- JSON5:
    - explicitly pos numbers, hexadecimal ints, Infinity and NaN
    - single quoted strings
    - more escape chars for strings
    - ECMA identifierName as object keys (regexes thanks to @Thom1729)
        - scoped as plain unquoted strings (thanks to @Thom1729)
        - support string interpolation (thanks to @deathaxe)
    - line continuation in strings (with tests thanks to @keith-hall)
- JSON.NET:
    - support requested by @keith-hall,
      built with feedback from @michaelblyons
- Objects:
    - Highlighting speed improvements for empty objects (thanks to
      @FichteFoll)
    - Make `mapping.*` contexts more modular
- Arrays:
    - Highlighting speed improvements for empty arrays (thanks to
      @FichteFoll)
- Numbers:
    - Correctly scope number signs with `constant.numeric.sign` instead
      of `keyword.operator.arithmetic`
    - Significantly improve number highlighting (thanks to @deathaxe)
- Completions:
    - completions have been added for language constants, including kind info
      and details (with links to docs)
        - `null`, `false`, `true` for JSON
        - `Infinity` and `NaN` for JSON5
- Settings:
    - a `default_extension` is now set for all JSON flavors
- Symbol index:
    - with an object structure at the top-level, only top-level keys
      within now show up in the index (including tests for symbols and
      syntax)
- Tests:
    - test files now test the base scope
    - Significantly extend tests to cover more parts of the syntaxes
    - Split original test file into logical parts
    - Add indentation tests for:
        - `json`, `json5` & `jsonc`
        - `mapping` (objects), `sequence` (arrays)
    - Add symbols tests for:
        - top-level keys of object structures (thanks to deathaxe)
        - languages: `json`, `json5` & `jsonc`
    - Fix tests for `meta.mapping meta.mapping.*`
- Leave `JSON` headers in `Markdown` as `json` only, but split up fenced
  code blocks into `json`, `json5` & `jsonc` to behave similarly to
  `GitHub Flavored Markdown`

BREAKING CHANGES:

- JSON does not have values that can be set via an inline calculation
  with the help of operators, but only simple number values. Scopes for
  number signs have changed from being `keyword.operator.arithmetic` to
  `constant.numeric.sign`. Color scheme authors should add this, should
  it be missing.
- The `JSON.sublime-syntax` now marks comments as `invalid`, third party
  plugin authors should instead target `JSONC.sublime-syntax` to keep
  the user experience as-is.
- Indexed symbols (i.e. top-level keys in JSON object structures) are
  scoped as `source.json meta.mapping.key - (meta.mapping.value meta.mapping.key | meta.sequence.list meta.mapping.key)`.
  Color scheme authors should add special highlighting to differentiate
  them from other keys.

- fix sublimehq#285
- address sublimehq#421 (thanks to @FichteFoll)
- address sublimehq#481 to remove incompatible regex patterns according
  to @wbond
- address sublimehq#757 to fix line comments for `JSONC` (thanks to
  @keith-hall)
- address sublimehq#2430 using sort-order (as requested by @deathaxe)
- address sublimehq#2711 with regards to `constant.language.null`
  vs. `constant.language.empty` (thanks to @FichteFoll)
- address sublimehq#2852 to fix scopes of curly braces & square
  brackets in `JSON` (thanks to @Thom1729)
- address sublimehq#3228 to fix `punctuation.separator` scopes,
  compare sublimehq#3270
- address sublimehq/sublime_text#3154 and add symbol tests

Co-authored-by: Ashwin Shenoy <Ultra-Instinct-05@users.noreply.github.com>
Co-authored-by: Jack Cherng <jfcherng@users.noreply.github.com>
Co-authored-by: Janos Wortmann <jwortmann@users.noreply.github.com>
Co-authored-by: Jon Skinner <jps@sublimetext.com>
Co-authored-by: FichteFoll <FichteFoll@users.noreply.github.com>
Co-authored-by: Keith Hall <keith-hall@users.noreply.github.com>
Co-authored-by: Michael B. Lyons <michaelblyons@users.noreply.github.com>
Co-authored-by: Rafał Chłodnicki <rchl@users.noreply.github.com>
Co-authored-by: Thomas Smith <Thom1729@users.noreply.github.com>
Co-authored-by: Will Bond <wbond@users.noreply.github.com>
Co-authored-by: deathaxe <deathaxe@users.noreply.github.com>
jrappen added a commit to jrappen/Packages that referenced this issue May 31, 2022
- Using inheritance split up `JSON.sublime-syntax` into:
    - `JSON.sublime-syntax` with `scope:source.json`
    - `JSONC.sublime-syntax` with `scope:source.json.jsonc`
    - `JSON5.sublime-syntax` with `scope:source.json.json5`
    - `JSON_dotNET.sublime-syntax` with `scope:source.json.json-dotnet`
- Add many more file extensions for `JSON` & `JSONC`:
    - add doc links to extensions where applicable as a reference to be
      able to more quickly verify that they (still) use said syntax
      flavor
- JSON:
    - Make use of newer syntax features including those only available
      in `version: 2` syntaxes
    - Make use of `variables` (with optimizations provided by @deathaxe
      and regex patterns provided by @Thom1729)
    - Context names now more closely match the naming scheme of other
      (recently re-written) default syntaxes
    - (correctly formatted) JSON code can now be prettified or minified
      via the context menu or the command palette. JSON code can
      optionally be auto-prettified on pre save events.
    - highlight leading, trailing & multiple commas as invalid
    - only allow exactly one structure (object, array) or value
      (constant, number, string) at top level (thanks to @keith-hall)
    - links (`meta.link.inet`) and email addresses (`meta.link.email`) are
      scoped the same as in Markdown (thanks to @deathaxe)
- JSONC:
    - highlight some files by default as `JSONC` (as decided by
      @jskinner in sublimehq#285)
    - highlight leading & multiple commas as invalid, trailing as valid
    - scope empty block comments as such
    - support syntax based folding of ST4131+,
      compare sublimehq#3291
- JSON5:
    - explicitly pos numbers, hexadecimal ints, Infinity and NaN
    - single quoted strings
    - more escape chars for strings
    - ECMA identifierName as object keys (regexes thanks to @Thom1729)
        - scoped as plain unquoted strings (thanks to @Thom1729)
        - support string interpolation (thanks to @deathaxe)
    - line continuation in strings (with tests thanks to @keith-hall)
- JSON.NET:
    - support requested by @keith-hall,
      built with feedback from @michaelblyons
- Objects:
    - Highlighting speed improvements for empty objects (thanks to
      @FichteFoll)
    - Make `mapping.*` contexts more modular
- Arrays:
    - Highlighting speed improvements for empty arrays (thanks to
      @FichteFoll)
- Numbers:
    - Correctly scope number signs with `constant.numeric.sign` instead
      of `keyword.operator.arithmetic`
    - Significantly improve number highlighting (thanks to @deathaxe)
- Completions:
    - completions have been added for language constants, including kind info
      and details (with links to docs)
        - `null`, `false`, `true` for JSON
        - `Infinity` and `NaN` for JSON5
- Settings:
    - a `default_extension` is now set for all JSON flavors
- Symbol index:
    - with an object structure at the top-level, only top-level keys
      within now show up in the index (including tests for symbols and
      syntax)
- Tests:
    - test files now test the base scope
    - Significantly extend tests to cover more parts of the syntaxes
    - Split original test file into logical parts
    - Add indentation tests for:
        - `json`, `json5` & `jsonc`
        - `mapping` (objects), `sequence` (arrays)
    - Add symbols tests for:
        - top-level keys of object structures (thanks to deathaxe)
        - languages: `json`, `json5` & `jsonc`
    - Fix tests for `meta.mapping meta.mapping.*`
- Leave `JSON` headers in `Markdown` as `json` only, but split up fenced
  code blocks into `json`, `json5` & `jsonc` to behave similarly to
  `GitHub Flavored Markdown`

BREAKING CHANGES:

- JSON does not have values that can be set via an inline calculation
  with the help of operators, but only simple number values. Scopes for
  number signs have changed from being `keyword.operator.arithmetic` to
  `constant.numeric.sign`. Color scheme authors should add this, should
  it be missing.
- The `JSON.sublime-syntax` now marks comments as `invalid`, third party
  plugin authors should instead target `JSONC.sublime-syntax` to keep
  the user experience as-is.
- Indexed symbols (i.e. top-level keys in JSON object structures) are
  scoped as `source.json meta.mapping.key - (meta.mapping.value meta.mapping.key | meta.sequence.list meta.mapping.key)`.
  Color scheme authors should add special highlighting to differentiate
  them from other keys.

- fix sublimehq#285
- address sublimehq#421 (thanks to @FichteFoll)
- address sublimehq#481 to remove incompatible regex patterns according
  to @wbond
- address sublimehq#757 to fix line comments for `JSONC` (thanks to
  @keith-hall)
- address sublimehq#2430 using sort-order (as requested by @deathaxe)
- address sublimehq#2711 with regards to `constant.language.null`
  vs. `constant.language.empty` (thanks to @FichteFoll)
- address sublimehq#2852 to fix scopes of curly braces & square
  brackets in `JSON` (thanks to @Thom1729)
- address sublimehq#3228 to fix `punctuation.separator` scopes,
  compare sublimehq#3270
- address sublimehq/sublime_text#3154 and add symbol tests

Co-authored-by: Ashwin Shenoy <Ultra-Instinct-05@users.noreply.github.com>
Co-authored-by: Jack Cherng <jfcherng@users.noreply.github.com>
Co-authored-by: Janos Wortmann <jwortmann@users.noreply.github.com>
Co-authored-by: Jon Skinner <jps@sublimetext.com>
Co-authored-by: FichteFoll <FichteFoll@users.noreply.github.com>
Co-authored-by: Keith Hall <keith-hall@users.noreply.github.com>
Co-authored-by: Michael B. Lyons <michaelblyons@users.noreply.github.com>
Co-authored-by: Rafał Chłodnicki <rchl@users.noreply.github.com>
Co-authored-by: Thomas Smith <Thom1729@users.noreply.github.com>
Co-authored-by: Will Bond <wbond@users.noreply.github.com>
Co-authored-by: deathaxe <deathaxe@users.noreply.github.com>
jrappen added a commit to jrappen/Packages that referenced this issue Jun 1, 2022
- Using inheritance split up `JSON.sublime-syntax` into:
    - `JSON.sublime-syntax` with `scope:source.json`
    - `JSONC.sublime-syntax` with `scope:source.json.jsonc`
    - `JSON5.sublime-syntax` with `scope:source.json.json5`
    - `JSON_dotNET.sublime-syntax` with `scope:source.json.json-dotnet`
- Add many more file extensions for `JSON` & `JSONC`:
    - add doc links to extensions where applicable as a reference to be
      able to more quickly verify that they (still) use said syntax
      flavor
- JSON:
    - Make use of newer syntax features including those only available
      in `version: 2` syntaxes
    - Make use of `variables` (with optimizations provided by @deathaxe
      and regex patterns provided by @Thom1729)
    - Context names now more closely match the naming scheme of other
      (recently re-written) default syntaxes
    - (correctly formatted) JSON code can now be prettified or minified
      via the context menu or the command palette. JSON code can
      optionally be auto-prettified on pre save events.
    - highlight leading, trailing & multiple commas as invalid
    - only allow exactly one structure (object, array) or value
      (constant, number, string) at top level (thanks to @keith-hall)
    - links (`meta.link.inet`) and email addresses (`meta.link.email`) are
      scoped the same as in Markdown (thanks to @deathaxe)
- JSONC:
    - highlight some files by default as `JSONC` (as decided by
      @jskinner in sublimehq#285)
    - highlight leading & multiple commas as invalid, trailing as valid
    - scope empty block comments as such
    - support syntax based folding of ST4131+,
      compare sublimehq#3291
- JSON5:
    - explicitly pos numbers, hexadecimal ints, Infinity and NaN
    - single quoted strings
    - more escape chars for strings
    - ECMA identifierName as object keys (regexes thanks to @Thom1729)
        - scoped as plain unquoted strings (thanks to @Thom1729)
        - support string interpolation (thanks to @deathaxe)
    - line continuation in strings (with tests thanks to @keith-hall)
- JSON.NET:
    - support requested by @keith-hall,
      built with feedback from @michaelblyons
- Objects:
    - Highlighting speed improvements for empty objects (thanks to
      @FichteFoll)
    - Make `mapping.*` contexts more modular
- Arrays:
    - Highlighting speed improvements for empty arrays (thanks to
      @FichteFoll)
- Numbers:
    - Correctly scope number signs with `constant.numeric.sign` instead
      of `keyword.operator.arithmetic`
    - Significantly improve number highlighting (thanks to @deathaxe)
- Completions:
    - completions have been added for language constants, including kind info
      and details (with links to docs)
        - `null`, `false`, `true` for JSON
        - `Infinity` and `NaN` for JSON5
- Settings:
    - a `default_extension` is now set for all JSON flavors
- Symbol index:
    - with an object structure at the top-level, only top-level keys
      within now show up in the index (including tests for symbols and
      syntax)
- Tests:
    - test files now test the base scope
    - Significantly extend tests to cover more parts of the syntaxes
    - Split original test file into logical parts
    - Add indentation tests for:
        - `json`, `json5` & `jsonc`
        - `mapping` (objects), `sequence` (arrays)
    - Add symbols tests for:
        - top-level keys of object structures (thanks to deathaxe)
        - languages: `json`, `json5` & `jsonc`
    - Fix tests for `meta.mapping meta.mapping.*`
- Leave `JSON` headers in `Markdown` as `json` only, but split up fenced
  code blocks into `json`, `json5` & `jsonc` to behave similarly to
  `GitHub Flavored Markdown`

BREAKING CHANGES:

- JSON does not have values that can be set via an inline calculation
  with the help of operators, but only simple number values. Scopes for
  number signs have changed from being `keyword.operator.arithmetic` to
  `constant.numeric.sign`. Color scheme authors should add this, should
  it be missing.
- The `JSON.sublime-syntax` now marks comments as `invalid`, third party
  plugin authors should instead target `JSONC.sublime-syntax` to keep
  the user experience as-is.
- Indexed symbols (i.e. top-level keys in JSON object structures) are
  scoped as `source.json meta.mapping.key - (meta.mapping.value meta.mapping.key | meta.sequence.list meta.mapping.key)`.
  Color scheme authors should add special highlighting to differentiate
  them from other keys.

- fix sublimehq#285
- address sublimehq#421 (thanks to @FichteFoll)
- address sublimehq#481 to remove incompatible regex patterns according
  to @wbond
- address sublimehq#757 to fix line comments for `JSONC` (thanks to
  @keith-hall)
- address sublimehq#2430 using sort-order (as requested by @deathaxe)
- address sublimehq#2711 with regards to `constant.language.null`
  vs. `constant.language.empty` (thanks to @FichteFoll)
- address sublimehq#2852 to fix scopes of curly braces & square
  brackets in `JSON` (thanks to @Thom1729)
- address sublimehq#3228 to fix `punctuation.separator` scopes,
  compare sublimehq#3270
- address sublimehq/sublime_text#3154 and add symbol tests

Co-authored-by: Ashwin Shenoy <Ultra-Instinct-05@users.noreply.github.com>
Co-authored-by: Jack Cherng <jfcherng@users.noreply.github.com>
Co-authored-by: Janos Wortmann <jwortmann@users.noreply.github.com>
Co-authored-by: Jon Skinner <jps@sublimetext.com>
Co-authored-by: FichteFoll <FichteFoll@users.noreply.github.com>
Co-authored-by: Keith Hall <keith-hall@users.noreply.github.com>
Co-authored-by: Michael B. Lyons <michaelblyons@users.noreply.github.com>
Co-authored-by: Rafał Chłodnicki <rchl@users.noreply.github.com>
Co-authored-by: Thomas Smith <Thom1729@users.noreply.github.com>
Co-authored-by: Will Bond <wbond@users.noreply.github.com>
Co-authored-by: deathaxe <deathaxe@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C: Syntax T: enhancement Improvement of existing language features
Projects
None yet
Development

No branches or pull requests

4 participants