CI for syntax tests using syntect

[FichteFoll]

Add CI (using Travis CI) for syntax tests. There is UnitTesting, which also supports running syntax tests (which I never tested), but to me it seems like using syntect is better. I don't see myself writing tests for the Python plugins and their integration into ST in the near future.

I'll document the current progress and obstacles in this issue.

Status

Blocked by: trishume/syntect#46

Not mature enough yet (quite a few false-positives) but could be set up as a non-failing command for travis.

[FichteFoll]

Current .travis.yml (untested)

language: rust

rust:
  - stable

addons:
  apt:
    packages:
      - python3 # ST3 uses py3.3, but we're only running flake8 anyway

install:
  - pip install flake8==3.3.0
  - cargo install syntect --git https://github.com/trishume/syntect --example syntest
  - export PATH="$HOME/.cargo/bin:$PATH"

script:
  - python3 -m flake8 -v
  # - TODO syntest ...

We can use the apt addon to install Python 3 into the container-based architecture, but not Rust, so our main "language" will be switching to rust.
Other resources:

• https://docs.travis-ci.com/user/trusty-ci-environment
• https://docs.travis-ci.com/user/installing-dependencies/

The syntest example is not included in the latest release on crates.io (1.2.0), which is why we point it to the git repo on Github and build from master.

---

Running syntest

syntest accepts two parameters, one for the syntax test files and one for syntax files. I was able to run it with syntest Packages/PackageDev Packages from ~/.config/sublime-text-3, which seems to be The Way To Go™ for now. I'm not exactly happy with it, but we'll manage.

I'm thinking of setting up symlinks in a CI-specific sub-directory that will establish the directory structure that syntest expects according to our paths in the syntax test files.

Additionally, we depend on the XML/XML.sublime-syntax and JavaScript/JSON.sublime-syntax syntaxes. We might have to pull the sublimehq/Packages repo (master or release tag) into the testing directory structure mentioned above.

That aside, syntest currently fails because we run into what will be fixed by trishume/syntect#46.
I have decided to hold this off until a later time (when that issue is fixed?).

[FichteFoll]

Results of my test run:

fichte@pinaceae> syntest Packages/PackageDev Packages                                                          ~/.config/sublime-text-3
loading syntax definitions from Packages
Testing file Packages/PackageDev/Package/Sublime Text Syntax Definition/syntax_test_onig-regexp
The test file references syntax definition file: Packages/PackageDev/Package/Sublime Text Syntax Definition/Oniguruma RegExp.sublime-syntax
Ok(Success(1122))
Testing file Packages/PackageDev/Package/Sublime Text Syntax Definition/syntax_test_sublime-syntax.yaml
The test file references syntax definition file: Packages/PackageDev/Package/Sublime Text Syntax Definition/Sublime Text Syntax Definition.sublime-syntax
Ok(Success(995))
Testing file Packages/PackageDev/Package/Sublime Text Keymap/syntax_test_keymap.json
The test file references syntax definition file: Packages/PackageDev/Package/Sublime Text Keymap/Sublime Keymap.sublime-syntax
  Assertion selector "comment.line.double-slash - meta.keybinding.collection.sublime-keymap - meta.sequence.json" from line 6 failed against line 5, column range 3-4 (with text "c") has scope [<source.json.sublime.keymap>, <invalid.illegal.expected-sequence.sublime-keymap>]
  Assertion selector "comment.line.double-slash - meta.keybinding.collection.sublime-keymap - meta.sequence.json" from line 6 failed against line 5, column range 4-5 (with text "o") has scope [<source.json.sublime.keymap>, <invalid.illegal.expected-sequence.sublime-keymap>]
  Assertion selector "comment.line.double-slash - meta.keybinding.collection.sublime-keymap - meta.sequence.json" from line 6 failed against line 5, column range 5-6 (with text "m") has scope [<source.json.sublime.keymap>, <invalid.illegal.expected-sequence.sublime-keymap>]
  Assertion selector "comment.line.double-slash - meta.keybinding.collection.sublime-keymap - meta.sequence.json" from line 6 failed against line 5, column range 6-7 (with text "m") has scope [<source.json.sublime.keymap>, <invalid.illegal.expected-sequence.sublime-keymap>]
  Assertion selector "comment.line.double-slash - meta.keybinding.collection.sublime-keymap - meta.sequence.json" from line 6 failed against line 5, column range 7-8 (with text "e") has scope [<source.json.sublime.keymap>, <invalid.illegal.expected-sequence.sublime-keymap>]
  Assertion selector "comment.line.double-slash - meta.keybinding.collection.sublime-keymap - meta.sequence.json" from line 6 failed against line 5, column range 8-9 (with text "n") has scope [<source.json.sublime.keymap>, <invalid.illegal.expected-sequence.sublime-keymap>]
  Assertion selector "comment.line.double-slash - meta.keybinding.collection.sublime-keymap - meta.sequence.json" from line 6 failed against line 5, column range 9-10 (with text "t") has scope [<source.json.sublime.keymap>, <invalid.illegal.expected-sequence.sublime-keymap>]
  Assertion selector "comment.line.double-slash - meta.keybinding.collection.sublime-keymap - meta.sequence.json" from line 6 failed against line 5, column range 10-11 (with text "\n") has scope [<source.json.sublime.keymap>]
  Assertion selector "string.quoted.double.json" from line 159 failed against line 155, column range 17-21 (with text "test") has scope [<source.json.sublime.keymap>, <meta.sequence.json>, <meta.keybinding.collection.sublime-keymap>, <meta.mapping.json>, <meta.mapping.value.json>, <meta.sequence.json>]
  Assertion selector "- invalid" from line 162 failed against line 155, column range 24-25 (with text " ") has scope [<source.json.sublime.keymap>, <meta.sequence.json>, <meta.keybinding.collection.sublime-keymap>, <meta.mapping.json>, <invalid.illegal.expected-comma.inside-mapping.json>]
  Assertion selector "string.quoted.double.json - support.function - meta.key - keyword" from line 199 failed against line 195, column range 81-86 (with text "\"cpp\"") has scope [<source.json.sublime.keymap>, <meta.sequence.json>, <meta.keybinding.collection.sublime-keymap>, <meta.mapping.json>, <meta.mapping.value.json>, <meta.mapping.json>, <meta.mapping.value.json>, <meta.sequence.json>]
  Assertion selector "punctuation.separator.sequence.json" from line 200 failed against line 195, column range 86-87 (with text ",") has scope [<source.json.sublime.keymap>, <meta.sequence.json>, <meta.keybinding.collection.sublime-keymap>, <meta.mapping.json>, <meta.mapping.value.json>, <meta.mapping.json>, <meta.mapping.value.json>, <meta.sequence.json>]
  Assertion selector "punctuation.separator.mapping.pair.json" from line 208 failed against line 204, column range 84-85 (with text ",") has scope [<source.json.sublime.keymap>, <meta.sequence.json>, <meta.keybinding.collection.sublime-keymap>, <meta.mapping.json>, <meta.mapping.value.json>, <meta.mapping.json>, <invalid.illegal.expected-key.json>]
  Assertion selector "constant.language.json" from line 210 failed against line 204, column range 95-96 (with text "f") has scope [<source.json.sublime.keymap>, <meta.sequence.json>, <meta.keybinding.collection.sublime-keymap>, <meta.mapping.json>, <meta.mapping.value.json>, <meta.mapping.json>, <meta.mapping.value.json>, <invalid.illegal.expected-comma.inside-mapping.json>]
  Assertion selector "constant.language.json" from line 210 failed against line 204, column range 96-97 (with text "a") has scope [<source.json.sublime.keymap>, <meta.sequence.json>, <meta.keybinding.collection.sublime-keymap>, <meta.mapping.json>, <meta.mapping.value.json>, <meta.mapping.json>, <meta.mapping.value.json>, <invalid.illegal.expected-comma.inside-mapping.json>]
  Assertion selector "constant.language.json" from line 210 failed against line 204, column range 97-98 (with text "l") has scope [<source.json.sublime.keymap>, <meta.sequence.json>, <meta.keybinding.collection.sublime-keymap>, <meta.mapping.json>, <meta.mapping.value.json>, <meta.mapping.json>, <meta.mapping.value.json>, <invalid.illegal.expected-comma.inside-mapping.json>]
  Assertion selector "constant.language.json" from line 210 failed against line 204, column range 98-99 (with text "s") has scope [<source.json.sublime.keymap>, <meta.sequence.json>, <meta.keybinding.collection.sublime-keymap>, <meta.mapping.json>, <meta.mapping.value.json>, <meta.mapping.json>, <meta.mapping.value.json>, <invalid.illegal.expected-comma.inside-mapping.json>]
  Assertion selector "constant.language.json" from line 210 failed against line 204, column range 99-100 (with text "e") has scope [<source.json.sublime.keymap>, <meta.sequence.json>, <meta.keybinding.collection.sublime-keymap>, <meta.mapping.json>, <meta.mapping.value.json>, <meta.mapping.json>, <meta.mapping.value.json>, <invalid.illegal.expected-comma.inside-mapping.json>]
thread 'main' panicked at 'Can only call resolve on linked references: ByScope { scope: <source.escapedregexp>, sub_context: None }', /home/fichte/.cargo/git/checkouts/syntect-e33b39f181e4f0f4/89a0780/src/parsing/syntax_definition.rs:185
note: Run with `RUST_BACKTRACE=1` for a backtrace.

TL;DR

• Oniguruma and sublime-syntax have no failures (fix for oniguruma not yet pushed)
• sublime-keymap has a couple false-positives that stem from wrong scopes on syntect's part, that the tests correctly do not validate. Note that syncat seems to do a better overall job, for some reason.
• sublime-keymap references a tmLanguage scope with scope:source.escapedregexp, upon which syntect panics

[FichteFoll]

Tried this again with syntect 1.7.1, but there are lots of failures (and the panic explained earlier).

• syntax_test_mousemap.json Ok(FailedAssertions(356, 387))
• syntax_test_completions.json Ok(FailedAssertions(201, 533))
• syntax_test_settings.json Ok(FailedAssertions(81, 199))
• syntax_test_keymap.json thread 'main' panicked at 'Can only call resolve on linked references: ByScope { scope: <source.escapedregexp>, sub_context: None }', /home/fichte/.cargo/git/checkouts/syntect-e33b39f181e4f0f4/c8b4775/src/parsing/syntax_definition.rs:191
• ...

The panic is resolvable, becase we need to redo that syntax eventually, but I'm at loss on why the other tests fail.

[FichteFoll]

Since 389d978 we use Sublime Text's own syntax testing binary.