TextMate Grammar scope specifications with multiple scopes is incorrectly handled

[msftrncs]

Prerequisites

• Put an X between the brackets on this line if you have done all of the following:
  • Reproduced the problem in Safe Mode: https://flight-manual.atom.io/hacking-atom/sections/debugging/#using-safe-mode
  • Followed all applicable steps in the debugging guide: https://flight-manual.atom.io/hacking-atom/sections/debugging/
  • Checked the FAQs on the message board for common solutions: https://discuss.atom.io/c/faq
  • Checked that your issue isn't already filed: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aatom
  • Checked that there is not already an Atom package that provides the described functionality: https://atom.io/packages

Description

If a TextMate grammar scope specifier (name or contentName) specifies multiple scopes, they are not correctly handled, and thus highlighting fails.

The above image shows a region which should have 6 scopes, but three of them are specified on a single specification, and so they appear as a single scope.

In addition to highlighting failing, automated tests using atom-grammar-test also fail.

The scope specifier in the grammar file is just the list of scopes separated with spaces.

I know VSCode-TextMate explicitly accepts this, as I have seen the code that makes sure to break up the specification list at the spaces, and so my grammar file works correctly in VS Code. I am certain I saw this in another textmate grammar file, and given enough time I will find it.

(VS Code, same grammar file)

I'll get a link posted to the grammar file in question, if its needed. These specific changes are not yet posted online.

(snippet of one repository item)

	interpolation:
		begin: "(\\$)(\\()"
		beginCaptures:
			"1":
				name: "keyword.other.substatement.powershell punctuation.definition.substatement.powershell punctuation.section.embedded.substatement.begin.powershell"
			"2":
				name: "punctuation.section.group.begin.powershell punctuation.section.embedded.substatement.begin.powershell"
		contentName: "interpolated.complex.source.powershell"
		end: "\\)"
		endCaptures:
			"0":
				name: "punctuation.section.group.end.powershell punctuation.section.embedded.substatement.end.powershell"
		name: "meta.embedded.substatement.powershell"
		patterns: [
			{
				include: "$self"
			}
		]

The purpose of these multiple scopes to that there are alternate ideas as to how the content should be scoped and themed, and this is the best to support them all. The last scope in the specification would have the highest priority if the theme has a selection for it.

Versions

Atom : 1.37.0
Electron: 2.0.18
Chrome : 61.0.3163.100
Node : 8.9.3

Windows 10 1903 18908

[Aerijo]

What does this do in actual TextMate?

[msftrncs]

I cannot speak for TextMate, as I am not a Mac user.

However I found a previous implementation, in https://github.com/davidrios/pug-tmbundle, which uses multiple scopes in a specification. The repository states it was tested in TextMate 2. I recently helped fix VS Code's handling of this particular grammar for items that Atom seemed to have no issue with. For reference to those, see microsoft/vscode-textmate#86 and microsoft/vscode-textmate#85.

VS Code added support for this approximately three years ago but I cannot find an issue or PR that documents why. This issue (microsoft/vscode#9436) seems to be what caused the change, but there are no links to the commit to confirm it, or any comments to explain what was wrong, so I am taking a guess looking at the code elements involved in the pug sample.

I'll continue to look for documentation or other references.

[msftrncs]

Another reference I found, though it does not confirm if the image showing TextMate's behavior is actually with this grammar, since the Atom image I am sure is NOT this grammar file, as I have tried this particular grammar file with Atom and I get almost no color at all, even though Atom does not have the issues that VS Code had.

davidrios/pug-tmbundle#26

However, Atom seems to be having some kind of trouble with the inline pug interpolated element. Both language-pug and this particular grammar fail to properly scope that only the first word (token) after a #[ is a tag. This is another issue I will work to properly report.

In another issue (microsoft/vscode#65983 (comment)), @neilsoult shows some sample pug in TextMate 2, but does not indicate what the grammar source for TextMate is. @neilsoult, was this the same pug-tmbundle that VS Code uses that you showed the example of in TextMate?

[neilsoult]

I'm really not sure, I downloaded Textmate specifically to compare it with VSCode, and I don't remember if I had to configure Textmate with any specific grammar to handle pug files. If I did, I would assume that I used the same pug-tmbundle, but I can't say for sure. Do you know how I can look at which grammar is being used by Textmate through the settings or configuration somewhere? It's still installed, I haven't touched it since.

[msftrncs]

Updated info, still not from TextMate, but the document that all of us grammar authors rely on, suggests numerous times that we should use multiple scopes on a given region:

Scope Naming – Sublime Text 3 Documentation

Take for instance, storage.type.function keyword.declaration.function. keyword is the preferred scope, so its listed last, but storage.type is acceptable if there is no theme for keyword.

Atom supports the reference, but it doesn't let themes apply because 'storage.type' will match, but the preferred 'keyword' will not because 'storage.type.function keyword.declaration.function' does not match 'keyword'.

[msftrncs]

I should probably also restate the original reason for posting this issue, was that the grammar test system fails when multiple scopes are specified. This could ultimately be a separate issue with the atom-grammar-test package.

[stale]

Thanks for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. Because the Atom team treats their issues as their backlog, stale issues are closed. If you would like this issue to remain open:

1. Verify that you can still reproduce the issue in the latest version of Atom
2. Comment that the issue is still reproducible and include:
   • What version of Atom you reproduced the issue on
   • What OS and version you reproduced the issue on
   • What steps you followed to reproduce the issue

Issues that are labeled as triaged will not be automatically marked as stale.

[jeff-hykin]

This should be kept open for an explanation since it there is important behavior happening here that I don't see documented.

I can explain what's going on in more detail but I'm a bit busy at the moment. So I'll just say here that Atom uses CSS scopes instead of TextMate scopes. CSS scopes are generally more powerful, but are different because they don't depend on ordering within a level.

The second issue here is the spaces. Spaces in the name: in Textmate are expected to be a list of nested scopes. It looks like Atom incorrectly handles that case and instead of nesting them, crams them all on the same level.