Thoughts on HTMLish Language Support

[ematipico]

from rome/tools#4077

The HTMLish languages include HTML and HTML-based template languages like: Vue templates, Angular templates, Grale templates, etc.

The goal of this discussion is to share design constraints and ideas.

Objective

Enable code reuse for lints and formatting for the different HTML languages. For example, implement the logic for accessibility lints once and use it for all HTMLish languages.

The objective is not to optimize binary size but enable code reuse.

Design considerations

HTMLish languages are not additive

What I mean by additive is that a language dialect like TypeScript or JSX adds additional syntax to a language:

• If you use TypeScript syntax in a JavaScript file, it becomes TypeScript.
• If you use JSX syntax, it becomes a JSX file
• If you use JSX and TypeScript, it becomes a TSX file

The same applies to JSON, JSON5, and JSON6: Using any JSON6-only syntax makes your whole file a JSON6 file.

That's because the "extension" dialects are supersets of the language they extend:

• JavaScript ⊆ TypeScript
• JavaScript ⊆ JSX, TypeScript ⊆ TSX, JSX ⊆ TSX (with a very few exceptions)
• JSON ⊆ JSON5 ⊆ JSON6

While it's true that you get an Angular template if you use any angular syntax in HTML, it doesn't work if you start mixing template dialects: What format does a file use if you use an Angular expression inside a Vue template? Vular, Angulue?

We'll run into the same problem with JavaScript if we add support for another JavaScript dialect, e.g., Flow. You could even argue that it's a situation we have today with SvelteJs, except that SvelteJs doesn't extend the JavaScript syntax but changes the language semantics.

Naming

HTML element and attribute names are not case insensitive but are case sensitive in Angular (at least when using TypeScript), JSX, and VueJS templates --- and potentially others.

Furthermore, some template languages support multiple syntaxes to assign an attribute value:

• attribute syntax <a href="./link">body<a>
• binding syntax: Vue.js <a v-bind:href="link">body</a>

That means it is necessary for some languages to check for static attributes and bindings when testing if an attribute is set.

JSX

JSX is defined as part of the JavaScript language but otherwise fits the bill of an HTML template language. Meaning reusing code for HTMLish lints rule for JSX is desired but doing so ultimately means that either:

• JSX must be extracted from JavaScript into the HTMLish language (unclear how this would work considering the tight coupling to the JS parser)
• The solution we seek must work across different syntax languages

Design Ideas

Single HTMLish language

Create a unified HTMLish grammar that unifies all different HTMLish dialects.

AnyHtmlishAttribute = VueBinding| AngualarBinding | HtmlAttribute | ... 

This does not mean that all dialects use the same parser. Language could either share the parser implementation, or each comes with its own.

This approach does not allow for code reuse between JSX and HTMLish

Unified HTMLish syntax

The idea is to have language-specific grammars with a unifying language on top.

AnyHtmlishRoot = AngularRoot | VueRoot | HtmlRoot | ... 
AnyHtmlishAttribute = AnyAngularAttribute | AnyVueAttribute | ...

This approach is very compelling at first but has two shortcomings

• The union nodes cannot implement AstNode because it's unclear what to use for the L: Language type parameter (the nodes are from many different languages)
• Much of our infrastructure is typed over L: Language.

HTMLish traits

Each HTMLish language has its own grammar and traits abstract the common operations used by the linter that allow iterating elements, finding attributes, testing attribute values, etc. Lints would then either be generic over the HTMLish trait implementation or a trait with methods to perform some language-specific operations.

This trait system may be supplemented by an HTMLish semantic model that allows fast queries for specific elements/attributes.

It's not entirely clear to me how this would work on the formatting side, but I could envision that the formatting of an Element is a trait that can then be implemented by language-specific formatters.

[denbezrukov]

I've been following this discussion on designing a flexible and reusable system for HTMLish languages, and I'd like to share some thoughts that might contribute to our collective understanding and approach.

Given the complexity and diversity of HTMLish languages (including HTML, Vue, Angular, etc.), I believe that employing separate parsers for each language, coupled with a trait-based system for linters and formatters, presents a viable and robust solution. This approach allows for nuanced parsing that can accurately reflect the unique syntax and semantics of each language while providing a unified interface for tooling, thereby facilitating code reuse and extensibility.

Separate Parsers for Each Language

Creating dedicated parsers for each HTMLish language ensures that the parsing process is tailored to the specific needs and features of each language. This specificity is crucial for accurately capturing the nuances of each language, including unique attributes, directives, and template syntaxes. It also allows for performance optimizations specific to the parsing and analysis needs of each language, enhancing efficiency and effectiveness.

Trait-Based Linters/Formatters

A trait-based approach for linters and formatters can significantly streamline the development of these tools. By defining common operations, checks, and analyses in traits, we can apply these tools across different languages with minimal code duplication. This not only speeds up the development process but also ensures consistency in how different languages are linted and formatted, providing a more uniform developer experience.

Extending Codegen for Grammar Reuse

One of the challenges in this approach is the potential duplication of effort in defining grammar for similar constructs across languages. To address this, I propose extending our code generation system to allow for the sharing of grammar components among the different language parsers. This would involve:

• Identifying Common Grammar Elements: We start by identifying the elements that are common across HTMLish languages, such as basic HTML tags, common attributes, and template syntax elements that are shared among languages.
• Creating a Shared Grammar Library: These common elements would then be abstracted into a shared library that can be included in the grammar definitions of each language. This allows us to define these elements once and reuse them across different parsers.
• Allowing Language-Specific Extensions: Each language-specific grammar can extend the shared definitions with its unique elements and rules. This ensures that while we maximize code reuse, we also retain the flexibility to accurately capture the unique features of each language.

This approach not only fosters code reuse but also ensures that our parsers remain flexible and accurate, capable of adapting to the specific requirements of each HTMLish language.

Conclusion

By combining separate parsers with a trait-based approach for tooling and an extended code generation system for grammar reuse, we can achieve a balance that allows for accurate, language-specific parsing and efficient, reusable tooling. This strategy respects the unique aspects of each language while promoting a more unified and efficient development process for linters, formatters, and other analysis tools.

I'm eager to hear your thoughts on this approach and how we might refine it further to address our collective goals.

[ematipico]

I also think that a trait system is the best choice in the long run.

The initial set up will be quite difficult and awkward, because we will need to modify our code gen just for this syntax .

Maybe, instead of starting from the grammar, we should start from an initial implementation.

Perhaps we could start by implementing a trait for the attributes, and then trying to make it "specific" for HTML and another language.

What do you think about it?

[Conaclos]

Just wondering: are there so many differences between htmlish languages that justify separate grammar and CST? We could use a common CST and a parser mode to parse accurately language?

[ematipico]

As explained in the beginning:

The union nodes cannot implement AstNode because it's unclear what to use for the L: Language type parameter (the nodes are from many different languages)

Much of our infrastructure is typed over L: Language.

[Conaclos]

If we use a single grammar, we have a single language. I don't understand your comment @ematipico .

[ematipico]

If we use a single grammar, we have a single language. I don't understand your comment @ematipico .

We can't re-use logic and nodes. One of the objectives would be to add JSX to the HTML dialects.

For example, Astro and Vue allow JSX syntax in their files. Astro uses a dialect of JSX. Although, today JSX nodes are typed over the JavaScript Language, so it's unclear how we could re-use what we have inside new files.

[conartist6]

I went a rather different way on this one: language embedding all the way down. I see the TSX language as TSX syntax embedded in TS syntax embedded in JS syntax embedded in comment syntax embedded in whitespace.

[jpike88]

I believe this approach shouldn't require 'rust for everything' as an approach out the gate. Parsing angular source code is a non-trivial task, and projects like https://github.com/prettier/angular-estree-parser exist that take that problem out of the equation. Why not call that somehow to get your syntax tree, pass it back into rust and then apply the transformations?

Maybe use https://docs.rs/neon/latest/neon/index.html for that.

Then, over time, when time permit, replace the AST syntax parser with a rust implementation.

For angular:

Template code is either in .component.html, or in in the template property of the @component decorator. When that is detected, activate a dedicated parsing pipeline.

Make a call