Fix formatting and syntax highlighting

[mark-wiemer]

There are currently many issues with formatting, and a few with syntax highlighting. The AHK documentation doesn't have any syntax highlighting issues, and presumably AHK has a much more consistent parser somewhere that correctly determines how the code is structured (what text is an "if" condition, what text is a command, what text is a string literal, etc.)

The parser should copy the behavior of the AHK's own parser. It will probably generate an AST like any standard compiler.

As of opening this bug, I've never even read C++ (which AHK is written in) but I have studied compilers.

• Get started with C++ (understand basic syntax, know its quirks from other languages)
• Understand AHK implementation at a high level I don't need to know how AHK works to know how to parse it 😄
• Understand working syntax definition (Previously: "Find the parsing logic within AHK itself")
• Copy the parsing logic into TypeScript for this project
• Integrate parsing logic with formatter for fully-featured formatter

Sub-issues:

• Fix the syntax highlighter #128
• Fix the formatter #129

References:

• TextMate regular expressions: VS Code uses TextMate to parse AHK code into tokens. This parsing logic can be shared with Prettier to create a plugin
• Prettier plugins: Prettier is an excellent code formatter, providing an AHK plugin is likely the easiest way forward for the formatting side

[anonymous1184]

I spent a few reading all of your recent comments, I'm glad you are back in the game and certainly very grateful. Looking forward to enjoy whatever comes out of this new effort.

However when I see this issue I can only think that you are going to just improve over the existing rather than go from scratch. With vscode-lsp you can do wonders... among the most useful things you can do is: better code formatting, improved symbol detection (which leads to simplified refactoring) and above all semantic highlighting (for me this one alone does it).

Given the beta status of the v2 I don't do anything on it, but seems is gaining popularity and I actually enjoyed the improvements, so I know as soon as it hits a stable release I'll ditch everything to go after the shinny new thing... and I guess I'm not the only one, many people will do the same.

The AutoHotkey2 Language Support extension by thqby is already built like this and is very compelling. Most likely the research on the LSP/LSIF will be a better investment than go trough AHK's Syntax Tree and C++ itself.

And if you go the route you have planned I started (never finished) a rewrite of the syntax highlight as there were many issues, if you keep things the way they are I'll be sure to spend some time finishing it. However I never touched a TextMate syntax files, so most of what I did was just intuition, if you have some some resources I'll be sure to put them to use.

Thanks! for past and new efforts, all the best.

[mark-wiemer]

Hey @anonymous1184, don't worry, I definitely plan to take advantage of existing frameworks whenever possible, including vscode-lsp. Thanks a lot for the link. Honestly, I've never built a standard VS Code extension from the ground up, so I'm going to start there and then move on to vscode-lsp.

I've also completed C++ essential training which has helped me understand the basics of C++ (turns out it's just C with classes, and I've written plenty of C!). Next step is to follow some of the basic VS Code extension tutorials :)

[mark-wiemer]

I've started a new repo, vscode-helloworld, if anyone is curious to watch me develop a VS Code extension from the ground up

[mark-wiemer]

This is still very much in the research phase. We're currently entering the formatting step in the standard way as documented in Format Source Code in an Editor - VS Code, but we're definitely not following the "smallest possible" rule:

You should always return the smallest possible text edits that result in the source code being formatted. This is crucial to ensure that markers such as diagnostic results are adjusted correctly and are not lost.

Oh well. Hopefully AST work here will mitigate the issue.

Next step is probably looking to existing AHK formatters to see how they're implemented

[mark-wiemer]

Writing a Prettier plugin could be a great option

[mark-wiemer]

Looks like the syntax definition was part of @stef-levesque's initial commit to vscode-autohotkey. Wild. @cweijan changed it to JSON later, and that's the version I'm using now. This seems to be a TextMate grammar, as doc'd in Syntax Highlight Guide - VS Code. So... I already have one? I just need to translate this into a parser function for a Prettier plugin and I'm good to go? Is it really that easy?

[mark-wiemer]

See also Official syntax definition anywhere? - AHK Forums

[mark-wiemer]

Reading list:

• Writing a TextMate Grammar: Some Lessons Learned

[anonymous1184]

Thanks for the info.

The the JSON has the same as the same data as the XML, just a different presentation (less verbose) but in the end the regular expressions work the same.

ahk.tmLanguage:37
ahk.tmLanguage.json:22

I just need to translate this into a parser function for a Prettier plugin and I'm good to go? Is it really that easy?

Well is not like mad hard but cumbersome, as Lexikos said, v1 presents many hidden complexities. From the top of my head: commands normally take one line or if split at the parameters/arguments more lines with an extra level of indentation if they are split, however the IfMsgBox has body and can have a whole block:

        MsgBox 0x40, Test, Hello World!

        MsgBox 0x40
                , Test
                , Hello World

        MsgBox 0x42, Question, If foo`, bar?
        IfMsgBox No
                ExitApp ; This must be indented
        IfMsgBox Yes  ; Block below, indented
        {
                OutputDebug Foo
                OutputDebug Bar
        }

Things like that might make it less easy but not impossible.

Next week I'll dust the old project where I started the rewrite of the highlighter, I'll keep you posted.

[mark-wiemer]

Issues will be tracked in specific issues and labels, not with an epic issue like this one.