Add SQF lexer #761

BaerMitUmlaut · 2017-08-22T18:27:35Z

This PR adds a lexer with a demo and an example for the SQF scripting language.

This scripting language is used in Bohemia Interactive's Arma series for creating singleplayer campaigns, multiplayer scenarios and gameplay addons. Github has had syntax highlighting for SQF for some time now, but Gitlab (which quite a few communities use for private developments) was unfortunately still lacking it.

Don't be confused by the large list of command keywords. In SQF there are only commands and variables, no keywords or operators, with the exception of private and =. Even if or : (for switch case) are commands. Commands are defined by the engine and cannot be defined by a user. The whole grammar looks something like this:

block := ((statement;)* statement)?;?
statement := (assignment)? code
assignment := (private)? variable =
code := literal
      | variable
      | { block }
      | [ ((code,)* code)? ]
      | nular
      | unary code
      | code binary code

The code in spec/visual/samples/sqf was used to generate the list of commands.

gfx · 2017-09-21T04:07:24Z

lib/rouge/lexers/sqf.rb

+      end
+
+      # high priority for filename matches
+      def self.analyze_text(*)


analyze_text is no longer availabe on the maste branch.

See #763 for details

BaerMitUmlaut · 2017-09-21T08:55:22Z

I assume it should just always return false if there is no doctype or shebang declaration for the language?

gfx · 2017-09-21T09:17:48Z

@BaerMitUmlau

Just use the default implementation of detect?: https://github.com/jneen/rouge/blob/master/lib/rouge/lexer.rb#L445-L447 (i.e. false), if a lexer can't detect the type from a source.

BaerMitUmlaut · 2017-09-21T18:24:27Z

Should be good now.

gfx · 2017-09-22T06:42:14Z

lib/rouge/lexers/sqf.rb

+        # Identifiers (variables and functions)
+        rule %r"[a-zA-Z0-9_]+" do |m|
+          name = m[0].downcase
+          if self.class.wordoperators.include? name


Use declarative DSL like Ruby's lexer: https://github.com/jneen/rouge/blob/master/lib/rouge/lexers/ruby.rb#L203

I actually prefer it this way for keyword lists - in my experience it tends to perform better than large | regexes.

gfx · 2017-09-22T06:45:26Z

lib/rouge/lexers/sqf.rb

+
+      def self.commands
+        @commands = Set.new %w(
+          abs acctime acos action actionids actionkeys actionkeysimages


How do you make the list?

If it's programmatically generated, add the script as a comment (if it can be a one liner), or add rake task to tasks/.

It's generated using the spec/visual/samples/sqf script, with some manual filtering afterwards (because as mentioned in the OP, pretty much everything is a command). Since that is run in game, that's not an option.
Alternatively, there's a wiki page that could be scraped. I will have to check how much it deviates from this list (there might be some outdated commands in there).

Could use the same that I used for highlight.js. But it's much simpler since we don't need case sensitivity here.

https://github.com/dedmen/highlight.js_SQFgen

Basically parse
https://github.com/intercept/intercept/blob/master/src/client/headers/client/sqf_assignments.hpp
with regex \("([^"]*)"
That list get's regularly(programmatically) refreshed with all commands.

You'll have to add commands that are only available in specific game builds (https://github.com/dedmen/highlight.js_SQFgen/blob/master/additonalCommandList.txt)

and filter out things like
https://github.com/jneen/rouge/pull/761/files/5a9d5b12121b4c50c15deaf4183e0c42b59b6ba7#diff-21afe91a2d45af3ae84baa20d85ce523R13
and
https://github.com/jneen/rouge/pull/761/files/5a9d5b12121b4c50c15deaf4183e0c42b59b6ba7#diff-21afe91a2d45af3ae84baa20d85ce523R25

JonBons · 2018-03-29T05:27:39Z

Whats the status on this? I have SQF code on gitlab and it would be nice for support for this language to be added. Anything I can help with?

dedmen · 2018-05-07T14:37:06Z

Same here. I don't see anything stopping this from actually being completed. And we need it for ACE wiki and the SQF blog.

BaerMitUmlaut · 2018-05-07T14:41:14Z

I can use that intercept list then. Is it automatically updated?

dedmen · 2018-05-07T14:42:59Z

The files are still manually copy-pasted from Arma directory into the repository. But the list itself is generated by Intercept code.

dedmen · 2018-05-07T16:04:23Z

^ only the intercept list that is. The "AdditionalCommands" list that I linked above is manual.
But seeing as Arma is very close to EOL and with it SQF too I wouldn't worry about new diagnose build only commands that much, because that's essentially all that's in the additionalCommands list.

BaerMitUmlaut · 2018-05-21T20:48:19Z

@gfx I've added a rake task following dedmen's advice. I tried to imitate the existing rake tasks since I don't really know ruby. Please let me know if this looks good to you.

dedmen · 2018-05-22T12:30:15Z

lib/rouge/lexers/sqf.rb

+      end
+
+      def self.commands
+        load Pathname.new(__FILE__).dirname.join("sqf/commands.rb")


Think you forgot to push that file
https://travis-ci.org/jneen/rouge/jobs/381856670#L558

Nobody saw 🤫

The rake script produces duplicates. Meaning the commands list contains for example diag_log and params which are already in self.diag_commands and self.initializers
Does that create any problems?

Commands are checked last in the list so I guess it's not a problem then.

BaerMitUmlaut · 2018-08-10T20:02:31Z

@gfx @jneen This PR is open since almost a year now, all issues that I know of have been fixed and there are no merge conflicts. Would really like to see this merged

dblessing · 2018-08-10T20:17:54Z

Thanks @BaerMitUmlaut Sorry for the delay. There are a couple of very minor things that I want to change here but in the interest of getting this merged and not causing you further work and delay, I'm merging and I'll fix those very minor things in another PR.

Add SQF lexer

16d2fb6

gfx reviewed Sep 21, 2017

View reviewed changes

Remove deprecated analyze_text in favour of detect

5a9d5b1

gfx reviewed Sep 22, 2017

View reviewed changes

dedmen reviewed May 22, 2018

View reviewed changes

BaerMitUmlaut added 2 commits May 22, 2018 14:46

Add SQF rake task

d238b13

Add new hash command

6528ed9

BaerMitUmlaut force-pushed the feature.sqf branch from 779d15e to 6528ed9 Compare May 22, 2018 12:48

dblessing merged commit ee66b98 into rouge-ruby:master Aug 10, 2018

BaerMitUmlaut mentioned this pull request Oct 31, 2021

Replace CPP highlighting with SQF highlighting acemod/ACE3#8656

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add SQF lexer #761

Add SQF lexer #761

BaerMitUmlaut commented Aug 22, 2017

gfx Sep 21, 2017

BaerMitUmlaut commented Sep 21, 2017

gfx commented Sep 21, 2017 •

edited

BaerMitUmlaut commented Sep 21, 2017

gfx Sep 22, 2017

jneen Sep 22, 2017

gfx Sep 22, 2017

BaerMitUmlaut Sep 22, 2017

dedmen May 7, 2018 •

edited

JonBons commented Mar 29, 2018

dedmen commented May 7, 2018

BaerMitUmlaut commented May 7, 2018

dedmen commented May 7, 2018 •

edited

dedmen commented May 7, 2018 •

edited

BaerMitUmlaut commented May 21, 2018

dedmen May 22, 2018

BaerMitUmlaut May 22, 2018

BaerMitUmlaut May 22, 2018

dedmen May 22, 2018 •

edited

BaerMitUmlaut commented Aug 10, 2018

dblessing commented Aug 10, 2018

Add SQF lexer #761

Add SQF lexer #761

Conversation

BaerMitUmlaut commented Aug 22, 2017

Choose a reason for hiding this comment

BaerMitUmlaut commented Sep 21, 2017

gfx commented Sep 21, 2017 • edited

BaerMitUmlaut commented Sep 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dedmen May 7, 2018 • edited

Choose a reason for hiding this comment

JonBons commented Mar 29, 2018

dedmen commented May 7, 2018

BaerMitUmlaut commented May 7, 2018

dedmen commented May 7, 2018 • edited

dedmen commented May 7, 2018 • edited

BaerMitUmlaut commented May 21, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dedmen May 22, 2018 • edited

Choose a reason for hiding this comment

BaerMitUmlaut commented Aug 10, 2018

dblessing commented Aug 10, 2018

gfx commented Sep 21, 2017 •

edited

dedmen May 7, 2018 •

edited

dedmen commented May 7, 2018 •

edited

dedmen commented May 7, 2018 •

edited

dedmen May 22, 2018 •

edited