Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Expose vimscript parser / interpreter #170

Closed
dbarnett opened this Issue · 26 comments
@dbarnett

Up to now, no vimscript parser or interpreter existed outside of the vim editor itself. As a result, there has been no syntax checker for vimscript, no such thing as vimscript "unit tests" (i.e., for utility functions), and some other neat projects I'd like to build on top of a such vimscript tools are just not possible without hacking up an iffy re-implementation of vimscript.

With neovim's proposed modular architecture, it seems like it's finally possible to provide access to an AST for parsed vimscript and an interpreter for sending commands to a headless vim environment. Is that in the plan?


Want to back this issue? Place a bounty on it! We accept bounties via Bountysource.

@dbarnett

I'm pretty sure a parser would have to be somewhat stateful since I think the parsing for command arguments may depend on the command definition (e.g., some commands can be followed by comments on the same line, others can't). Maybe that doesn't affect custom commands or anything else that changes at runtime, though, in which case there would be a lot less complexity to the parser.

@rjw57 rjw57 added the enhancement label
@ZyX-I
Collaborator

You are absolutely right about depending on command definition. If I were writing one I would compile

Foo | let a=1

to something like {'type': 'UserCommand', 'name': 'Foo', 'text': '| let a=1} and leave further parsing to the evaluator (which will in turn call parser once again, but it now will call parser with specific state).

Foo | Bar | Baz

will use parser three times if all but last commands are defined with -bar: to parse the function this text is located in, to parse | Bar | Baz when it will come to evaluation of Foo, to parse | Baz when it will come to evaluation of Bar. And that on each function call.

There are problems here:

  1. I am almost 100% sure that async support will be added to VimL because it is required for e.g. VimResized autocommand. So you cannot rely on order of execution.
  2. I would really like to see how you implement support for e.g. calling this function that defines command and then calling newly defined command in the parser.
@tarruda
Owner

Right now I can't spend much energy on this because there are more important problems to be solved.

With that said, I already did some thinking about how to implement the VimL->Lua compiler. Since we are going to use lua(more specifically its higher-level 'dialect' moonscript I thought about using lpeg, a parsing library for PEG grammars.

@leafo used this library to implement moonscript and I think it might be good enough for vimscript, the only downside is that PEG parsers are somewhat slow. @ZyX-I, you have extreme vimL knowledge, so if anyone has a chance of quickly implementing a vimL parser on top of lpeg it's you.

If a PEG solution proves to be slow, I have some experience with LR parsing(I wrote a parse table generator a few months ago) so I can probably port it to moonscript.

@leafo , any thoughts?

@ZyX-I
Collaborator

@tarruda I have much better chances to implement it on top of vanilla lua (with a possible future C rewrite in mind if lua solution proves slow): I do not have too good knowledge of lua, but lpeg is something I have not ever seen at all.

@tarruda
Owner

@ZyX-I have you used a parser generator(antlr, bison, etc)? lpeg is something similar, but implemented as a library instead of a compile-time tool(that isn't a problem for highly dynamic languages such as lua)

Lua is a very simple language, after a few hours of skimming through the documentation(especially the C API which is much cleaner than the API of any other language out there) you will have a good understanding of it, especially if you have python experience.

I doubt lua will be slow. Lua reference implementation is probably faster than python(not sure though) and Luajit runs circles around it. Apparently it's even faster than google's v8 as shown by this benchmark

Even if a parser implemented with lpeg is slow, we can probably perform a lot of optimization with a hand-written parser before even thinking about reimplementing it in C.

Perhaps you can join with @Shougo, he seems to have a lot of interest in lua.

@leafo

I think using a parser generator would be very helpful because it enforces how the grammar is written opposed to something hand written (vim script's current parser?). I don't know much about how vim script's grammar is defined but PEGs are significantly more flexible state machine based alternatives (at the cost of speed), so it would potentially be easier to write a grammar in one.

Another interesting idea would be to compile vim script to Lua instead of writing an interpreter. The huge advantage would be when you use LuaJIT and you get a free JIT runtime. The downside of this would be additional latency when loading scripts because now you have to generate Lua after parsing to AST. It might also be tricky to map vim script semantics to Lua ones.

@ZyX-I
Collaborator
@aktau aktau referenced this issue
Closed

lua or moonscript #207

@ZyX-I
Collaborator

There is another idea which came to mind: why write a parser if I can take code from eval.c and transform it into the one? Ex commands are trickier, but most likely I can borrow at least some code from them.

@ZyX-I
Collaborator

Since I have some free time and no reply from @tarruda I have started rewriting eval.c as a parser. This approach has already proved itself: I had no idea that vim allows 1.2.3.4 (it is string concatenation).

@jszakmeister
Collaborator

@ZyX-I you rock!

@ZyX-I ZyX-I referenced this issue from a commit in ZyX-I/neovim
@ZyX-I ZyX-I First working version of expression parser
WARNING: part of expr.c after //FIXME!!! must be deleted after I figure out how 
         to properly do tests.

TODO: tests, documentation, command-line parser (expr.c holds only expression 
      parser)

Ref #170
b95dec9
@simendsjo

@zyx-I, add a ticket so people more easily see your working on that file.

@tarruda
Owner

@ZyX-I great initiative, thank you!

@ZyX-I ZyX-I referenced this issue
Open

[WIP] VimL to lua translator #243

79 of 132 tasks complete
@Shougo

Perhaps you can join with @Shougo, he seems to have a lot of interest in lua.

OK. I know the Vim script parser. It may be useful for this.

vim-jp/issues#340

And, this is Lua translator idea in Japanese.

@dbarnett

@dbarnett, this may be somewhat off topic, but there is a syntax checker …

@myint: Oh, you're right, I hadn't noticed the syntastic bug I linked to was closed without comment a while back. There is apparently a vimscript linter available now, but it looks like rather a lot of code (including a javascript interpreter implemented in vimscript?!), so still it would be nice to have a native vimscript parser.

@myint

@dbarnett, I'm not sure, but I think it contains parsers implemented in multiple languages (Python, Vim, etc.).

@aktau aktau referenced this issue from a commit in aktau/neovim
@ZyX-I ZyX-I First working version of expression parser
WARNING: part of expr.c after //FIXME!!! must be deleted after I figure out how 
         to properly do tests.

TODO: tests, documentation, command-line parser (expr.c holds only expression 
      parser)

Ref #170
1ac0a98
@aktau aktau referenced this issue from a commit in aktau/neovim
@ZyX-I ZyX-I First working version of expression parser
WARNING: part of expr.c after //FIXME!!! must be deleted after I figure out how 
         to properly do tests.

TODO: tests, documentation, command-line parser (expr.c holds only expression 
      parser)

Ref #170
d9775fa
@luke-gru

Hi everyone, I don't know how helpful this is but I've been working on a language called riml that compiles to Vimscript. It's not 100% compatible with Vimscript (does not support most ex commands) and it adds features like interpolation and classes.

The parser is generated by racc (ruby's version of yacc/bison). Unfortunately the lexer is handwritten right now instead of being generated.

Also, there are lots of tests that test weird Vimscript edge cases I've run into. Right now these tests are mixed in with the tests for the added features, but it might be helpful if I isolate them for reuse.

What are your thoughts?

@tarruda
Owner

@luke-gru that seems interesting but what we'll do is the contrary: compile vimscript to lua

@luke-gru

Yes I know, but I was wondering if the BNF grammar would be of any use, as the language is an extension of Vimscript, and most rules are there for parsing Vimscript.

@justinmk
Owner

I would think the unit tests would be valuable, if ported to moonscript.

@tarruda
Owner

@luke-gru that is another story, if your language already parses a subset of vimscript then yes, its will be very useful. Thanks for sharing it here!

@tarruda
Owner

@luke-gru Thanks again, you already saved us a lot of work.

About the hand-written lexer, I was expecting having to write one to deal with vimL edge cases(it also helps simplifying the grammar), so I can certainly learn a lot from your code.

@luke-gru

No problem, let me know if anything is unclear or if you have general questions about the grammar that I might be able to help with.

Also, in compiling Vimscript to Lua (keep in mind I don't know Lua well at all), you will have to type check variables somehow because of the way Vimscript deals with the '.' (concat) operator.

For example:

let s = 'string1'
let s2 = 'string2'
echo s.s 
" => 'string1string2'

let d = { 'key' : 'val' }
echo d.key
" => 'val'

This is just an example of the trickiness that is going to be involved in the Vimscript => Lua compiler.

@aktau
Collaborator

@luke-gru Very good points! I'm sure we can use your considerable knowledge!

This is the global issue: #243

The point you just raised about concat vs. subscript already has a running issue: #240

Another ambiguity: #251

We hope you can contribute some expertise ;)

@luke-gru

@aktau Thanks for pointing me to the proper issues! I did not know about #251, but it makes perfect sense. I'll post over in #243 for now, and I'll definitely find time to document some of the troubles and edge cases I found.

@justinmk justinmk added this to the first release milestone
@ZyX-I ZyX-I referenced this issue from a commit in ZyX-I/neovim
@ZyX-I ZyX-I First working version of expression parser
WARNING: part of expr.c after //FIXME!!! must be deleted after I figure out how
         to properly do tests.

TODO: tests, documentation, command-line parser (expr.c holds only expression
      parser)

Ref #170
58beaee
@Grimy Grimy referenced this issue from a commit in Grimy/neovim
@Valloric Valloric Adding the any_of.hpp file to BoostParts
Fixes #171, fixes #170
5d97e70
@elmart
Collaborator

Closing, as it doesn't seem relevant anymore.

@elmart elmart closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.