Expose vimscript parser / interpreter #170

Closed
dbarnett opened this Issue Feb 26, 2014 · 26 comments

Comments

Projects
None yet
Contributor

dbarnett commented Feb 26, 2014

Up to now, no vimscript parser or interpreter existed outside of the vim editor itself. As a result, there has been no syntax checker for vimscript, no such thing as vimscript "unit tests" (i.e., for utility functions), and some other neat projects I'd like to build on top of a such vimscript tools are just not possible without hacking up an iffy re-implementation of vimscript.

With neovim's proposed modular architecture, it seems like it's finally possible to provide access to an AST for parsed vimscript and an interpreter for sending commands to a headless vim environment. Is that in the plan?


Want to back this issue? Place a bounty on it! We accept bounties via Bountysource.

Contributor

dbarnett commented Feb 26, 2014

I'm pretty sure a parser would have to be somewhat stateful since I think the parsing for command arguments may depend on the command definition (e.g., some commands can be followed by comments on the same line, others can't). Maybe that doesn't affect custom commands or anything else that changes at runtime, though, in which case there would be a lot less complexity to the parser.

@rjw57 rjw57 added the enhancement label Feb 26, 2014

Contributor

ZyX-I commented Feb 26, 2014

You are absolutely right about depending on command definition. If I were writing one I would compile

Foo | let a=1

to something like {'type': 'UserCommand', 'name': 'Foo', 'text': '| let a=1} and leave further parsing to the evaluator (which will in turn call parser once again, but it now will call parser with specific state).

Foo | Bar | Baz

will use parser three times if all but last commands are defined with -bar: to parse the function this text is located in, to parse | Bar | Baz when it will come to evaluation of Foo, to parse | Baz when it will come to evaluation of Bar. And that on each function call.

There are problems here:

  1. I am almost 100% sure that async support will be added to VimL because it is required for e.g. VimResized autocommand. So you cannot rely on order of execution.
  2. I would really like to see how you implement support for e.g. calling this function that defines command and then calling newly defined command in the parser.
Owner

tarruda commented Feb 27, 2014

Right now I can't spend much energy on this because there are more important problems to be solved.

With that said, I already did some thinking about how to implement the VimL->Lua compiler. Since we are going to use lua(more specifically its higher-level 'dialect' moonscript I thought about using lpeg, a parsing library for PEG grammars.

@leafo used this library to implement moonscript and I think it might be good enough for vimscript, the only downside is that PEG parsers are somewhat slow. @ZyX-I, you have extreme vimL knowledge, so if anyone has a chance of quickly implementing a vimL parser on top of lpeg it's you.

If a PEG solution proves to be slow, I have some experience with LR parsing(I wrote a parse table generator a few months ago) so I can probably port it to moonscript.

@leafo , any thoughts?

Contributor

ZyX-I commented Feb 27, 2014

@tarruda I have much better chances to implement it on top of vanilla lua (with a possible future C rewrite in mind if lua solution proves slow): I do not have too good knowledge of lua, but lpeg is something I have not ever seen at all.

Owner

tarruda commented Feb 27, 2014

@ZyX-I have you used a parser generator(antlr, bison, etc)? lpeg is something similar, but implemented as a library instead of a compile-time tool(that isn't a problem for highly dynamic languages such as lua)

Lua is a very simple language, after a few hours of skimming through the documentation(especially the C API which is much cleaner than the API of any other language out there) you will have a good understanding of it, especially if you have python experience.

I doubt lua will be slow. Lua reference implementation is probably faster than python(not sure though) and Luajit runs circles around it. Apparently it's even faster than google's v8 as shown by this benchmark

Even if a parser implemented with lpeg is slow, we can probably perform a lot of optimization with a hand-written parser before even thinking about reimplementing it in C.

Perhaps you can join with @Shougo, he seems to have a lot of interest in lua.

leafo commented Feb 27, 2014

I think using a parser generator would be very helpful because it enforces how the grammar is written opposed to something hand written (vim script's current parser?). I don't know much about how vim script's grammar is defined but PEGs are significantly more flexible state machine based alternatives (at the cost of speed), so it would potentially be easier to write a grammar in one.

Another interesting idea would be to compile vim script to Lua instead of writing an interpreter. The huge advantage would be when you use LuaJIT and you get a free JIT runtime. The downside of this would be additional latency when loading scripts because now you have to generate Lua after parsing to AST. It might also be tricky to map vim script semantics to Lua ones.

Contributor

ZyX-I commented Feb 27, 2014

27.02.14, 21:18, "leaf" notifications@github.com":

I think using a parser generator would be very helpful because it enforces how the grammar is written opposed to something hand written (vim script's current parser?). I don't know much about how vim script's grammar is defined but PEGs are significantly more flexible state machine based alternatives (at the cost of speed), so it would potentially be easier to write a grammar in one.
Another interesting idea would be to compile vim script to Lua instead of writing an interpreter. The huge advantage would be when you use LuaJIT and you get a free JIT runtime. The downside of this would be additional latency when loading scripts because now you have to generate Lua after parsing to AST. It might also be tricky to map vim script semantics to Lua ones.

Translating VimL to lua is precisely the idea that was being discussed from the time I joined (and it was the one expressed in funding campaign). In any case this requires parser.


Reply to this email directly or view it on GitHub.

@aktau aktau referenced this issue Feb 27, 2014

Closed

lua or moonscript #207

Contributor

ZyX-I commented Feb 28, 2014

There is another idea which came to mind: why write a parser if I can take code from eval.c and transform it into the one? Ex commands are trickier, but most likely I can borrow at least some code from them.

Contributor

ZyX-I commented Mar 1, 2014

Since I have some free time and no reply from @tarruda I have started rewriting eval.c as a parser. This approach has already proved itself: I had no idea that vim allows 1.2.3.4 (it is string concatenation).

Member

jszakmeister commented Mar 1, 2014

@ZyX-I you rock!

ZyX-I added a commit to ZyX-I/neovim that referenced this issue Mar 1, 2014

First working version of expression parser
WARNING: part of expr.c after //FIXME!!! must be deleted after I figure out how 
         to properly do tests.

TODO: tests, documentation, command-line parser (expr.c holds only expression 
      parser)

Ref #170
Contributor

simendsjo commented Mar 1, 2014

@ZyX-I, add a ticket so people more easily see your working on that file.

Owner

tarruda commented Mar 1, 2014

@ZyX-I great initiative, thank you!

@ZyX-I ZyX-I referenced this issue Mar 1, 2014

Open

[WIP] VimL to lua translator #243

78 of 134 tasks complete
Contributor

Shougo commented Mar 2, 2014

Perhaps you can join with @Shougo, he seems to have a lot of interest in lua.

OK. I know the Vim script parser. It may be useful for this.

vim-jp/issues#340

And, this is Lua translator idea in Japanese.

Contributor

dbarnett commented Mar 3, 2014

@dbarnett, this may be somewhat off topic, but there is a syntax checker …

@myint: Oh, you're right, I hadn't noticed the syntastic bug I linked to was closed without comment a while back. There is apparently a vimscript linter available now, but it looks like rather a lot of code (including a javascript interpreter implemented in vimscript?!), so still it would be nice to have a native vimscript parser.

Contributor

myint commented Mar 3, 2014

@dbarnett, I'm not sure, but I think it contains parsers implemented in multiple languages (Python, Vim, etc.).

aktau added a commit to aktau/neovim that referenced this issue Mar 4, 2014

First working version of expression parser
WARNING: part of expr.c after //FIXME!!! must be deleted after I figure out how 
         to properly do tests.

TODO: tests, documentation, command-line parser (expr.c holds only expression 
      parser)

Ref #170

aktau added a commit to aktau/neovim that referenced this issue Mar 6, 2014

First working version of expression parser
WARNING: part of expr.c after //FIXME!!! must be deleted after I figure out how 
         to properly do tests.

TODO: tests, documentation, command-line parser (expr.c holds only expression 
      parser)

Ref #170

luke-gru commented Mar 6, 2014

Hi everyone, I don't know how helpful this is but I've been working on a language called riml that compiles to Vimscript. It's not 100% compatible with Vimscript (does not support most ex commands) and it adds features like interpolation and classes.

The parser is generated by racc (ruby's version of yacc/bison). Unfortunately the lexer is handwritten right now instead of being generated.

Also, there are lots of tests that test weird Vimscript edge cases I've run into. Right now these tests are mixed in with the tests for the added features, but it might be helpful if I isolate them for reuse.

What are your thoughts?

Owner

tarruda commented Mar 6, 2014

@luke-gru that seems interesting but what we'll do is the contrary: compile vimscript to lua

luke-gru commented Mar 6, 2014

Yes I know, but I was wondering if the BNF grammar would be of any use, as the language is an extension of Vimscript, and most rules are there for parsing Vimscript.

Owner

justinmk commented Mar 6, 2014

I would think the unit tests would be valuable, if ported to moonscript.

Owner

tarruda commented Mar 6, 2014

@luke-gru that is another story, if your language already parses a subset of vimscript then yes, its will be very useful. Thanks for sharing it here!

Owner

tarruda commented Mar 6, 2014

@luke-gru Thanks again, you already saved us a lot of work.

About the hand-written lexer, I was expecting having to write one to deal with vimL edge cases(it also helps simplifying the grammar), so I can certainly learn a lot from your code.

luke-gru commented Mar 6, 2014

No problem, let me know if anything is unclear or if you have general questions about the grammar that I might be able to help with.

Also, in compiling Vimscript to Lua (keep in mind I don't know Lua well at all), you will have to type check variables somehow because of the way Vimscript deals with the '.' (concat) operator.

For example:

let s = 'string1'
let s2 = 'string2'
echo s.s 
" => 'string1string2'

let d = { 'key' : 'val' }
echo d.key
" => 'val'

This is just an example of the trickiness that is going to be involved in the Vimscript => Lua compiler.

Member

aktau commented Mar 6, 2014

@luke-gru Very good points! I'm sure we can use your considerable knowledge!

This is the global issue: #243

The point you just raised about concat vs. subscript already has a running issue: #240

Another ambiguity: #251

We hope you can contribute some expertise ;)

luke-gru commented Mar 6, 2014

@aktau Thanks for pointing me to the proper issues! I did not know about #251, but it makes perfect sense. I'll post over in #243 for now, and I'll definitely find time to document some of the troubles and edge cases I found.

@justinmk justinmk added this to the first release milestone May 24, 2014

ZyX-I added a commit to ZyX-I/neovim that referenced this issue Jun 29, 2014

First working version of expression parser
WARNING: part of expr.c after //FIXME!!! must be deleted after I figure out how
         to properly do tests.

TODO: tests, documentation, command-line parser (expr.c holds only expression
      parser)

Ref #170

Grimy pushed a commit to Grimy/neovim that referenced this issue Jan 7, 2015

Member

elmart commented Jan 20, 2015

Closing, as it doesn't seem relevant anymore.

@elmart elmart closed this Jan 20, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment