Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

[WIP] VimL to lua translator #243

Open
wants to merge 188 commits into from
@ZyX-I
Collaborator

This is the current work on VimL to lua translator.

TODO:

  • expression parser
  • expression parser tests
  • expression parser documentation
  • Ex commands parser

    • Structure commands (branching, cycles, function definitions, try block)
    • Modifier command family
    • :*do command family (argdo, bufdo, etc)
    • :edit command family
    • Commands that use patterns (:s, :g, …)
    • Other commands
  • Ex commands parser tests

    • Structure commands (branching, cycles, function definitions, try block)
    • Modifier command family
    • :*do command family (argdo, bufdo, etc)
    • :edit command family
    • Commands that use patterns (:s, :g, …)
    • Other commands
  • Ex commands parser documentation

    • (partial) Present functions documentation
    • Developer documentation
  • VimL to lua translator

    • Complex variable names (curly braces names)
    • Block commands

      • :if block
      • :while block
      • :for block
      • :try block
      • flow control commands: break and continue
    • Assignment support

      • :let assignment
      • :let slice assignment
      • :function definitions
      • :let modifying assignment (+=, -=, .=)
      • :unlet
      • :delfunction
    • :command definitions

    • Error handling (saving error positions)
    • Commands taking other commands as arguments (modifier, :*do, :e +cmd, …)
    • Other commands
  • VimL to lua translator tests

    • Complex variable names (curly braces names)
    • Block commands

      • :if block
      • :while block
      • (partial: no tests for modifying lists during iteration) :for block
      • :try block
      • flow control commands: break and continue
    • Assignment support

      • (partial: only local and global scopes) :let assignment
      • :let slice assignment
      • (partial) :function definitions
      • (partial: local and global scopes) :let modifying assignment (+=, -=, .=)
      • (partial) :unlet
      • (indirect) :delfunction
    • :command definitions

    • Commands taking other commands as arguments (modifier, :*do, :e +cmd, …)
    • Other commands
  • VimL to lua translator documentation

    • Present functions documentation
    • Developer documentation
  • Backend lua module

    • Core types

      • Scalar types
      • Container types

        • Locks
        • Type literals
      • Function references

    • Operator support

      • Arithmetic operators (+-*/%)
      • String concatenation
      • Pattern matching
      • Comparison operators (less/greater then (or equal to), equality)
      • Identity operators (is/isnot)
      • Type coercions when using operators
    • Subscripting support

      • [key] dictionary, list, string and number subscripting
      • [idx1:idx2] list, string and number slicing
      • (args) function call
    • User command calls

    • Error handling
    • Built-in functions implementation

      • Implemented functions: string(), call(), type(), copy(), deepcopy()
      • Partially implemented functions: function()
      • Other functions were not implemented
    • User functions support

      • :function definitions
      • Dictionary functions
      • Defining functions right inside a dictionaries
      • Varargs functions
      • Range functions
      • Functions with abort modifier
    • Built-in commands implementation

    • User commands support
    • Translations support
  • Backend lua module tests

    • Core types

      • Scalar types
      • Container types

        • Locks
        • Type literals
      • Function references

    • Profiling

    • Operator support

      • Arithmetic operators (+-*/%)
      • String concatenation
      • Pattern matching
      • (partial: no check whether &ic option applies) Comparison operators (less/greater then (or equal to), equality)
      • (partial: no check whether &ic option applies) Identity operators (is/isnot)
      • Type coercions when using operators
    • Subscripting support

      • [key] dictionary, list, string and number subscripting
      • [idx1:idx2] list, string and number slicing
      • (args) function call
    • User command calls

    • Error handling
    • Built-in functions implementation

      • Tested functions: string(), copy(), deepcopy()
      • Partially or indirectly tested functions: function()
      • Other functions were not tested
    • User functions support

      • :function definitions
      • Dictionary functions
      • Defining functions right inside a dictionaries
      • Varargs functions
      • Range functions
      • Functions with abort modifier
    • Built-in commands implementation

    • User commands support
  • Backend lua module documentation

  • Old implementation replacement (need to replace some entry point functions)
  • Old implementation removal (need to remove old backend functions that were replaced by lua code)
  • vim/neovim incompatibilities documentation (ref #387)
@ZyX-I
Collaborator

Ref #170.

@tarruda
Owner

@ZyX-I since you are doing this in C, perhaps you'd like to use bison with a hand-written lexer?

@ZyX-I
Collaborator

@tarruda Since I am taking code from eval.c and other vim files I do not like to use bison. Also note that lexer can only be used with expressions (there are no other cross-command entities) and expressions parser is already written and working.

@ZyX-I
Collaborator

Ref #246.
Ref #240.

src/expr.c
((1,176 lines not shown))
+ puts((char *) error.message);
+ puts((char *) error.position);
+ free_node(result);
+ return NULL;
+ }
+
+ return result;
+}
+
+//FIXME!!!
+#ifdef COMPILE_TEST_VERSION
+#include <stdio.h>
+static void print_node(int indent, ExpressionNode *node)
+{
+ char *name = NULL;
+ switch (node->type) {

I'm unclear on how much of this is WIP and how much isn't, but I'm uncomfortable with how liberally this switch relies on fall-through. It's, like, 40 cases.

@ghost
ghost added a note

Everything after 1185 is temporary I believe

@ZyX-I Collaborator
ZyX-I added a note
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@oni-link
Collaborator

In parse2() and parse3() the node types are switched.

src/expr.c
((1,152 lines not shown))
+
+ next_node = &((*next_node)->next);
+
+ // Get the third variable.
+ *arg = skipwhite(*arg + 1);
+ if (parse1(arg, next_node, error) == FAIL)
+ return FAIL;
+ }
+
+ return OK;
+}
+
+ExpressionNode *parse0_err(char_u *arg, ExpressionParserError *error)
+{
+ ExpressionNode *result = NULL;
+ char_u *p = arg;
@dpelle
dpelle added a note

Why initialize p to arg here, when it's initialized again later before it is used.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
src/expr.c
((89 lines not shown))
+ */
+static char_u *find_id_end(char_u **arg)
+{
+ char_u *p;
+
+ // Find the end of the name.
+ for (p = *arg; isnamechar(*p); ++p) {
+ }
+ if (p == *arg) // no name found
+ return NULL;
+ *arg = p;
+ return p - 1;
+}
+
+/*
+ * Return 5 if "p" starts with "<SID>" or "<SNR>" (ignoring case).
@luke-gru
luke-gru added a note

Not sure if this is taken into account, but a scope name can be used with <SID>/<SNR>

ex:

function! s:<SID>SomeFunc()
endfunction

is valid Vimscript

@ZyX-I Collaborator
ZyX-I added a note

It is not. <SID> and <SNR> can only be present at the start of the function name.

@luke-gru
luke-gru added a note

Oh sorry, I meant:

function! <SID>s:SomeFunc()
endfunction
@ZyX-I Collaborator
ZyX-I added a note

@luke-gru Both by vim and by my parser this is handled identically: just like it handles

function abc:def:ghi:jkl()
endfunction

. There is nothing special in s: in your example.

@luke-gru
luke-gru added a note

Ah okay great, btw awesome work!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
src/expr.h
((34 lines not shown))
+ kTypeModulo, // %
+
+ // Unary operators
+ kTypeNot, // !
+ kTypeMinus, // -
+ kTypePlus, // +
+
+ // Simple value nodes
+ kTypeNumber, // 0
+ kTypeFloat, // 0.0, 0.0e0
+ kTypeDoubleQuotedString, // "abc"
+ kTypeSingleQuotedString, // 'abc'
+ kTypeOption, // &option
+ kTypeRegister, // @r
+ kTypeEnvironmentVariable, // $VAR
+
@luke-gru
luke-gru added a note

Do you have a type for literal scope names like g:?

ex:

echo g:

Edit: just noticed that it's handled correctly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
src/expr.c
((640 lines not shown))
+ case '1':
+ case '2':
+ case '3':
+ case '4':
+ case '5':
+ case '6':
+ case '7':
+ case '8':
+ case '9': {
+ char_u *p;
+
+ s = *arg;
+ p = skipdigits(*arg + 1);
+ e = p - 1;
+ type = kTypeNumber;
+
@luke-gru
luke-gru added a note

Missing octal/hex/unicode number parsing

@ZyX-I Collaborator
ZyX-I added a note

It is not “missing”. It is omitted. Number and string parsing is deferred to translator because in most cases I can just put the value into lua code unchanged.

@ZyX-I Collaborator
ZyX-I added a note

Wait, I do not actually have code that determines borders of such numbers. So you are right, it is missing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@aktau
Collaborator

Fantastic work, I didn't actually understand all of it on a quick readthrough, but I can see it's impressive!

@ZyX-I
Collaborator

Ref #387

@jfelchner

It's amazing to see such quick progress and our donations being put to good use. I am so excited for this you all have no idea. :smile:

@aktau
Collaborator

It's amazing to see such quick progress and our donations being put to good use. I am so excited for this you all have no idea. :smile:

As far as I know, the donations are only used to pay for @tarruda 's work on neovim (that's all the features in his branches + management), so the (indeed amazing as well) work @ZyX-I has been doing is entirely volunteer, unless people have been talking in private channels for sharing donations. So I think an applause and some props is in order. Great work @ZyX-I ! Let's keep this issue on topic now though ;)

@jfelchner

Let's keep this issue on topic now though ;)

I think that telling someone how awesome they are for their contributions is always on-topic.

Thanks @ZyX-I for all your hard work!! :smile:

@mhinz

@ZyX-I Do you think that backporting patches is worthwhile if they concern VimL (and therefore eval.c)? I ask here, since you're primarily working at eval.c at the moment.

E.g. your own 7.4.247

@ZyX-I
Collaborator
@aktau
Collaborator

so to make testing easier I may create an unfinished VimL code beautifier before proceeding with lua emitter

So you'd make a vim fmt analogous to go fmt? That actually sounds pretty neat, it's done wonders for the Golang community at the very least.

They cannot be backported.

As I surmise, eval.c hasn't changed a while lot from stock vim, and you aren't working on it, so why could the patches not be applied to neovim?

@justinmk
Owner

@aktau eval.c will be removed after the VimL translator is done. We could apply the patches in the meantime, I guess.

@aktau
Collaborator

@justinmk yea, I'll be happy to see it go, was just wondering why @ZyX-I was saying that it cannot be done. Wasn't making a statement about whether or not it should be done. In my opinion, if the changes are small, we might just as well apply them.

@ZyX-I
Collaborator
@aktau
Collaborator

I would say that this depends on the decision whether this branch should be the part of the first release or not. If it should it does not make sense. If it should not it makes sense. Note that as I can work on this branch only at weekends or during summer (or when I am officially ill so do not have to work like two or three weeks ago) it is unlikely to be finished until summer.

@ZyX-I understood. I'm not even sure what the release policy is (I believe it might be a bit early, we still have a lot of churn going on). As for when to incorporate luaviml: as soon as possible, and it doesn't need to be perfect. The catch is that we'll put it behind a runtime "experimental" switch, so the adventurous can turn it on and report bugs. With that in mind, I'm sure we can patch the current eval.c and you can just keep on working with the current luaviml, the only people that will experience trouble will be those who have scripts that explicitly make use of the new patches (which I'm going to guess will be 0 people for the next year).

@mhinz

Putting it in a nutshell: there should be no problem just backporting all missing Vim patches and if parts of them get removed in the first release, so be it.

Perhaps that's even better then forgetting to include certain important Vim patches that won't be removed over the course of luaviml development or, well, any other ideas/refactorings.

@ZyX-I
Collaborator

Rebased and squashed everything into one commit.

It appears that I should not have spent a few hours trying to rebase properly because git has thrown away all the work done during merges it could throw. Editing existing rebased commits so that they make any sense will require another few hours, so I just squashed everything. luavim-old branch (note the typo: luavim-old, not luaviml-old) with merges is available for historical purposes.

@aktau
Collaborator

@ZyX-I, first of: great work! A squash might've been the right idea with the incredible amount of commits you had.

Second: aren't vim variables always varnumber_T (or go through a varnumber_T variable, at least), which is defined as int. Which would effectively mean 32-bits on all bit the weirdest platforms. Since a double can represent 2^53 (or 2^52, I keep forgetting) integers, it sounds like it shouldn't be a problem. (I believe this is also the reason that the Lua authors only use double).

@ZyX-I
Collaborator

@aktau Vim numbers were already decided to be lua numbers. I remember the discussion, though not sure whether I expressed explicitly the result. Problem is not numbers, problem is buffer/window/etc identifiers: they are uint64_t and casting them to doubles risks changing them. In that discussion it was concluded that lua integers should be used for them. I do think they will be assigned sequentially incremented numbers, but I must not rely on this. Thus translator needs to be changed to use userdata values. I do not think this will present any problems.

@aktau
Collaborator

@ZyX-I I don't know if Lua does any bit wrangling, but double and uint64_t fit in the same space and it's not like they're used for printing and/or calculating, so perhaps just passing it will be fine.

@ZyX-I
Collaborator

@aktau You mean using union hack/memcpy/pointer cast? This should work (I am not going to expose these identifiers to VimL code), but I usually try to avoid such code.

@aktau
Collaborator

As far as I know, a union "hack" is perfectly legal C though (which is why I used it for the high resolution timer / profiling code that interacts with VimL). The undefined behaviour is when you just plain cast (through void or whatever). You see the union type punning in a lot of codebases as well (Windows (QPC), Linux, ...). I find it rather clean, as long as you only perform operations on one of the "aliases".

@ZyX-I
Collaborator

@aktau All three variants are perfectly legal C:

uint64_t src;
double tgt;

// 1
union {
    uint64_t u;
    double d;
} tmp;
tmp.u = src;
tgt = tmp.d;

// 2
tgt = *((double *) &src);

// 3
memcpy(&tgt, &src, sizeof(src));

Either three need static assert that sizeof(src) == sizeof(tgt). First and last are not going to have problems with alignment, second probably requires -malign-double for converting double to uint64_t (double is 4 bytes aligned without this option, uint64_t is 8 bytes aligned). Second and third are less verbose, but I understand that union hack is preferred because second may have problems with unaligned memory access and third will be slower unless compiler is able to optimize that memcpy call.

@aktau
Collaborator

I was under the impression that the second one wasn't truly advisable. Perhaps it's because it messes with many compilers' (cough, gcc, cough) strict aliasing optimizations. Anyway, I have a preference for union type-punning, fast and clear. I've used static asserts for the profiling related code as well, as you've noticed.

@philix
Collaborator
@ZyX-I
Collaborator

@philix To know whether changes are necessary you need to wait until I actually start removing old code. I am currently in process of making tests work, then I was going to add tests for actual translator and only then I was going to start looking at APIs that will make Neovim able to call new translator.

After I know how to communicate with Neovim in both directions I can replace existing functions that call VimL with new ones and at this point neovim will be utterly broken (commands implementations are far from finished). At this point replacement may be done with conditional macros and only after it is finished it makes sense to merge. Reaching this point means that only trivial, but very time-consuming tasks are left: create implementations, test. (And, most likely, non-trivial “optimize”.) Before I cannot say how many refactorings I will have to do, merging will mean that doing refactoring will be harder on my side.

As a contributor I would like to know how my changes would interact with the translator code (are they necessary? Am I changing the latest code? Would my refactorings apply to the translator code?...) and I think @ZyX-l would benefit from the improvements in the rest of the codebase that keep coming (e. g. simplification of eval.c, auto generated headers as himself implemented...) avoiding duplicate work and complicated merges that are so hard we would rather squash everything and lose history.

Merges are not too complicated. What is complicated is rebasing. I hope there will not such refactorings that apply to the whole codebase like moving files to nvim and autogenerating headers. I had more then 3 hundred commits, and most of them were things like “Fix typo” and “Fix XXX”, they should definitely have been squashed (with better precision though).

@ZyX-I
Collaborator

Now I need #905 to make sure unit tests pass.

@ZyX-I
Collaborator

@Hinidu @aktau Bad. I am wondering why there is extension pattern at all: it means that busted will run things like various helpers.moon as tests. It is bad because test/unit/helpers.moon contains initialization calls and I just realized that they are run twice.

So the correct approach is to rename all tests to test_* and use test_ as a pattern.

@Hinidu

So the correct approach is to rename all tests to test_* and use test_ as a pattern.

@ZyX-I we can remove --pattern option and use default busted pattern _spec in filenames.

@aktau
Collaborator

_spec seems like another decent option, indeed.

@Hinidu

_spec seems like another decent option, indeed.

I can make a PR if @ZyX-I also agree with this option.

@ZyX-I
Collaborator

@Hinidu Yes, I think using default _spec is a good idea. Got used to naming my own tests as test_* in powerline.

@aktau
Collaborator

Alright, glad we agree on that, good suggestion @Hinidu. Will be happy to see the PR whir by ;).

@ZyX-I
Collaborator

Note: new prefixes in commit messages have nothing to do with tasks. These are component prefixes to help me find commits which should be squashed.

@aktau
Collaborator

Note: new prefixes in commit messages have nothing to do with tasks. These are component prefixes to help me find commits which should be squashed.

This is how I usually do it for commits I know I need to squash as well, makes things easier.

@ZyX-I
Collaborator

Good news here: I have finished locks support and it still runs faster (17s vs 20s) than vim on $'let i = 0\nwhile i < 10000000\nlet i += 1\nendwhile' which is rather surprising. After locks support is finished such things as assignments (and basic container type operations that modify them (as in Vim scopes are almost regular dictionaries)) should not grow any slower.

@aktau
Collaborator

It really is very good news that an unoptimized implementation is already a bit faster than vanilla. Great work @ZyX-I!

@LeonB

@ZyX-I sounds great!

@ZyX-I
Collaborator

More then 75% of commands are implemented in parser.

@justinmk
Owner

Awesome!

@justinmk justinmk added the WIP label
@xueyayang

great job.

@ZyX-I
Collaborator

Finally added the last (:syntax) command parsing support. The next plan is:

  • Rebase another time.
  • Do a bit of refactoring (I have actually happened came up with a better architecture for the parser, but doing such kind of things now is not a good idea, so only a tiny bit).
  • Implement some TODOs/FIXMEs, fix some known bugs.
  • Fix travis tests.
  • Fix some non-fatal valgrind errors.
  • Proceed with #1536.
@Jookia

Amazing work!

@ZyX-I
Collaborator

Finally at least clang and clint tests were fixed.

@ZyX-I
Collaborator

I love gcc -Wconversion:

/home/travis/build/neovim/neovim/src/nvim/viml/testhelpers/parser.c:186:24: error: conversion to ‘char’ from ‘int’ may alter its value [-Werror=conversion]

182   do {
183     uint8_t digit = (uint8_t) ((offset >> ((i - 1) * 4)) & 0xF);
184     const char s[] = {(digit < 0xA
185                        ? (char) ('0' + digit)
186                        : (char) ('A' + (char) (digit - 0xA)))};
187     if (write(s, 1, 1, cookie) != 1) {
188       return FAIL;
189     }
190   } while (--i);

Where did this thing find fucking int? And I don’t have this warning with gcc-4.8.3.

@ZyX-I
Collaborator

And here as well:

/home/travis/build/neovim/neovim/src/nvim/viml/testhelpers/parser.c:186:24: error: conversion to ‘char’ from ‘int’ may alter its value [-Werror=conversion]

182   do {
183     uint8_t digit = (uint8_t) ((offset >> ((i - 1) * 4)) & 0xF);
184     const char s[] = {(digit < 0xA
185                        ? (char) ('0' + digit)
186                        : (char) ((char) 'A' + (char) (digit - (uint8_t) 0xA)))};
187     if (write(s, 1, 1, cookie) != 1) {
188       return FAIL;
189     }
190   } while (--i);

. This is insane.

@Oberon00

Where did this thing find fucking int?

The literals (e.g. 1) have the type int. Also (some?) operators promote their operands before performing the computation.

@ZyX-I
Collaborator

The literals (e.g. 1) have the type int. Also (some?) operators promote their operands before performing the computation.

All literals were casted in some of the latest versions. Even casting absolutely everything inside ?: expression did not fix the problem because I needed to cast the result of the ternary operator.

CLang and later gcc do not have this problem.

ZyX-I added some commits
@ZyX-I ZyX-I core: Move MENUDEPTH macros to menu.h 6b34b77
@ZyX-I ZyX-I core: Make replace_termcodes and friends accept length and cpo_flags
Reasons:
- One does not have to do `s[len] = NUL` to work with these functions if they do
  not need to replace the whole string: thus `s` may be const.
- One does not have to save/restore p_cpo to work with them.
708e2e7
@ZyX-I ZyX-I core: Add findoption_len function
It is like findoption(), but works with non-NUL-terminated strings.
460e376
@ZyX-I ZyX-I core: Add get_option_properties function f897b5d
@ZyX-I ZyX-I core: Add MAX_CHAR_LEN macros 59bdd2a
@ZyX-I ZyX-I core: Move event definitions to the generator script 2d2ab4c
@ZyX-I ZyX-I converter: Do not use lua_*integer functions 3f541b9
@ZyX-I ZyX-I *: Start creating viml-lua translator
Progress so far:
- Slightly more then 50% of commands are covered by parsers.
- Almost no commands are supported by translator (mainly supporting block
  (:while/:if/:for/:try)).
- No commands use Vim API.
- No developer documentation.
- Existing implementation was not replaced.

- Top-level parser was created.
- Top-level translator was created, most of commands not covered by translator
  are expected to work automatically once underlying implementation is added.
- Lua bindings have incomplete types support, and complete operator and
  assignment support.

- Most of used functions that are to be documented are documented.

- Lua bindings do not support integers that do not fit into float: luajit
  appears to have integers, but not visible C function for creating them, lua
  5.3 integers support was not tested.
882631a
@ZyX-I ZyX-I contrib: Make YCM work with .c.h files 4ff10bb
@ZyX-I ZyX-I core: Make get_histtype accept length argument ebeb988
@ZyX-I ZyX-I core: Make get_histtype return HIST_DEFAULT if needed 7be1360
@ZyX-I ZyX-I core: Add find_key_option_len function b8a9678
@ZyX-I ZyX-I core: Use findoption_len in do_set d5834f8
@ZyX-I ZyX-I core: Split out get_option_properties_idx() function d5fe6a0
@ZyX-I ZyX-I core: Allow zero-length options
Code that expected NUL-terminated strings allowed them and this behaviour is
actually used.
4d527df
@ZyX-I ZyX-I contrib: Fix ycm_extra_conf
fix! Somehow raise ValueError() used for debugging purposes slipped here.
46917fa
@ZyX-I ZyX-I scripts: Make it build with GCC and macros-generated declarations
GCC may emit declarations like the following:

    static size_t

    # 42 "this/file.c"
    function(args)

for code like this:

    static FDEF(function)

. Declaration generator script does not expect `# …` inside declarations. This
commit makes it treat such things as comments.
bb4ea04
@ZyX-I ZyX-I translator/tests: Fix format warning emitted by GCC
`%o` format is only used for unsigned integers. Also add test to make sure that
`-octal` is parsed as `(unary-)((octal))`: otherwise translator will have to add
special handling for leading unary minus.
acd820d
@ZyX-I ZyX-I testhelpers: Add code for dumping objects
Untested
1f16837
@ZyX-I ZyX-I translator/lua: Use function that returns top-level state dictionary 99e3fb3
@ZyX-I ZyX-I translator: Add support for modifying assignments
I.e. let +=, let -=, let .=.
81d8e93
@ZyX-I ZyX-I translator: Get rid of indent variables: use lua blocks instead 37c7b2d
@ZyX-I ZyX-I testhelpers: Add executor test helpers, including Object dumper d0e1635
@ZyX-I ZyX-I translator/lua: Add support for v:exception 1da2bbe
@ZyX-I ZyX-I lua: Assume that type functions are always passed correct values cddca77
@ZyX-I ZyX-I lua: Get rid of non_nil decorator in most functions
It is slow.
1f6d8ad
@ZyX-I ZyX-I lua: Refactor and extend comparison support
In previous state it was too slow and did not work for lists and dictionaries.
415bb29
@ZyX-I ZyX-I lua: Types layout refactoring e4c504d
@ZyX-I ZyX-I tests: Test for failing list assignments b0971fa
@ZyX-I ZyX-I tests: Start creating executor tests f260889
@ZyX-I ZyX-I tests: Add v:exception tests 7298acc
@ZyX-I ZyX-I lua: Improve numeric operations support
Integer division is rather hacky: math.floor(15 / 7) will return 2 (same as
integer division), but math.floor(-15 / 7) will return -3, while integer
division returns -2.

Thus I use math.floor() on absolute value and then apply sign.
53017e1
@ZyX-I ZyX-I translator/lua: Add :unlet and :delfunction support
No tests yet.
e08b235
@ZyX-I ZyX-I tests: Use :unlet to fix tests
Problem: blocks.lua tests are executed after assignments.lua, but require g:a
variable to be undefined. Assignments tests define this variable though.
55989f9
@ZyX-I ZyX-I lua: Improve integer division by zero
According to :h expr-/ division by zero must have always the same results for
Numbers:

>     When dividing a Number by zero the result depends on the value:
>     	  0 / 0  = -0x80000000	(like NaN for Float)
>     	 >0 / 0  =  0x7fffffff	(like positive infinity)
>     	 <0 / 0  = -0x7fffffff	(like negative infinity)
>     	(before Vim 7.2 it was always 0x7fffffff)
4339f27
@ZyX-I ZyX-I lua: Improve modulo by zero support
According to :h expr-%

> When the righthand side of '%' is zero, the result is 0.
e8758db
@ZyX-I ZyX-I lua: Make modulo with negative numbers behave like in Vim
n1 | n2 | n1 % n2 (Vim) | n1 % n2 (lua) | Difference
-- | -- | ------------- | ------------- | ----------
 1 |  2 |       1       |       1       |   absent
-1 |  2 |      -1       |       1       |     -+
 1 | -2 |       1       |      -1       |     +-
-1 | -2 |      -1       |      -1       |   absent

I do not see this behavior mentioned in help though.
e2e2674
@ZyX-I ZyX-I lua: Disable modulo operator for floating-point values 77051c1
@ZyX-I ZyX-I tests: Add some :unlet tests 8e2433a
@ZyX-I ZyX-I lua: Add number.negate and number.promote_integer
Should be a bit faster then generic versions from scalar dictionary.
e36b379
@ZyX-I ZyX-I lua: Add support for hexadecimal numbers in string e1dc060
@ZyX-I ZyX-I tests: Start adding type conversions tests 1d21d73
@ZyX-I ZyX-I lua: Add support for function() function e064acd
@ZyX-I ZyX-I lua: Add string() support and :echo {} support 6adb313
@ZyX-I ZyX-I tests: Start creating operator tests e706b07
@ZyX-I ZyX-I tests: Add funcref type conversions and operator tests 68f8ebc
@ZyX-I ZyX-I testhelpers: Show translated code if NEOVIM_SHOW_TRANSLATED_LUA is set b058688
@ZyX-I ZyX-I translator: Assign a: scope values 8ed8eae
@ZyX-I ZyX-I tests: Add tests for some a: access
Only positive tests currently, no tests for exceptions (as they are not
implemented anyway).
7f89131
@ZyX-I ZyX-I tests: Add :unlet! test 348b881
@ZyX-I ZyX-I translator/lua: Refactor function calls support
- Record l:self.
- Create binding in subscript.subscript, get rid of special subscript.func
  function.
- Add bound function type.

Previous solution did not work.
946f515
@ZyX-I ZyX-I tests: Start adding :function tests for function definitions c57f80b
@ZyX-I ZyX-I parser: Improve documentation in parser/ex_commands.h 4fd9773
@ZyX-I ZyX-I *: Get rid of char_u where possible
Macros near the top of the file that redefine “external” char_u users are some
sort of TODO list that lists things that need to be replaced or adjusted.
85bd58a
@ZyX-I ZyX-I *: Replace some strings.h functions with memory.h/string.h ones b4fc9d7
@ZyX-I ZyX-I *: Save offsets in expression nodes
Saves expression in functions from expression.c in place of parsing newly
allocated string in ex_commands.c.h. One of the alternatives is saving (start,
len) pair in place of (start, end). (start, end) in the current state was a bit
easier to refactor to.
d93d7d3
@ZyX-I ZyX-I translator/lua: Add support for :let+= operation with lists 5d049ad
@ZyX-I ZyX-I translator/lua: Use vim.iter for :for loop
Adds proper error reporting and makes it easier to add iterators or lazy lists.
56871d5
@ZyX-I ZyX-I parser: Record skips (removed escape characters) in get_cmd_arg
Untested. Will be used for computing error offsets.
a7432c0
@ZyX-I ZyX-I testhelpers/lua: Move some helpers code to vim.lua d7f997b
@ZyX-I ZyX-I lua: Check for new/modified globals in test.finish 0f59915
@ZyX-I ZyX-I *: Use function that saves parsed strings in a number of places 96cacbd
@ZyX-I ZyX-I parser: Remove file name from CommandPosition, add it to ParserResult a503ded
@ZyX-I ZyX-I translator/lua: Save original strings 6dd600d
@ZyX-I ZyX-I *: Dump position information for error reporting
TODO? Check whether using strings is really faster then using table  literals.

Though there is another consideration: strings are for sure more compact and
translation result is already bloated enough to make it even more bloated by
using table literals.
d72f1f7
@ZyX-I ZyX-I parser: Rename parse_(one|mult|exprs?)
Renamings:

Old           | New
------------- | ----------------------
parse_one     | parse_one_expression
parse_mult    | parse_many_expressions
do_parse_expr | do_parse_expr_cmd
parse_expr    | parse_expr_cmd
parse_exprs   | parse_expr_seq_cmd
eec1732
@ZyX-I ZyX-I lua: Report a error when trying to call non-Funcref c1da0fa
@ZyX-I ZyX-I lua/tests: Prepend Vim(command) to v:exception 2516c6f
@ZyX-I ZyX-I lua/tests: Record throwpoint d6b3d1b
@ZyX-I ZyX-I translator/lua/tests: Support :throw command 8ae4f91
@ZyX-I ZyX-I translator/lua/tests: Improve :function definition support
Now user-defined functions support E118 and E119 error messages (not enough/too
many arguments).
7a775cd
@ZyX-I ZyX-I testhelpers: Fix one memory leak 870b097
@ZyX-I ZyX-I translator/lua: Record some function properties
Will be later used for :function without arguments, string() and so on.

Function properties are recorded in ephemeron table, so I do not have to care
about GC’ing them. Exact set of recorded options:

1. Function name.
2. Starting position (line number and column) of the first child of :function.
3. Position of the character just before `:endfunction` command.
4. Function arguments (stringified version).
5. `dict`, `abort` and `range` attributes.
6. `state.code` and `state.fname`.

Reasoning:

- 1 till 6 will be used for `:function Func` implementation. It looks like this:

           function {1}({4}){" abort" if 5.abort}{" range" if 5.range}{" dict" if 5.dict}
           {"        Last set from "..6.fname if verbose and 6.fname:sub(1, 1) ~= '<'}
        1  {6.code[2.line]:sub(2.column, -1)}
        …  {6.code[2.line + 1] till 6.code[3.line - 1]}
        N  {6.code[3.line]:sub(1, 3.column)}
           endfunction

  . More understandable example:

           function d.Abc(a, b, ...) range
                   Last set from ~/foo.vim
        1    echo a:a
        2    echo a:b
        3    return a:000
           endfunction
- 1. will be used in error messages (except for E118/E119 where exactly the same
  value is embedded by the translator without requiring to use value from
  ephemeron table).
- 5.dict will be used to raise E725 (calling dict function without
  a dictionary).
070e62b
@ZyX-I ZyX-I translator/lua: Fix dictionary functions support
Mostly untested. Should fix conditions under which bound functions are created:
previously d.Abc() did not create bound functions, but d['Abc'] always created
bound functions, even if it is was a simple subscripting.
c345e1c
@ZyX-I ZyX-I tests: Test for some intentionally incompatible behavior 9e55352
@ZyX-I ZyX-I lua: Raise E725 error when appropriate
This error is raised when trying to call dict function without a dictionary.
c978290
@ZyX-I ZyX-I lua: Add call() function support
Untested as usual
d0ca772
@ZyX-I ZyX-I lua: Remove val_position values that are never assigned or used 290d1fb
@ZyX-I ZyX-I lua: Finish string() support: support stringifying funcrefs 1e0a562
@ZyX-I ZyX-I tests: Add string() function tests f2a2c2a
@ZyX-I ZyX-I lua: Record globals at the very beginning of the file
Should catch globals that were accidently created below, in vim.lua. The only
exception is `vim` global: it is created just after `vim.lua` is loaded.
b0aaba0
@ZyX-I ZyX-I testhelpers/lua: Also test for differences in state
Tests should not alter used state: all variables and functions should be
deleted
78c1cda
@ZyX-I ZyX-I translator/lua: Add support for :function abort 5a2b653
@ZyX-I ZyX-I translator/lua: Fix exception handling: pcall() returns 2 values
Do not know where I have seen 3.
43cf64a
@ZyX-I ZyX-I tests: Check function return values cc95dcc
@ZyX-I ZyX-I parser/tests: Allow :call to accept only function calls
For ranges support I will need to obtain function reference separately from
function arguments.
7a606d7
@ZyX-I ZyX-I translator/lua/tests: Add basic support for :call command
Not done: range handling (ranges are globally not supported now).
68e926f
@ZyX-I ZyX-I translator: Fix :let modifying assignments
Previously they did not work for constructs like

    let [a, b] += [1, 2]

. Though such code is not normally written by sane programmers, I used to make
an optimization to squash successive :let’s (including :let+=) into one big
:let.
fff304e
@ZyX-I ZyX-I tests: Test :let list modifying assignments b8bf942
@ZyX-I ZyX-I lua: Add slice assignment support c2e78c5
@ZyX-I ZyX-I lua: Copy values in test_echo for testing purposes
Note: copy_value crashes if it tries to copy type dictionary as well.
3760211
@ZyX-I ZyX-I tests: Add slice assignment tests 0448231
@ZyX-I ZyX-I lua/tests: Add copy() and deepcopy() functions support 266bb38
@ZyX-I ZyX-I translator/lua: Remove all occurences of list.raw_subscript 2d69d3e
@ZyX-I ZyX-I lua: Add support for comparing self-referencing structures
Algorithm is taken from [Python mailing list][1], translated to lua and heavily
reworked to support existing types.

[1]: https://mail.python.org/pipermail/python-dev/2003-October/039445.html
da91fac
@ZyX-I ZyX-I tests: Test for recursive types comparison 6d939ff
@ZyX-I ZyX-I executor/lua: Use MB_STRICMP macros for ic string comparison 2f3e5ff
@ZyX-I ZyX-I tests: Test case-ignorant string comparison 0c9db24
@ZyX-I ZyX-I translator: Make empty subscripts use values different from nil’s
nil’s usually indicate errors. They definitely must do this in that context.
This change makes empty subscripts work.
aad2c7e
@ZyX-I ZyX-I tests: Add value creation and subscript tests a58e171
@ZyX-I ZyX-I lua: Raise an error when trying to slice a Dictionary 918e5f9
@ZyX-I ZyX-I lua: Fix scalar indexes b074662
@ZyX-I ZyX-I translator/lua: Add support for locks
No :lockvar yet and no tests.
b32a3c8
@ZyX-I ZyX-I lua: Add support for `{scope_char}:` c26a16a
@ZyX-I ZyX-I tests: Add :for loop tests 8e55af1
@ZyX-I ZyX-I translator: Refactor translate_lval to accept two additional arguments
Will be useful for (un)lockvar implementation
1424e53
@ZyX-I ZyX-I translator/lua: Add :(un)lockvar support f01e36e
@ZyX-I ZyX-I lua: Do not check locks when deleting key c2ee1b1
@ZyX-I ZyX-I lua: Adjust slice indicies computation for :lockvar
Will later be needed for slice :unlet
16bcab7
@ZyX-I ZyX-I lua: Do not copy() locks f6f0d81
@ZyX-I ZyX-I tests: Test how (deep)copy() preserves locks 85d4dca
@ZyX-I ZyX-I tests: Test :lockvar 922b50d
@ZyX-I ZyX-I lua/tests: When deleting subscript check locks 628b6af
@ZyX-I ZyX-I lua/tests: Add support for :unlet’ting slices eede882
@ZyX-I ZyX-I translator/lua: Make curly braces names work 5101134
@ZyX-I ZyX-I tests: Add tests for curly braces names f36318e
@ZyX-I ZyX-I tests: Add :while loop test e968a5a
@ZyX-I ZyX-I translator: Add :break command support a8c2cc1
@ZyX-I ZyX-I tests: Add :break command tests 666ebbe
@ZyX-I ZyX-I translator: Add :continue command support fb51bd9
@ZyX-I ZyX-I tests: Add :continue command tests f0b7000
@ZyX-I ZyX-I parser: Fix glob parser function, add some code for :edit dad3a7b
@ZyX-I ZyX-I printer/tests: Add some :edit tests, use glob AST when printing
Unless using glob AST it is impossible to distinguish between glob parsed as
shell call and glob parsed as a literal string.
86177b2
@ZyX-I ZyX-I parser: Use the same code for EDITCMD and FILES 310eda8
@ZyX-I ZyX-I parser: Use same code for all XFILE commands 6ffcf96
@ZyX-I ZyX-I parser/tests: Save the fact that enew needs no arguments c007d84
@ZyX-I ZyX-I parser/tests: Add tests for :edit command family 14375be
@ZyX-I ZyX-I parser: Hide some function declarations in a macros 3dc8a3f
@ZyX-I ZyX-I parser: Move code that creates comment nodes to a separate function ebdf92f
@ZyX-I ZyX-I parser: Replace xstrndup with xmemdupz
It is absolutely useless to check for zero-terminated byte in all places here in
xstrndup because surrounding code already did the checks when determining
length.
6ecc74e
@ZyX-I ZyX-I parser/printer/tests: Add various commands parsing support
Added:

- `:autocmd`
- `:*unmenu`
- `:buffer` and some other `:b*` buffer commands
- `:behave`
- `:breakadd`/`:breakdel`/`:profile`
- `:cbuffer` and friends
- `:left`/`:right`/`:center`
- `:checktime`
- `:clist`/`:llist`
- `:ilist` and friends
- `:catch`
- `:copy`/`:move`/`:t`
- `:command`
- `:yank`/`:delete`/`:put`: REGSTR commands
- `:!`
- `:debug`
- `:delcommand`
- `:display`/`:registers`
- `:diffget`/`:diffput`
- `:digraphs`
- `:doautocmd`
- `:doautoall`
- `:drop`
- `:earlier`/`:later`
- `:filetype`
- `:history`
- `:mark`/`:k`
- `:*make`/`:*grep*`
- `:nbstart`
- `:popup`
- `:retab`
- `:resize`
- `:redir`
- various script commands
- `:mode`
- `:open`
- `:profdel`
- `:*select` and `:*jump`
- `:[v]global`
- `:*vimgrep*`
- `:marks`
- `:match`
- `:py3`
- `:sbuffer`
- `:set*`
- `:sleep`
- `:substitute` and friends
- `:sort`
- `:spell*`
- `:syntime`
- `:tabmove` (with function for `:resize`, slightly modified)
- `:tearoff` (with function for `:popup`)
- `:winsize`/`:winpos`
- `:wincmd`
- `:z`
- `:*`/`:@`
- `:&`/`:~`
- `:wsverb`
- `:help`
- `:*helpgrep`

Note: kArgAddress is now using Range *range union member, but is not named
kArgRange. Reason: it still represents *one* address, but I have to use Range
structure due to `10;20` being a valid address (will use line 20, but in case of
`10;/abc` first part should be taken into account: this variant will use line
that matches `/abc` that is located after line `10`).

Note 2: last :command argument is not parsed. Reasoning: things like <args> need
to be expanded before parsing anything. I can redirect parsing of some things
like <f-args> to expressions parser or <line1> to range parser, but this is not
compatible and is not going to work for things like

    command -nargs=1 Sil silent! <args>
9f56709
@ZyX-I ZyX-I parser: Place top parse_one_cmd arguments differently cbb32ba
@ZyX-I ZyX-I parser: Move some code to parse_no_cmd function 6fb4c06
@ZyX-I ZyX-I parser: Move error returns down and do more frees b6faff7
@ZyX-I ZyX-I printer/translator: Use `p_` in place of `p` inside macros 5cb4bc6
@ZyX-I ZyX-I parser: Fix out-of-bounds memory access and some other valgrind errors
Out-of-bounds memory acess can be seen in all commands that have NOTRLCOM in
their definition and use get_cmd_arg to preprocess their arguments.
2f0e151
@ZyX-I ZyX-I parser: Move parsing a sequence of files to a separate function 8871ab7
@ZyX-I ZyX-I parser/tests: Add support for environment variables in patterns 05779f2
@ZyX-I ZyX-I *: Fix all -Wconversion warnings 1ec1ad1
@ZyX-I ZyX-I parser/tests: Add support for :cstag 3a20658
@ZyX-I ZyX-I parser/printers/tests: Add :gui/:gvim command support d2e9a17
@ZyX-I ZyX-I parser/printer/tests: Add :helptags support ccce989
@ZyX-I ZyX-I parser/printer/tests: Add support for :language command c3c9c26
@ZyX-I ZyX-I parser/tests: Mark the fact that :wq is already supported abeb6a5
@ZyX-I ZyX-I parser: Make some `s` variables more constant 0228218
@ZyX-I ZyX-I parser/printer/tests: Add support for :w, :up and :r eea5b1c
@ZyX-I ZyX-I parser/tests: Add support for :loadview e859ff2
@ZyX-I ZyX-I parser: When freeing command arguments free completion argument 2f2a328
@ZyX-I ZyX-I parser: Fix memory leak when creating error nodes for NOTDONE return 82f049c
@ZyX-I ZyX-I parser/printer/tests: Add support for :loadkeymap 8f31dd0
@ZyX-I ZyX-I parser/printer/tests: Add support for :mkspell command 69f62a8
@ZyX-I ZyX-I executor: End all strings passed to API with NUL
At least `vim_strwidth` API function expects NUL-terminated string.
9cbf803
@ZyX-I ZyX-I parser/printer/tests: Add :menutranslate parsing support a6ec471
@ZyX-I ZyX-I parser/printer/tests: Add support for :cscope command 8620adf
@ZyX-I ZyX-I tests: Make sure :si is treated correctly
It is rather weird case that deserves to be tested specifically.
9168e3a
@ZyX-I ZyX-I *: Fix gcc warnings
Fucking gcc thinks that `(char) (c << 3) + (char) (*curp - '0')` is doing some
int → char conversion. There are lots of other weird examples like
`-Wconversion`

    const char *curp = …;
    char c;
    c = *curp - '0';

.
ff12f8d
@ZyX-I ZyX-I parser/printer/tests: Add support for parsing of all remaining cmds
- :simalt
- :sniff
- :highlight
- :sign
- :syntax
0dd3056
@ZyX-I ZyX-I printer: Move printing of start/end character to print_regex
This way it could be escaped in one place.
295fb44
@ZyX-I ZyX-I printer/tests: Escape patterns properly ab37857
@ZyX-I ZyX-I printer: Remove one offending include a3108ae
@ZyX-I ZyX-I printer/tests: Escape ~ and expressions when needed 52b9bc7
@ZyX-I ZyX-I core: Export nr2hex function cc70b21
@ZyX-I ZyX-I parser/printer/tests: Add support for pattern collections 742a3e2
@ZyX-I ZyX-I tests: Remove [[ nesting 1282ae1
@ZyX-I ZyX-I parser/printer/tests: Add support for cmdline specials and modifiers 9262806
@ZyX-I ZyX-I tests: Add some regression tests 91aa38c
@ZyX-I ZyX-I executor: Rename vim_module.generated.c to .h
This way clang does not emit warnings about unused variables.
3672b65
@ZyX-I ZyX-I parser/printer/testhelpers: Silence most GCC warnings
The only one left is

    …/neovim/src/nvim/viml/parser/ex_commands.c: In function 'parse_highlight':
    …/neovim/src/nvim/viml/parser/ex_commands.c:5930:37: warning: conversion to 'unsigned int:24' from 'unsigned int' may alter its value [-Wconversion]
               current_color.rgb_1.rgb = (unsigned) strtoul(arg_start + 1, NULL, 16);
                                         ^

which looks like reincarnation of [the GCC bug][1].

[1]: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=35701
1a80daf
@ZyX-I ZyX-I parser: Silence last GCC warning
Apparently it was not reincarnation of that bug. Not very helpful since I don’t
need that bitwise AND and comment explaining why I don’t just assign.
117d577
@ZyX-I ZyX-I printer: Fix clint warnings in ch_macros file ff5100c
@ZyX-I ZyX-I printer: Ignore ch_macros.h file
Reason: due to defined macros clint.py has lots of false positives:

Warning                        | Explanation
------------------------------ | -----------------------------------------------
No #define header guard found. | None needed because this file is intended to be
                               | included multiple times.
                               |
Extra spaces before ( in       | This is about size_t (*FTYPE(t)) which has
function call.                 | nothing to do with a function call.
                               |
07380b3
@ZyX-I ZyX-I dumpers: Do not suggest to examine errno for fwrite function 07e92e4
@ZyX-I ZyX-I executor: Use STATIC_ASSERT in converter.c 069e09d
@ZyX-I ZyX-I *: Fix linter errors 0384d6e
@ZyX-I ZyX-I testhelpers: Make more casts 34b19e6
@ZyX-I ZyX-I fix! Default to utfTerminalDetailed e9da4d2
@ZyX-I ZyX-I testhelpers: Cast the whole expression 628dcf2
@ZyX-I ZyX-I parser: Fix gcc-32 -Wconversion warning 056865c
@ZyX-I ZyX-I parser/printer: Fix some valgrind errors that cause travis tests crash f822fad
@ZyX-I ZyX-I testhelpers: Make it possible to run parse_one_cmd from cmd executable e15bf98
@ZyX-I ZyX-I printer: Fix another valgrind error that leads to a crash d95475f
@ZyX-I ZyX-I tests: Add (commented out) code that writes test arguments bbbe3e8
@ZyX-I ZyX-I parser: Fix some memory leaks found by valgrind
Not all leaks were fixed yet.
736db50
@ZyX-I ZyX-I testhelpers: Use mch_exit to exit cmd
Reason: less blocks are reported as “still reachable” by valgrind if libnvim was
compiled with -DEXITFREE.
2550301
@ZyX-I ZyX-I parser: Fix remaining memory leaks found by valgrind 2ca5599
@ZyX-I ZyX-I parser/printer/tests: Add support for :profile start/pause/continue 09097de
@ZyX-I ZyX-I fix! Remove --lazy busted argument
It is not present in the latest busted version luarocks was able to install two days ago.
706fd10
@ZyX-I ZyX-I printer/parser/tests: Solve most remaining printer TODOs
Also fixes incorrect handling of &magic option in :substitute and friends.
1e62c26
@tarruda
Owner

I have finally took some time for a quick read through the code and have to say: Amazing work @ZyX-I, this will be by far the biggest code improvement to Neovim.

I'm still finding my way around the code, but I've seen that you are implementing builtin functions and commands in src/nvim/viml/executor/vim.lua. That seems to be the kind of horizontal work that could be split across multiple contributors. Correct me if I'm wrong, but porting a single builtin function or command consists of the following tasks:

  • Ensure the existence of any API functions(src/nvim/api) required to implement the function or command(eg: the :append command can probably be implemented on top of buffer_set_line_slice)
  • Implement the function or command in src/nvim/viml/executor.vim.lua(perhaps split this file into functions.lua, commands.lua, etc). If an API function is required, it will be available in the "api" object(wrappers generated by msgpack-gen.lua and added by nlua_add_api_functions, though I haven't seen you use this object in vim.lua)
  • Add some tests

This is something anyone can do if provided with instructions/examples of where/how to add the functions, tests, etc.

If you push this branch to Neovim repo and send a PR that implements a single command or function, we could use it as an example for distributng entry-level tasks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.