[WIP] Create lua interface #4411

Open
wants to merge 20 commits into
from

Projects

None yet

6 participants

@ZyX-I
Contributor
ZyX-I commented Mar 4, 2016
  • luaeval()
  • :lua* commands
  • luaeval() tests
  • :lua tests
  • :lua << tests
  • :luafile tests
  • :luado tests
  • compatibility vim module
  • compatibility vim module tests
  • documentation update

Ref #3823

@marvim marvim added the WIP label Mar 4, 2016
@ZyX-I
Contributor
ZyX-I commented Mar 4, 2016

Blocked by #4131: I think I can reuse encode.c again, it has very useful macros for traversing nested typval_T structures without using any recursion. I guess I can and should rewrite vim_to_object as well, currently it can crash when converting deeply nested objects: e.g. on my system vim_to_object_rec crashes with

nvim --cmd 'let l=[]|call map(range(10000), "extend(g:, {\"l\":[l]})")' --cmd 'call rpcnotify(1, "", l)'

(you may need to increase number inside range()).

@justinmk
Member
justinmk commented Mar 4, 2016

What do RPC client authors need to know in order to migrate after #4131, if anything? If we can provide clear answers to that, then I say just merge it.

@ianks
ianks commented Mar 4, 2016

Will this bring along with it the functionality generally required for if_lua?

@ZyX-I
Contributor
ZyX-I commented Mar 4, 2016

@ianks Look at the TODO list.

@ZyX-I
Contributor
ZyX-I commented Mar 4, 2016

@justinmk Where I should place notes regarding migration? Also there is still unresolved question with *set_var, I did not implement last proposal.

@justinmk
Member
justinmk commented Mar 4, 2016

@ZyX-I I'm speaking of #4131 the JSON PR. I haven't had time to fully understand the implications of the v:null change for the API, so I am not prepared to explain to client authors why their clients broke (if that happens).

Also there is still unresolved question with *set_var, I did not implement last proposal.

Oh. Ok, I will try to offer an opinion on it in the next days.

@ianks
ianks commented Mar 4, 2016

I did. I just don't know enough about Neovim internals to know what it all means.

@ZyX-I
Contributor
ZyX-I commented Mar 4, 2016

Because I got no replies whether it is the right path.

@justinmk
Member
justinmk commented Mar 4, 2016

@ianks Answer is "yes", mostly, though I am curious about how userdata will be handled, if at all.

(I'm not opposed to including lua/luajit in-process for nvim, I think it's important enough for the project to offer this as a first-class alternative to VimL, though it does make other language hosts "second class". Note that we should consider allowing lua to be substituted for luajit at build-time if possible, @jamessan has mentioned this for cross-platform support)

@ZyX-I
Contributor
ZyX-I commented Mar 4, 2016

At the current state differences are minor:

  1. Everything which returns converted typval_T (vim_eval, *_set_var, *_get_var) may return nil and boolean objects now and this (especially nil) no longer indicates any error.
  2. Everything which accepts a value which will be converted to typval_T (*_set_var) now converts nil objects to v:null and not 0, similarly boolean objects will be converted to v:true/v:false and not numbers.
  3. *_set_var(name, nil) will not delete the variable, clients need to use *_del_var(name) instead.
  4. *_set_var(name, val) returning nil no longer means that variable did not exist, it now means that variable did not exist or existed, but was set to v:null: that is last thing discussed.

1 and 2 should not break much, but 3 and any way to fix problem in 4 I can imagine will break some functionality.


@justinmk I always said that I am planning to be compatible with lua without jit.

@ZyX-I
Contributor
ZyX-I commented Mar 4, 2016

@justinmk Also note the branch name. I thought it is good idea to break luaviml branch into something smaller which is why this PR was added, not because I actually want :lua* and luaeval().

@DemiMarie

I would like for Lua to be able to add built-in functions to NeoVIM.

@ZyX-I
Contributor
ZyX-I commented Mar 17, 2016

@justinmk Last commit adds api/ref.c. It does not actually contain any new functionality, but it provides data about how I see Reference type and thus may be reviewed from this point of view.

@bfredl
Member
bfredl commented Mar 20, 2016

What is the ReferenceDef interface for? Is it the different viml container/function types, or do we imagine even more different reference kinds?

@ZyX-I
Contributor
ZyX-I commented Mar 20, 2016

@bfredl For different types. Specifically for deeper integration between languages, I wanted something like this for Vim: to be able to use Python functions or containers directly from Ruby etc. At a cost of converting Python types to VimL and only then to Ruby, of course.

In the current state using Python objects in Ruby is probably not the best idea (too many IPC calls: first from Ruby to Neovim, then from Neovim to Python, then sending replies back through this chain), though still applicable for some uses. But this allows adding things like partials with no changes on the client side.

I also considered adding Reference ref_environ(void) function which will simply provide interface to Vim environment without adding a set of new functions to the API. Same applies to accessing mappings, signs, autocommands, etc: everything which may have a mapping or sequence interface, it is easier to create “generic” objects which work with any Reference and forget about this than add a bunch of new functions every time access to new Vim internals is added. Note that only mappings and matches have something like a usable API (I mean functions like maparg() and getmatches()/setmatches()), signs and autocommands require you relying on :redir and parsing output of the commands which is both not very reliable and is easier to break when new functionality is added (like <cmd> from original variant of #4419 (currently it was replaced with <cmd> key)).


And another idea: make functions like ref_get_item accept an argument which will cause reference to be coerced to Object without additional request. This would reduce *_*_var (e.g. window_get_var) function family to ref_get_*_scope/*_get_scope (returns scope dictionary, example: ref_get_window_scope (ref.c) / window_get_scope (window.c)), but is clumsier to use: even though it is only +2 API calls (no matter how many times you need to use vim_*_var it adds only two API calls), with window_get_var you do not need to care about unreferencing anything.


I would also ask whether anybody actually wants *_del_* and *_set_* remove previous values? In existing API this is a huge waste of resources because 90% of time you do not need this and if you e.g. delete a large list you will get it serialized and transferred to the client. In the ref_* API it is huge waste of resources only when it comes to deleting a huge slice or deleting a lengthy string because deleting large list from a scope will only make API send a Reference to it. But developers of the clients will have to code sending ref_unref call back or there will be a memory leak on each *_del_* and *_set_* when deleting/replacing containers.

@bfredl
Member
bfredl commented Mar 20, 2016

This would reduce *_*_var (e.g. window_get_var) function family to ref_get_*_scope/*_get_scope

We could avoid the extra roundtrip by allowing the client construct the scope reference ( have an EXT type such that the object EXT can be converted to the scope EXT by a client)

@ZyX-I
Contributor
ZyX-I commented Mar 21, 2016

@bfredl I do not think this is a good idea:

  1. Server needs to know that client needs to keep scope dictionary. -1 initial API call is nothing compared to API that leaves client writers with the idea that scope reference does not need to be unreferenced (which is logical because reference was not created by the server or it does not look like it was created: from my point of view this looks like requesting something from a resource that needs to be opened and closed without opening it explicitly which makes me expect that closing it explicitly is also not needed).
  2. Two EXT types for one Reference type is not really good and I do not like to give up an idea “first n bits are capabilities listed in --api-info, everything else is none of your business” for an existing type.
  3. Depending on the implementation this either will make it impossible to compare reference identity by simple EXT data contents comparison or will make code that serializes Reference more complicated: because now you need to check whether given dictionary is the scope one and if yes serialize it one way else another. I fear determining what exactly scope dictionary needs to be serialized to will not be too easy (determining that dictionary is a scope one is easy because this information is saved).
@bfredl
Member
bfredl commented Mar 21, 2016

As I said before, I don't mean client constructable named references to replace (parts of) your proposal.
I would imagine a named reference to look something like this (NameRef( is here just a name for the EXT object type code, which can contain nested msgpack as its binary data)
NameRef(['gvars']) - g:
NameRef(['bvars', Buffer(objid)]) - b: for buffer EXT obj (or we can have ovars as the object type is understood)
or posibly even as an addon
NameRef(['gvars', mydyct]) - g:mydict
NameRef(['vars', Buffer(objid), 'mydict']) for b:mydict
It is understood that these namerefs are nullable (bvars will fail if the buffer handle was invalid), but otherwise pure (no generic viml eval with side-effects). Remote clients also understand that these cannot be compared for referential equlity, they need to be interned for comparison and for having a permanent handle to the actual object.

These will be converted to References early in the request dispatch logic. No bookkeeping, just a one-shot ext_nameref_to_reference. So this is a pure add-on and can be implemented later as another PR.

@ZyX-I
Contributor
ZyX-I commented Mar 21, 2016

@bfredl I think this is a good idea. Though I would not add it here, such references are not needed for lua interface.

@justinmk
Member
justinmk commented Apr 5, 2016

@ZyX-I Regarding the compatibility question:

*_set_var(name, nil) will not delete the variable, clients need to use *_del_var(name) instead.

*_set_var(name, val) returning nil no longer means that variable did not exist, it now means that variable did not exist or existed, but was set to v:null: that is last thing discussed.

We could use a similar approach as in #4083 (6eda7c0): create new functions and deprecate the old ones. But I think we can break back-compat here.

  • Deleting a variable is relatively uncommon task compared to typical task.
    • Doing this via an API call (instead of eval'd VimL) is even less common.
  • Checking the result of a set call is even less common; I would be surprised if anyone uses this feature.

Last time we made a breaking change to the API I just notified client/plugin authors to be aware of the change, and it worked out ok. We can still use this approach, though #4083 and more explicit versioning will be required when Neovim gets more popular.

@ZyX-I
Contributor
ZyX-I commented Apr 5, 2016

@justinmk

Checking the result of a set call is even less common; I would be surprised if anyone uses this feature.

Why does it exist then? I can simply make set and del void, this will avoid lots of unnecessary allocs/frees on both sides of the channel.


For backward compatibility I know the following approaches that are applicable here:

  1. Break it.
  2. Deprecated functions, removed in the major version. Note: it is better to give out information in the --api-info: at least Python with warnings module may give deprecation warnings without breakage.
  3. Versioned function: you request some API function and some version of it. Old versions of functions are kept, new versions do not need different name. Basically this is like naming set_vim_var set_vim_var__1_0 then replacing with set_vim_var__1_1.
  4. Versioned channels: when connecting to Neovim you request specific API version (or, to be compatible, use a separate API call until which version just before that call was introduced is assumed). Like previous variant, but version suffix is implicit. It is better to use semantic versioning here: when requesting API version 1.1 1.2 and 1.3 may be used as well. No patch version.

Points 3 and 4 may also have an addition “older version is removed in next major release”.

I do not remember API compatibility being discussed somewhere: did I miss some issue or should it be a new one?

@justinmk
Member
justinmk commented Apr 5, 2016

Why does it exist then? I can simply make set and del void, this will avoid lots of unnecessary allocs/frees on both sides of the channel.

I can't think of a reason why it would be needed, maybe @tarruda can comment.

Deprecated functions

I suggest we take this approach as far as we can because it is the least clever. "Versioned channels" would be the next step.

#1393 may be used to discuss API compatibility/versioning, though that issue regards "middleware" versions and not the nvim server itself.

@bfredl
Member
bfredl commented Apr 5, 2016

Another approach is: adding new parameters with default values for old clients that don't supply it. Though it is probably overblown for this case.

It might be useful for a rplugin to say get and delete a key atomically, but this is probably better served either by a general concept of atomic transactions

vim_exclusive_begin()
get_var(somekey)
del_var(somekey)
vim_exclusive_end()

implemented correctly this could even be streamed (only one roundtrip needed).
With this branch, another alternative for an rplugin to do an atomic transaction is of course to inject some lua code to do it.

@ZyX-I
Contributor
ZyX-I commented Apr 5, 2016

@bfredl This approach is rather limited which is why I did not list it:

  1. You may not use it to change return type. You can have some workarounds (in the current case that would be e.g. having optional argument force returning NIL always), but they are not always applicable.
  2. You may not use it to change type of one of the first arguments.
  3. --api-info does not support this currently, so in the current situation it still will introduce an incompatibility.

I think that the idea of atomic requests (not transactions) can be implemented with the current code rather easily: just make an API call which will receive list of pairs method_name, arguments and run this directly: not much of a problem, you just need handle_request that accepts String plus Array in place of msgpack_object request and runs on_request_event directly in place of queueing it, and on_request_event for this job which returns Object in place of writing serialized msgpack. Since there is a single thread which processes all requests, as well as scripts and user input, this will be atomic. But not transaction: if some API call fails previous calls’ effects are not undone.

@bfredl
Member
bfredl commented Apr 5, 2016

Well it will work with a new nvim with a client either statically generated for the old api info, or a client doing no checking to the metadata (like python), but versioned/deprecated functions as your suggestions will definitely scale better when things get Sufficiently Complicated (which they will).

make an API call which will receive list of pairs method_name, arguments and run this directly

This would be very useful, for sure.

@DemiMarie

Any thoughts on having a Lua API call to construct Lua-implemented functions and commands?

@ZyX-I
Contributor
ZyX-I commented Jul 10, 2016 edited

Current/planned incompatibilities:

  • (current) luaeval("_A", dict) converts second argument to lua table and then back. Vim: passes by reference, creating special userdata.
  • (planned) luaeval("1, 2, 3") is an error. Vim: returns 1.
  • (planned) luaeval("{foo=1}") ==# {"foo": 1}. Vim: is 0.
  • (planned) luaeval("{}") ==# []. Vim: is 0.
  • (planned) luaeval("{4,2}") ==# [4, 2]. Vim: is 0.
  • (planned) luaeval("{4, foo=1}") (mixed dictionary/list),luaeval("{[5]=1}") (non-list: just number key), luaeval("{[{}]=1}") (non-string non-number key) are errors. Vim: returns 0.
  • (planned) msgpack#equal(luaeval('{["\0\n\0"]="\0\n\0"}'), msgpack#eval('{="\0\n\0": "\0\n\0"}')) (luaeval() may return special dictionary for strings and dictionaries, though it is limited to only tables with string keys; as usual for msgpack dictionary keys are STR strings, all other strings are BIN strings).
  • (planned) luaeval("1.0") is 1. Vim: is 1.0 (lua numbers are always returned as floats).
  • (planned) luaeval("userdata") is an error. Vim: returns 0.
  • (planned) luaeval('true'), luaeval('false'), luaeval('nil') return corresponding v: values. Vim: returns v:true (same), v:false (same) and 0 (v:null).

Reworded planned incompatibilities:

  1. luaeval() may return integers for integral numbers (if they fit into varnumber_T, of course).
  2. luaeval() converts lua tables into VimL values.
  3. luaeval() converts nil to v:null.
  4. luaeval() does not support multiple returns.
  5. luaeval() supports special dictionaries: they are respected when encountered as a second argument and they may be returned under some circumstances.
  6. Trying to return a value that luaeval() cannot convert is an error.
ZyX-I added some commits Mar 4, 2016
@ZyX-I ZyX-I eval: Add luaeval function
No tests yet, no documentation update, no :lua* stuff, no vim module.

converter.c should also work with typval_T, not Object.

Known problem: luaeval("1", {}) results in

    PANIC: unprotected error in call to Lua API (attempt to index a nil value)

Ref #3823
cced40a
@ZyX-I ZyX-I viml/executor: Directly generate typval_T values
Note: this will *still* crash when using API in cases similar to the one
described in first commit. Just it needs different code to reproduce.
5bb4eb7
@ZyX-I ZyX-I *: Silence linter 3e57020
@ZyX-I ZyX-I executor: Make sure it works with API values 3ecf96c
@ZyX-I ZyX-I executor/converter: Fix conversion of self-containing containers 89e7811
@ZyX-I ZyX-I executor/converter: Fix how maxidx is determined cf43acb
@ZyX-I ZyX-I executor/converter: Make nlua_pop_Object not recursive acb008d
@ZyX-I ZyX-I executor/converter: Make it possible to supply `{}` to Dictionary arg 83563c4
@ZyX-I ZyX-I functests: Test for error conditions
During testing found the following bugs:

1. msgpack-gen.lua script is completely unprepared for Float values either in 
   return type or in arguments. Specifically:

   1. At the time of writing relevant code FLOAT_OBJ did not exist as well as 
      FLOATING_OBJ, but it would be used by msgpack-gen.lua should return type 
      be Float. I added FLOATING_OBJ macros later because did not know that 
      msgpack-gen.lua uses these _OBJ macros, otherwise it would be FLOAT_OBJ.
   2. msgpack-gen.lua should use .data.floating in place of .data.float. But it 
      did not expect that .data subattribute may have name different from 
      lowercased type name.

2. vim_replace_termcodes returned its argument as-is if it receives an empty 
   string (as well as _vim_id*() functions did). But if something in returned 
   argument lives in an allocated memory such action will cause double free: 
   once when freeing arguments, then when freeing return value. It did not cause 
   problems yet because msgpack bindings return empty string as {NULL, 0} and 
   nothing was actually allocated.
3. New code in msgpack-gen.lua popped arguments in reversed order, making lua 
   bindings’ signatures be different from API ones.
5a9a417
@ZyX-I ZyX-I executor/converter: Allow converting self-referencing lua objects 9139d59
@ZyX-I ZyX-I api: Reserve more numbers for internal calls
Reasoning; currently INTERNAL_CALL is mostly used to determine whether it is 
needed to deal with NL-used-as-NUL problem. This code is useful for nvim_… API 
calls done from VimL, but not for API calls done from lua, yet lua needs to 
supply something as channel_id.
ffc1db7
@ZyX-I ZyX-I executor/converter: Use readable lua numbers for handles 37e9c8b
@ZyX-I ZyX-I functests: Add some tests
a6a8508
@ZyX-I ZyX-I eval/decode: Fix memory leak in JSON functions 8aabe77
@ZyX-I ZyX-I genmsgpack: Include error source in error messages 63c8b02
@ZyX-I ZyX-I executor/executor: When reporting errors use lua string length ac45c13
@ZyX-I ZyX-I functests: Move existing tests from lua_spec to lua/*, fix them
a929299
@ZyX-I
Contributor
ZyX-I commented Jan 20, 2017

With latest commit I get the following error locally:

=================================================================
==6755==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fff902a1578 at pc 0x000000f54e0d bp 0x7fff902a00b0 sp 0x7fff902a00a8
READ of size 4 at 0x7fff902a1578 thread T0
    #0 0xf54e0c in vim_vsnprintf /home/zyx/a.a/Proj/c/neovim/src/nvim/message.c:3234:48
    #1 0xf5f3cd in emsgf /home/zyx/a.a/Proj/c/neovim/src/nvim/message.c:581:3
    #2 0x18cfa6f in nlua_error /home/zyx/a.a/Proj/c/neovim/src/nvim/viml/executor/executor.c:89:3
    #3 0x18cf423 in nlua_eval_lua_string /home/zyx/a.a/Proj/c/neovim/src/nvim/viml/executor/executor.c:236:5
    #4 0x7f12a08081f5 in _init (/usr/lib64/libluajit-5.1.so.2+0x91f5)
    #5 0x18cea3d in executor_eval_lua /home/zyx/a.a/Proj/c/neovim/src/nvim/viml/executor/executor.c:264:3
    #6 0x8f5c63 in f_luaeval /home/zyx/a.a/Proj/c/neovim/src/nvim/eval.c:12927:3
    #7 0x7ed480 in call_func /home/zyx/a.a/Proj/c/neovim/src/nvim/eval.c:7303:11
    #8 0x802c3a in get_func_tv /home/zyx/a.a/Proj/c/neovim/src/nvim/eval.c:7108:11
    #9 0x8a0889 in eval7 /home/zyx/a.a/Proj/c/neovim/src/nvim/eval.c:4354:15
    #10 0x89c275 in eval6 /home/zyx/a.a/Proj/c/neovim/src/nvim/eval.c:4074:7
    #11 0x896ef9 in eval5 /home/zyx/a.a/Proj/c/neovim/src/nvim/eval.c:3926:7
    #12 0x892101 in eval4 /home/zyx/a.a/Proj/c/neovim/src/nvim/eval.c:3667:7
    #13 0x8916a3 in eval3 /home/zyx/a.a/Proj/c/neovim/src/nvim/eval.c:3589:7
    #14 0x890c43 in eval2 /home/zyx/a.a/Proj/c/neovim/src/nvim/eval.c:3526:7
    #15 0x7e5632 in eval1 /home/zyx/a.a/Proj/c/neovim/src/nvim/eval.c:3459:7
    #16 0x85051e in ex_echo /home/zyx/a.a/Proj/c/neovim/src/nvim/eval.c:20273:9
    #17 0xb402cc in do_one_cmd /home/zyx/a.a/Proj/c/neovim/src/nvim/ex_docmd.c:2198:5
    #18 0xb1ee07 in do_cmdline /home/zyx/a.a/Proj/c/neovim/src/nvim/ex_docmd.c:601:20
    #19 0xb25115 in do_cmdline_cmd /home/zyx/a.a/Proj/c/neovim/src/nvim/ex_docmd.c:273:10
    #20 0xe16531 in exe_pre_commands /home/zyx/a.a/Proj/c/neovim/src/nvim/main.c:1566:7
    #21 0xe0863a in main /home/zyx/a.a/Proj/c/neovim/src/nvim/main.c:332:3
    #22 0x7f129fd5766f in __libc_start_main (/lib64/libc.so.6+0x2066f)
    #23 0x443d28 in _start (/home/zyx/a.a/Proj/c/neovim/build/bin/nvim+0x443d28)

Address 0x7fff902a1578 is located in stack of thread T0 at offset 376 in frame
    #0 0xf5f13f in emsgf /home/zyx/a.a/Proj/c/neovim/src/nvim/message.c:574

  This frame has 1 object(s):
    [32, 56) 'ap' <== Memory access at offset 376 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow /home/zyx/a.a/Proj/c/neovim/src/nvim/message.c:3234:48 in vim_vsnprintf
Shadow bytes around the buggy address:
  0x10007204c250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10007204c260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10007204c270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10007204c280: f1 f1 f1 f1 00 00 00 f3 f3 f3 f3 f3 00 00 00 00
  0x10007204c290: 00 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2
=>0x10007204c2a0: f2 f2 00 00 00 00 f2 f2 f2 f2 00 00 f3 f3 f3[f3]
  0x10007204c2b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10007204c2c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10007204c2d0: f1 f1 f1 f1 00 f3 f3 f3 00 00 00 00 00 00 00 00
  0x10007204c2e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10007204c2f0: 00 00 00 00 f1 f1 f1 f1 00 00 00 f3 f3 f3 f3 f3
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==6755==ABORTING

for nvim --cmd 'echo luaeval("vim.api.nvim_set_var(1, 2, 3)")' --cmd qa (basically works for errors like “Expected {less} arguments” if {less} is 2 or less: checked with a few functions).

valgrind sees nothing here.

@ZyX-I
Contributor
ZyX-I commented Jan 20, 2017

(valgrind run when compiled the same code with sanitizers disabled.)

ZyX-I added some commits Jan 20, 2017
@ZyX-I ZyX-I executor/executor: Free lcmd on error 858cecc
@ZyX-I ZyX-I functests: Some more tests
776d9a4
@ZyX-I
Contributor
ZyX-I commented Jan 20, 2017

Blacklisted vim_vsnprintf, got different error:

==7185==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffc5c883620 at pc 0x0000004d7ebc bp 0x7ffc5c8832d0 sp 0x7ffc5c882a80
WRITE of size 1032 at 0x7ffc5c883620 thread T0
    #0 0x4d7ebb in __asan_memset /var/tmp/portage/sys-devel/llvm-3.8.1-r2/work/llvm-3.8.1.src/projects/compiler-rt/lib/asan/asan_interceptors.cc:410
    #1 0x547e28 in nlua_msgpack__vim_id_float /home/zyx/a.a/Proj/c/neovim/build/src/nvim/auto/msgpack_lua_c_bindings.generated.c:1529:9
    #2 0x7ffaf76681f5 in _init (/usr/lib64/libluajit-5.1.so.2+0x91f5)
    #3 0x7ffaf7675e0e in lua_pcall (/usr/lib64/libluajit-5.1.so.2+0x16e0e)
    #4 0x18c9c10 in nlua_eval_lua_string /home/zyx/a.a/Proj/c/neovim/src/nvim/viml/executor/executor.c:238:7
    #5 0x7ffaf76681f5 in _init (/usr/lib64/libluajit-5.1.so.2+0x91f5)
    #6 0x18c927d in executor_eval_lua /home/zyx/a.a/Proj/c/neovim/src/nvim/viml/executor/executor.c:267:3
    #7 0x8f5c63 in f_luaeval /home/zyx/a.a/Proj/c/neovim/src/nvim/eval.c:12927:3
    #8 0x7ed480 in call_func /home/zyx/a.a/Proj/c/neovim/src/nvim/eval.c:7303:11
    #9 0x802c3a in get_func_tv /home/zyx/a.a/Proj/c/neovim/src/nvim/eval.c:7108:11
    #10 0x7fbbfd in ex_call /home/zyx/a.a/Proj/c/neovim/src/nvim/eval.c:2844:9
    #11 0xb402cc in do_one_cmd /home/zyx/a.a/Proj/c/neovim/src/nvim/ex_docmd.c:2198:5
    #12 0xb1ee07 in do_cmdline /home/zyx/a.a/Proj/c/neovim/src/nvim/ex_docmd.c:601:20
    #13 0x8532a3 in ex_execute /home/zyx/a.a/Proj/c/neovim/src/nvim/eval.c:20413:7
    #14 0xb402cc in do_one_cmd /home/zyx/a.a/Proj/c/neovim/src/nvim/ex_docmd.c:2198:5
    #15 0xb1ee07 in do_cmdline /home/zyx/a.a/Proj/c/neovim/src/nvim/ex_docmd.c:601:20
    #16 0xb25115 in do_cmdline_cmd /home/zyx/a.a/Proj/c/neovim/src/nvim/ex_docmd.c:273:10
    #17 0x655e63 in nvim_command /home/zyx/a.a/Proj/c/neovim/src/nvim/api/vim.c:45:3
    #18 0x59adb5 in handle_nvim_command /home/zyx/a.a/Proj/c/neovim/build/src/nvim/auto/api/private/dispatch_wrappers.generated.h:1670:3
    #19 0x1017c63 in on_request_event /home/zyx/a.a/Proj/c/neovim/src/nvim/msgpack_rpc/channel.c:455:19
    #20 0xa3f52f in multiqueue_process_events /home/zyx/a.a/Proj/c/neovim/src/nvim/event/multiqueue.c:146:7
    #21 0x10ed04f in nv_event /home/zyx/a.a/Proj/c/neovim/src/nvim/normal.c:7894:3
    #22 0x108f683 in normal_execute /home/zyx/a.a/Proj/c/neovim/src/nvim/normal.c:1143:3
    #23 0x1706033 in state_enter /home/zyx/a.a/Proj/c/neovim/src/nvim/state.c:58:26
    #24 0x104522b in normal_enter /home/zyx/a.a/Proj/c/neovim/src/nvim/normal.c:463:3
    #25 0xe0a212 in main /home/zyx/a.a/Proj/c/neovim/src/nvim/main.c:542:3
    #26 0x7ffaf6bb766f in __libc_start_main (/lib64/libc.so.6+0x2066f)
    #27 0x443d28 in _start (/home/zyx/a.a/Proj/c/neovim/build/bin/nvim+0x443d28)

Address 0x7ffc5c883620 is located in stack of thread T0 at offset 832 in frame
    #0 0x547cff in nlua_msgpack__vim_id_float /home/zyx/a.a/Proj/c/neovim/build/src/nvim/auto/msgpack_lua_c_bindings.generated.c:1528

  This frame has 1 object(s):
    [32, 1064) 'err' <== Memory access at offset 832 partially overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow /var/tmp/portage/sys-devel/llvm-3.8.1-r2/work/llvm-3.8.1.src/projects/compiler-rt/lib/asan/asan_interceptors.cc:410 in __asan_memset
Shadow bytes around the buggy address:
  0x10000b908670: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10000b908680: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10000b908690: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10000b9086a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10000b9086b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x10000b9086c0: 00 00 00 00[f2]f2 f2 f2 f2 f2 f2 f2 00 00 00 00
  0x10000b9086d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10000b9086e0: 00 f3 f3 f3 f3 f3 f3 f3 f3 f3 f3 f3 f3 f3 f3 f3
  0x10000b9086f0: f3 f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00
  0x10000b908700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10000b908710: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
@ZyX-I
Contributor
ZyX-I commented Jan 20, 2017

Note that last error points to line

  Error err = {.set = false};

. One can’t really expect stack overflow here. Maybe this has something to do with lua error handling? Though changing luajit to lua does not really fix things, only expands stack traces.

@ZyX-I ZyX-I cmake: Allow switching from luajit to lua
197fd3f
@ZyX-I
Contributor
ZyX-I commented Jan 21, 2017

If I build with lua which was also built with address sanitizer enabled then error does not occur.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment