[RDY] making the API callable from vimscript #4934

Merged
merged 12 commits into from Sep 1, 2016

Projects

None yet
@bfredl
Member
bfredl commented Jun 17, 2016 edited

ref #4091

Instead of generating a f_ wrapper per function, this implements a single api_wrapper function in analogy with float_op_wrapper. Extend the functions table to allow an extra argument to an implementation, this allows several builtin function to share a single implementation (as demonstrated this can be used in many cases for float_op_wrapper as well).

This reuses the handlers implemented for msgpack dispatch. This saves us from generating more code, but means the type checking will be more or less the same. A reason for generating separate f_vim_... wrappers could be to behave more like other builtins with automatic conversion to an int if an int is expected and such (so "somestring" becomes 0 without any error or warning). Though IMO this is not a problem as the names of these functions clearly stand out and hey proper typechecking is nice.

Still the functions need to be inserted in to the table of builtins. We could use #3922 for this to autogenerate this and sort it into the table by lua code.

Arguably buffer nrs should be identified with buffer "handles", otherwise would be a disaster. (It seems reasonable a nvim session would only have <2^31 buffer creation events but I could be wrong...) I would also suggest 0 being a special value always meaning curbuf/curwin etc so one don't need to do buffer_...(bufnr(''), ...) for a one-off api call.

@justinmk
Member

A reason for generating separate f_vim_... wrappers could be to behave more like other builtins with automatic conversion to an int if an int is expected and such

I think it's reasonable and even preferable that the API is a bit more rigid.

I would also suggest 0 being a special value always meaning curbuf/curwin

Can't think of a reason against.

I can't shake the idea that it might be useful to somehow namespace the API functions:

  • a global v:api dict (I don't like this idea), e.g. call v:api.vim_get_current_buffer()
  • nvim#vim_get_current_buffer
  • a new "scope style" qualifier, e.g. n:vim_get_current_buffer()
  • put the functions in v:, e.g. v:vim_get_current_buffer()

Maybe it's not useful though, and it's better that they are treated as first-class VimL functions. After all, VimL is itself an implicit namespace.

@justinmk
Member

Arguably buffer nrs should be identified with buffer "handles", otherwise would be a disaster. (It seems reasonable a nvim session would only have <2^31 buffer creation events but I could be wrong...)

Is this the reason for replace int64 handle with tp_id, etc.?

@marvim marvim added the WIP label Jun 17, 2016
@bfredl
Member
bfredl commented Jun 17, 2016

Is this the reason for replace int64 handle with tp_id, etc.?

Not really, I replaced buf->handle with buf->b_fnum and tried to be consistent, but I guess it didn't fall out too well. Another way is to #define b_fnum handle and then when porting 1557 we could just use win->handle instead of w_id internally.

I can't shake the idea that it might be useful to somehow namespace the API functions

I don't have any super strong opinion really, but for me vim_/buffer_ etc is already a "namespace". Remember vim has deprecatedbuffer_number() in favor for bufnr() etc so it seems vim will stay away from these sort of names. Relatedly I thought of blacklisting/whitelisting functions if we don't wan't to expose everyone, to reduce namespace clutter. We could disallow vim_eval() for instance as it is better to be explicit with deepcopy(eval(...)) (is more efficient as well).

@justinmk
Member

Remember vim has deprecatedbuffer_number() in favor for bufnr() etc so it seems vim will stay away from these sort of names

You're probably right, I'm overthinking it. The API methods will tend to be very verbose, which is unlikely to conflict with future vim functions. No extra namespace ceremony needed.

Relatedly I thought of blacklisting/whitelisting functions if we don't wan't to expose everyone, to reduce namespace clutter.

👍

@ZyX-I
Contributor
ZyX-I commented Jun 17, 2016

@bfredl vim_eval is not needed if you want to evaluate expression. But it is a perfect candidate to test how evaluation result would be received on remote end (without actually setting this remote end up). Also deepcopy(eval(…)) and vim_eval(…) have way too much differences in the current state:

  1. No funcrefs support.
  2. No recursive containers support.
  3. If one container is present twice (e.g. let l=[[]]|call add(l, l[0])) after vim_eval there will be two containers.

You're probably right, I'm overthinking it. The API methods will tend to be very verbose, which is unlikely to conflict with future vim functions. No extra namespace ceremony needed.

I would still go for api_*() “namespace”: there are much differences in how values are treated by regular and API functions, above is one of the examples. Such naming is also an explanation to “why there exists both eval() and vim_eval(), what’s the difference?”


I don't have any super strong opinion really, but for me vim_/buffer_ etc is already a "namespace". Remember vim has deprecatedbuffer_number() in favor for bufnr() etc so it seems vim will stay away from these sort of names. Relatedly I thought of blacklisting/whitelisting functions if we don't wan't to expose everyone, to reduce namespace clutter. We could disallow vim_eval() for instance as it is better to be explicit with deepcopy(eval(...)) (is more efficient as well).

Also remember that new Vim functions are named like json_decode() or ch_open(). Or win_getid(). Given that we are likely to merge win_getid() it is also likely that there will be a relevant call in API your variant will be confusing (win_getid() and window_get_id(), which one to use?). Existence of api_window_get_id() here does not look that confusing: one function is Neovim-specific API binding, another one is regular VimL function.

Also note that all variants listed after echo win<Tab> fit in one line and present me unique functions. What will this look like after adding a growing bunch of functions from api/window.c? Same for buf<Tab>.

And you completely forgot about tabpage: new Vim naming scheme is using underscores, but all existing tab* functions start with tabpage*, not tab* like win* or buf* like window- and buffer-related ones. So here naming conflict is a lot more likely.


Regarding function table: I would really suggest to turn this into a hash. I even started doing something like this, but stopped at problem “how to keep khash (or hashtab) reimplementation in lua consistent with it in C code”. Patch is still available: 821958a.

@bfredl
Member
bfredl commented Jun 17, 2016

But it is a perfect candidate to test how evaluation result would be received on remote end.

Good point.

And you completely forgot about tabpage

Not completely, it is still tabpagenr where the api name would be of the form tabpage_get_number.

For me names like buffer_add_highlight just sound more natural than api_buffer_add_highlight or api_buffer_set_lines. The plugin will add highlights and lines to the buffer, exactly what the name says. Speaking of tabbing, changes are the plugin writer would try buf[fer]<tab> for these operations and not api_buf<tab> . That's also why I mentioned blacklisting, we don't need to import every name into the "global" namespace if we are afraid of cluttering. (to avoid complete duplicates) Also window_get_id() won't exist with this design either as the handle would be the id.

@bfredl
Member
bfredl commented Jun 17, 2016

Regarding function table: I would really suggest to turn this into a hash. I even started doing something like this, but stopped at problem “how to keep khash (or hashtab) reimplementation in lua consistent with it in C code”.

I though of a hash too, but I thought instead of just initializing the hash with c code (then in principle we could modify msgpack_rpc_add_method_handler to do the part for API functions). I wonder if it really would bring significant time cost to the startup (given that we already build the rpc method table in this way).

@ZyX-I
Contributor
ZyX-I commented Jun 17, 2016

@bfredl Currently all tabpage* functions have no underscore. But just as well all new functions have underscore like win_getid (older functions look like winheight()), so the point is that name conflict is a lot more likely in the future, not that it is possible right now.

@ZyX-I
Contributor
ZyX-I commented Jun 17, 2016

@bfredl I do not think this will bring significant cost to the startup, but neither I think that increasing time and memory usage while there is such an easy optimization available is a good idea. Only problem there was “rewriting khash.h in lua” vs “making cross-compiling impossible” vs “adding gperf as a dependency”. Do not remember why I did not post this as a PR with this question: probably 1 or 3 would be accepted.

@bfredl
Member
bfredl commented Jun 18, 2016

@ZyX-I Ok, I went with api_ prefix and I cherry-picked your functions hash branch. I wonder, can't we just require that on cross-compiling the host c compiler is available as well? (Mean, neovim is certainly not the first project that wants to support cross-compiling and also use host c as part of the build process)

Still a lot of expected warnings I hope to go though this weekend (mostly conversion and add dummy void * arguments) but should be working for basic testing.

@ZyX-I
Contributor
ZyX-I commented Jun 18, 2016

I currently do not know how to do this with cmake. Using gperf seems to be easier, AFAIK it should be available everywhere we currently want to build (linux, windows and mac os).

@bfredl
Member
bfredl commented Jun 18, 2016

Yes gperf looks what we want to use, but even then we might want to build it as a dependency when building in USE_BUNDLED mode as we do for other dependencies. It seems our third-party cmakelists already defines HOSTDEPS_C_COMPILER for this purpose.

@justinmk
Member
justinmk commented Jun 18, 2016 edited

Couldn't we commit the result of gperf, instead of requiring gperf as a dependency?

We could automate this in travis:

  • when travis builds any PR, it runs gperf
  • if the result is different than the current source, post it to gist and leave a comment on the PR
    • alternative: leave a comment with instructions for the developer to run gperf locally
  • developer manually commits the result and updates the PR
@bfredl
Member
bfredl commented Jun 18, 2016

We could, but it seems to just be more complicated for the developer, for no real benifit. We are already requiring (host) lua to be present for building nvim, and building it and running it automatically for the developer, why can't we do the same with gperf? Now, I could imagine shipping the gperf output as part of release tarballs, as the expectaction there is that no modification will be made to the code.

@justinmk
Member

We are already requiring (host) lua to be present for building nvim, and building it and running it automatically for the developer, why can't we do the same with gperf?

I don't feel strongly about it, but gperf is a bit more obscure, and our build stack already gets plenty of complaints. Lua doesn't seem comparable: we will eventually ship lua to the end user, so it's not only a build-time dependency.

@bfredl
Member
bfredl commented Jun 18, 2016 edited

we will eventually ship lua to the end user,

As I suggested neither would end user nor distro people (the ones mostly complaining of our build system, I would presume) need to have gperf if we ship the result as part of the release tarballs, so gperf won't either be a pure bulid-time dependency. So the situation do seem comparable with lua generation to me.

It would only affect the developers situation, and having a small c utility built for you (after a bunch of other c dependencies) seems less burdensome to me than requesting the developer to copy some text from a gist for some reason or getting a weird error message suggesting the developer to build and run gperf themselves.

@bfredl
Member
bfredl commented Jun 18, 2016

I guess my main point is, the developer does not need to care at all about most of our auto-generating, say, for internal prototypes and the option table, but if the developer happens to change api prototypes now the developer is suddenly forced to care a lot of our process. At least to me this inconsistency would seem a bit odd and hard to explain.

@bfredl
Member
bfredl commented Jun 26, 2016

I made an attempt of building gperf only if it's needed. The generated hash could be shipped in a release tarball by
cp build/src/nvim/auto/funcs.generated.h src/auto/
which will turn off building gperf. Now the logic is quite ad-hoc (partially as "deps" and "nvim" are two completely separate cmake domains) but this could be extended to cache other generated source or I could imagine enabling shipping a completely static source distribution by cp -r build/src/nvim/auto/ src/auto/ (if we find a reason for it that is)

@bfredl bfredl changed the title from [WIP] making the API callable from vimscript to [RFC] making the API callable from vimscript Jun 28, 2016
@bfredl
Member
bfredl commented Jun 28, 2016

The feature itself is more or less "done", so marking RFC.
Some gaps:

  • check if building gperf on mingw works
  • building gpref on msvc: I have no idea what to do here, some windows expert might want to jump in (see third-party/cmake/BuildGperf.cmake)
  • could use a bit more testing
  • make api docs accessible from nvim. doxygen exports machine readable xml, we "just" need to convert it to something reasonably ft=help compatible. (but probably this should be a separate PR)
@marvim marvim added RFC and removed WIP labels Jun 28, 2016
@justinmk
Member
justinmk commented Jun 30, 2016 edited

I don't want to hold this up, but something that's bugging me is: I can see the API functions becoming a very common part in vimscripts, and also a way for us to add new functions/features without worrying about conflict with Vim. So the api_ prefix might not be the best option.

For a followup PR would anyone be opposed to putting the API functions in the v: "scope" (which currently doesn't have any functions--and it's probably guaranteed that there never will be any, because Vim would never have a reason to put functions in there).

@bfredl
Member
bfredl commented Jun 30, 2016

It seems a lot of work to bend the parsing rules (or introduce some special kind of Funcref) just to save 2 characters.
Nothing stops us from, in addition to the api_ names, whitelisting a few select functions implementing novel features into the global namespace like buffer_add_highlight (or even give them vimLesque names such as bufhladd but then people would expect to find them in eval.txt and have old-style conversion semantics, so it's probably a bad idea). The risk for conflict or namespace bloat is much less than if importing all functions without prefix.

@ZyX-I
Contributor
ZyX-I commented Jun 30, 2016

@justinmk If you use n: I would expect n: scope dictionary to exist, probably with the same semantic as v:, but additionally marking variable as “Neovim-specific, not expected to appear in Vim”. It needs not to contain funcrefs though: g: and s: function “scopes” do not add anything to the relevant scopes. But still I do not see much sense in further breaking naming scheme: current one is in line with recent upstream (with exception that upstream prefers win and buf over full window and buffer), n: or v: is something rather exceptional, yet telling nothing about what is exceptional about this (api_ is rather self-explanatory reminder for everybody who knows about msgpack-rpc API, n: is not, not to mention v:).

@ZyX-I
Contributor
ZyX-I commented Jun 30, 2016

By the way, how about a few automatic renamings: api_window_… to api_win_… and api_buffer_… to api_buf_…? Less to type, still no name conflicts due to api_… prefix and more in line with Vim upstream which uses abbreviations.

@justinmk
Member

whitelisting a few select functions implementing novel features into the global namespace like buffer_add_highlight

I would strongly prefer to avoid special cases. If there's opposition to v: then it's fine, consider my proposal retracted :) Thanks for the feedback.

By the way, how about a few automatic renamings: api_window_… to api_win_… and api_buffer_… to api_buf_…

I'd suggest renaming the API functions and deprecating the old ones. Avoids ambiguity and extra layers/magic for developers to grok.

@bfredl
Member
bfredl commented Jun 30, 2016

So if I understand correctly:

  • rename the API functions to new names buf_...(accessible as api_buf_... from vimL, or should the C functions be renamed to this as well?)
  • allow buffer_... as a deprecated alias from msgpack requests, maybe also in the metadata (which could learn a deprecatedor even deprecated_since field, which be a api version number bumped every time we deprecate or remove something).

An alternative naming could be nvim_buf_set_lines if the main point for the vimscript developer is that the function is nvim-specific (which should be enough to signify a change in conversion semantic etc) rather than it also being accessible from msgpack. nvim_ prefix sounds also like a good convention for a libnvim C api.

@justinmk
Member
justinmk commented Jun 30, 2016 edited

rename the API functions to new names buf_...(accessible as api_buf_... from vimL, or should the C functions be renamed to this as well?)

IMO any rename of the API functions should be reflected everywhere, including the C functions.

allow buffer_... as a deprecated alias from msgpack requests, maybe also in the metadata (which could learn a deprecatedor even deprecated_since field, which be a api version number bumped every time we deprecate or remove something).

👍 separate PR for deprecated_since?

An alternative naming could be nvim_buf_set_lines if the main point for the vimscript developer is that the function is nvim-specific

The name api_ wasn't really my complaint, but just n_ could be an improvement. Actually, I like n_...

@bfredl
Member
bfredl commented Jul 9, 2016

I tried to do something consistent and simple, so the last commit implements using "nvim_"/"nvim_buf" everywhere (c api/metadata/msgpack-rpc/eval).The prefix might seem redundant is some places, but I like functions having a single canonical name to refer to in documentation/discussions etc (well + the deprecated aliases in msgpack which will need to stay for quite some time).

@justinmk
Member
justinmk commented Jul 9, 2016 edited

It's hard to argue with that reasoning. This might be a good time to mention I think we should rename the project to Ni.

Even if there's a large opposition to renaming "Neovim" the project, I think the "nvim" abbreviation should be renamed everywhere (including the nvim binary) to "ni". Because it's shorter, it's unique, it's fewer syllables, and many other reasons.

@bfredl
Member
bfredl commented Jul 9, 2016

I think the "nvim" abbreviation should be renamed everywhere (including the nvim binary) to "ni".

Not sure what I think of this, but it would be a good opportunity to really test practical use of the livesub feature.

@ZyX-I
Contributor
ZyX-I commented Jul 9, 2016 edited

What about keeping deprecated functions untouched? If I am not mistaking they are to be removed in some future release in any case, and they definitely do not need a new name that may be called remotely.

@bfredl
Member
bfredl commented Jul 9, 2016

@ZyX-I yes I planned to to this, it was mostly out of lazyness to not fix this (or documentation for that matter) until I got some feedback on the renaming.

@bfredl
Member
bfredl commented Jul 9, 2016

also for that matter we could deprecate vim_*_current_line now as these can be done in one request using the "0" value for curbuf.

@ZyX-I
Contributor
ZyX-I commented Jul 9, 2016 edited

And not renaming them also removes the need for FUNC_API_NOEVAL: script just needs to check whether function name starts with nvim_.

@bfredl
Member
bfredl commented Jul 9, 2016

Not completely, there are some other functions like ui_ and get_api_info that also are NOEVAL (we can't really use channel_id for this as it could be useful for some functions to regard eval as a separate "client", or get_lines that uses it to handle the NUL vs NL special case).
If we special case deprecated functions, nvim_ could be used to distinguish exported functions from NOEXPORT, however.

@bfredl
Member
bfredl commented Jul 9, 2016

Also I don't really see the point of the ni rename. "nvim" is already short and one syllable (at least in my pronunciation). And I've grown to like the "Neovim is the project/general concept, nvim is the product/binary/individual instance" convention (import neovim; nvim = neovim.attach(...) in python and similar clients makes a lot of sense, at least)
Anyway if it is a serious proposal it needs its own dedicated issue.

@justinmk
Member
justinmk commented Jul 9, 2016 edited

It does need a separate issue, reason I mentioned it here is because we wouldn't want to rename things more than once.

"ni" is as you noted a monty python reference (as well as an ongoing pun in Vim's doc and source), means "not implemented", and also happens to look like "vi" upside down. 2 letters is better than 4. And our constant mention of "vim" in our name I think appears a bit combative ("new vim"), as well as being pretty cliche (cf. "nvi"). Not to mention, as superficial as it might seem, users will love typing "ni" instead of "nvim", I promise.

@bfredl
Member
bfredl commented Jul 9, 2016 edited

So you would suggest this PR should just move ahead of the general rename and use ni_ as the API prefix?

@justinmk
Member
justinmk commented Jul 9, 2016

@bfredl That is what I would prefer. I am expecting pushback on the rename, I will of course commit to dealing with the churn. In the meantime ni_ is a reasonable prefix even if the general rename is blocked for some reason.

@ZyX-I
Contributor
ZyX-I commented Jul 9, 2016

@justinmk Current convention is that technical abbreviation is nvim, full name is Neovim. And I am against renaming anything to ni, the less letters the less it is possible to find anything with Google. “Ni” is “national instruments”, “nvim” is unique and search services are fine with it.

@ZyX-I
Contributor
ZyX-I commented Jul 9, 2016

@justinmk Also if general rename is “blocked” then ni must not be used as well, it is unconventional.

@justinmk
Member
justinmk commented Jul 9, 2016

@ZyX-I most users search for "neovim", which we will continue to route from neovim.io, etc. "ni editor" would be searchable without issue, just like "vi editor", "sublime editor", "xi editor", etc.

It's definitely important to be searchable, but this should not block improving the user experience for new installs as well as people using the editor every day. Defaults and naming are far more important in fact.

@justinmk
Member
justinmk commented Jul 9, 2016

Also if general rename is “blocked” then ni must not be used as well, it is unconventional.

Fine. There are strong reasons in favor of a rename, chief among them is that "new vim" is a very weak name. Naming is important. Defaults are important. Marketing is important. Though I don't find any joy in it, someone needs to think about it (marketing) irrespective of the discomfort of those who pretend it's not important. Thinking about these unpleasant things is the job of a maintainer. I'm the maintainer unless someone else wants to do the boring work. I think we should rename "nvim" unless there is a strong argument against it (churn/migration is not strong, it's an implementation detail that I'll take ownership of.)

@ZyX-I
Contributor
ZyX-I commented Jul 9, 2016

@justinmk For any user that can seriously consider using Neovim difference between typing ni and typing nvim is insignificant. Also nvim is “new Vim”, ni is “something negative”. You mentioned “not implemented” abbreviation, Russian language has a particle or conjunction with also somewhat negative meaning (hard to explain to non-Russian speaker).

Also Google with “ni editor” is not the only place where ni may be searched: e.g. in shell’s <C-r> it is easier to find “some complex command where I used nvim editor” then same thing with “ni”.

Fine. There are strong reasons in favor of a rename, chief among them is that "new vim" is a very weak name. Naming is important. Defaults are important. Marketing is important. Though I don't find any joy in it, someone needs to think about it (marketing) irrespective of the discomfort of those who pretend it's not important. Thinking about these unpleasant things is the job of a maintainer. I'm the maintainer unless someone else wants to do the boring work. I think we should rename "nvim" unless there is a strong argument against it (churn/migration is not strong, it's an implementation detail that I'll take ownership of.)

If you want to alter the convention it should be a separate issue, here it is unconventional. Also there were only three opinions expressed here and most were negative.

“New vim” applies to “Neovim” in first place, there is absolutely no point in renaming “nvim” to “ni” to avoid this association. “Ni” name for me is in first place “something negative” as I said earlier. In the second place it associates with “vi” which is even worse because my associations here are “hard to use (unconfigured) vim” (sometimes I happened to work with vim in nocompatible mode) and “editor from the past era”. It does not look like ni here (from #4934 (comment)) is something positive as well.

@justinmk
Member
justinmk commented Jul 9, 2016 edited

The monty python reference (the video you linked) is whimsical. <c-r>ni<space> is probably just as precise in readline as <c-r>nv. I think we should rename the entire project, but renaming the nvim abbreviation is just a first (major step). I am sure you could quickly overcome whatever brief negative association you have. Whereas it will be difficult to live with the name "neovim" forever.

I'll be very unhappy if this is blocked. Will you be very unhappy if it is not?

@bfredl
Member
bfredl commented Jul 9, 2016 edited

I was almost going to suggest nv (which happens to be one of my zsh aliases) but /usr/bin/nv already exists in search results and is probably too close to nvi editor. An option could be to use nv_ as an api prefix (and possibly other internal prefixes) as nvim degrades more easily to nv than to ni (so it wouldn't be as much trouble if they end up coexisting), but then again I'm personally fine with names likenvim_buf_set_lines. Two chars doesn't really save a lot there IMO.

@bfredl
Member
bfredl commented Jul 9, 2016

I'll be very unhappy if this is blocked. Will you be very unhappy if it is not?

This PR can wait a day or two or three until #5028 gets some kind of conclusion. No need to rush.

@justinmk
Member
justinmk commented Jul 9, 2016

An option could be to use nv_ as an api prefix (and possibly other internal prefixes) as nvim degrades more easily to nv than to ni

@ZyX-I Is right that it should be consistent with the current convention. I created #5028 to get a feel for whether there is any interest in a rename. If there isn't then at least I can stop thinking about it :)

@justinmk
Member

Rename issue was settled, so prefix should stay at nvim_.

@bfredl
Member
bfredl commented Jul 13, 2016

This is feature-wise starting to get quite ready, apart from a bit of linting and cleanup, and a bit more tests, the main blocker is still compiling gperf on Windows. (to remind, gperf will not needed to build release tarballs, just for building master). As gperf is basically a bunch of ANSI C files with no dependencies beyond libc it shouldn't be too complicated, but it would probably be a lot easier for someone with knowledge of our windows build. Ping @equalsraf ?

@equalsraf equalsraf commented on an outdated diff Jul 13, 2016
third-party/cmake/BuildGperf.cmake
+ CONFIGURE_COMMAND "${_gperf_CONFIGURE_COMMAND}"
+ BUILD_COMMAND "${_gperf_BUILD_COMMAND}"
+ INSTALL_COMMAND "${_gperf_INSTALL_COMMAND}")
+endfunction()
+
+set(GPERF_BUILDARGS CC=${HOSTDEPS_C_COMPILER} LD=${HOSTDEPS_C_COMPILER})
+
+if(UNIX OR (MINGW AND CMAKE_CROSSCOMPILING))
+
+ BuildGperf(
+ CONFIGURE_COMMAND ${DEPS_BUILD_DIR}/src/gperf/configure
+ --prefix=${HOSTDEPS_INSTALL_DIR}
+ INSTALL_COMMAND ${MAKE_PRG} install)
+
+elseif(MSVC)
+ message(FATAL_ERROR "TODO")
@equalsraf
equalsraf Jul 13, 2016 Contributor

Can skip the MSVC case, its just better to use the regular error message

@equalsraf
Contributor

There are a couple of ways to handle gperf. MSYS has gperf packages and if we are only depending on the gperf executable we can probably grab a pre built binary from somewhere else. The best we can do for now is either to disable the the bundled gperf in windows or document -DUSE_BUNDLED_GPERF=OFF for windows.

@bfredl
Member
bfredl commented Jul 14, 2016

So, for this PR, when building master with MSVC require gperf.exe to be in PATH, and then later look into auto-downloading a prebuilt binary if needed?

@equalsraf
Contributor

@bfredl precisely

@fwalch
Member
fwalch commented Jul 15, 2016 edited

Would it make sense to add some Travis job that runs when new tags are created on the repo and creates a release tarball/uploads it to Github automatically? If yes, I can look into it.

OTOH, maybe we can just forgo that for now? It would be nicer to not have a separate release tarball.. Having an additional build-time dependency on gperf doesn't seem like such a big deal to me, especially since it is at least in the Ubuntu/Debian/Arch Linux repos.

@bfredl
Member
bfredl commented Jul 15, 2016

hooray, gperf generates C code that is invalid to clang per default...

@bfredl
Member
bfredl commented Jul 16, 2016

Close to passing the tests now... I would hope to get this merged soon, as the rebase will conflict with any change to a vimscript builtin or API function addition or signature change. Any review would be appreciated.

Also the systematic rename is an opportunity to fix/change the signature of a API function without breaking compatibility with existing 0.1.4 compatible clients. After this PR is merged, but before 0.1.5 is released, make the new signature available under nvim_[buf_] name and vim_/buffer_ with the old one. (the infrastructure will automatically mark any function not starting with nvim_ as deprecated). I will do this for ui_attach in #4432 .

@brcolow brcolow commented on an outdated diff Jul 16, 2016
runtime/doc/api.txt
@@ -14,13 +14,13 @@ C API for Nvim *API* *api*
==============================================================================
1. Introduction *api-intro*
-Nvim exposes a public API for external code to interact with the Nvim core. In
-the present version of Nvim the API is primarily used by external processes to
-interact with Nvim using the msgpack-rpc protocol, see |msgpack-rpc|. The API
-will also be used from vimscript to access new Nvim core features, but this is
-not implemented yet. Later on, Nvim might be embeddable in C applications as
-libnvim, and the application will then control the embedded instance by
-calling the C API directly.
+Nvim exposes a public API for external code to interact with the Nvim core.
+The API is used by external processes to interact with Nvim using the
+msgpack-rpc protocol, see |msgpack-rpc|. The API is used from vimscript to
+access some new Nvim core features. See |eval-api| for how api functions is
@brcolow
brcolow Jul 16, 2016 Contributor

api functions is -> api functions are

@brcolow brcolow commented on an outdated diff Jul 16, 2016
runtime/doc/api.txt
@@ -14,13 +14,13 @@ C API for Nvim *API* *api*
==============================================================================
1. Introduction *api-intro*
-Nvim exposes a public API for external code to interact with the Nvim core. In
-the present version of Nvim the API is primarily used by external processes to
-interact with Nvim using the msgpack-rpc protocol, see |msgpack-rpc|. The API
-will also be used from vimscript to access new Nvim core features, but this is
-not implemented yet. Later on, Nvim might be embeddable in C applications as
-libnvim, and the application will then control the embedded instance by
-calling the C API directly.
+Nvim exposes a public API for external code to interact with the Nvim core.
+The API is used by external processes to interact with Nvim using the
+msgpack-rpc protocol, see |msgpack-rpc|. The API is used from vimscript to
+access some new Nvim core features. See |eval-api| for how api functions is
+called from vimscript. Later on, Nvim might be embeddable in C applications as
+libnvim, and the application will then control the embedded instance by calling
+the C API directly
@brcolow
brcolow Jul 16, 2016 Contributor

Missing period at end of "directly"

@brcolow brcolow and 1 other commented on an outdated diff Jul 16, 2016
runtime/doc/api.txt
@@ -91,10 +91,19 @@ functions in the python client:
buf.clear_highlight(src)
<
If the highlights don't need to be deleted or updated, just pass -1 as
-src_id (this is the default in python). |buffer_clear_highlight| can be used
+src_id (this is the default in python). |nvim_buf_clear_highlight| can be used
to clear highligts from a specific source, in a specific line range or the
@brcolow
brcolow Jul 16, 2016 Contributor

typo: "highligts" -> "highlights"

Note: not changed by you, but just saw it.

@bfredl
bfredl Jul 17, 2016 Member

Well, this was written by me earlier :P

@brcolow brcolow commented on an outdated diff Jul 16, 2016
runtime/doc/api.txt
@@ -91,10 +91,19 @@ functions in the python client:
buf.clear_highlight(src)
<
If the highlights don't need to be deleted or updated, just pass -1 as
-src_id (this is the default in python). |buffer_clear_highlight| can be used
+src_id (this is the default in python). |nvim_buf_clear_highlight| can be used
to clear highligts from a specific source, in a specific line range or the
entire buffer by passing in the line range 0, -1 (the later is the default
@brcolow
brcolow Jul 16, 2016 Contributor

"later" -> "latter"

@brcolow brcolow commented on an outdated diff Jul 16, 2016
runtime/doc/api.txt
to clear highligts from a specific source, in a specific line range or the
entire buffer by passing in the line range 0, -1 (the later is the default
in python as used above).
+An example calling the raw api from vimscript: >
@brcolow
brcolow Jul 16, 2016 Contributor

"An example" -> "An example of" sounds more natural IMHO

@brcolow brcolow commented on an outdated diff Jul 16, 2016
runtime/doc/msgpack_rpc.txt
So for an object-oriented language, an API client contains the classes
representing Nvim special types, and the methods of each class could be
-defined by inspecting the method name prefix. There could also be a singleton
-Vim class with methods mapped to functions prefixed with `vim_`.
+defined by stripping the prefix for the type as defined in the `types` metadata
+(this will always be the first two "_"-separated parts of the function name).
+There could also be a singleton Vim class with methods where the `nvim_`
+prefix is stripped of.
@brcolow
brcolow Jul 16, 2016 Contributor

"of" -> "off"

@bfredl
Member
bfredl commented Jul 17, 2016

Hmm, it seems the runtime is not built in quickbuild. @jszakmeister would you mind adding "runtime" to the list of targets in the "configure-neovim-and-build-nvim" step?

@bfredl
Member
bfredl commented Jul 19, 2016

All tests are now passing. (except that the travis mac build hangs 50% of the attempts, but this seems to be the behavior also on master, and thus unrelated)

@bfredl bfredl referenced this pull request Aug 17, 2016
Merged

[RDY] Incsub 3 #5226

@bfredl
Member
bfredl commented Aug 22, 2016 edited

Updated with new function entries, and window ids (#5117) unified with api window handles.

@bfredl
Member
bfredl commented Aug 22, 2016

Hmm, without adding C:\msys64\usr\bin\ to PATH gperf won't be found, but with it in PATH (latest commit) "MinGW make" get's very upset sh.exe can be found in PATH and says "MSYS makefiles" should be used then. (Not that understand why, if no sh should be in PATH then no one could depend on the sh in the PATH in the first place). And there seems to not be any gperf package in the mingw-w64-x86_64 prefix (not that I understand what sets them apart from "global" msys64 packages). Thoughts @equalsraf ?

@equalsraf
Contributor

The mingw-w64 packages are native windows packages while the msys2 ones from /usr/bin link against the the magic msys UNIX adaptation layer from msys-2.0.dll

Strangely, there is a mingw-w64-gperf package in the source repos, but no package is available via pacman. I've tried building it myself and it is failing to build. Opened an issue upstream Alexpux/MINGW-packages#1653

The whole sh.exe issue is because cmake or mingw32-make wont run at all if sh.exe in the path.

Quick fix would be to try and use the regular gperf package, either

  • pass gperf path when calling cmake, GPERF_PRG (??)
  • or since adding its folder to the path is not an option, we need some type of indirection, a batch file that calls gperf.exe or a junction (maybe)

A bat file would be

c:\msys64\usr\bin\gperf.exe %*
@bfredl
Member
bfredl commented Aug 22, 2016

Thanks, let's wait that issue a bit.

@bfredl
Member
bfredl commented Aug 24, 2016

msys-gperf complains about the input even though the input is identical (the header at least, where the complaint is) and the gperf version exactly the same :(

@equalsraf
Contributor

I'm seeing an error here, junk on line 2. Its the line terminators, its probably using \r\n instead of \n and gperf dislikes it.

Opening the generated header and saving with ff=unix fixes that one for me.

@bfredl
Member
bfredl commented Aug 24, 2016

because luajit is "native" windows app and uses \r\n and gperf is "msys" thus "POSIX" and uses \n? I guess we can tell luajit to operate in binary mode and avoid emitting "junk".

@bfredl
Member
bfredl commented Aug 25, 2016

yay, looks like it is building on all platforms now. I don't think there is any outstanding issues. As I said, while vim_ interface should remain compatible, we can do minor changes to nvim_ in the 0.1.6 cycle after this PR.

@justinmk
Member
justinmk commented Aug 26, 2016 edited

Ok, so we can potentially rename nvim_* methods as discussed in #5031 (comment) after merging this.

@equalsraf
Contributor

@bfredl .ci/msys_*.bat were moved into a single .ci/build.bat

@timeyyy timeyyy commented on the diff Aug 26, 2016
src/nvim/buffer_defs.h
@@ -936,8 +934,8 @@ struct matchitem {
* All row numbers are relative to the start of the window, except w_winrow.
*/
struct window_S {
- uint64_t handle;
- int w_id; ///< unique window ID
+ handle_T handle; ///< unique identifier for the window
+
@timeyyy
timeyyy Aug 26, 2016 edited

You forgot to delete the existing buf_T b_fnum below

@bfredl
bfredl Aug 27, 2016 Member

it is deleted but in struct file_buffer.

@timeyyy
timeyyy Sep 1, 2016

do you mean in another commit? When i tried compiling it was complaining about b_fnum being defined twice.

@bfredl
bfredl Sep 1, 2016 Member

You mean the PR itself, not when basing extmark on it? Does rm -rf build help?

@timeyyy
timeyyy Sep 1, 2016

When I based extmarks on it.

@bfredl
bfredl Sep 1, 2016 Member

might be rebase error in extmark then. I'll merge this, please push rebased extmark then and I'll look into it.

@fwalch fwalch and 1 other commented on an outdated diff Aug 27, 2016
CMakeLists.txt
@@ -407,6 +407,10 @@ endif()
find_program(LUACHECK_PRG luacheck)
+if(NOT GPERF_PRG)
@fwalch
fwalch Aug 27, 2016 Member

Can be just:

find_program(GPERF_PRG gperf)

If the variable is already set, it won't be overwritten by find_program.

@bfredl
bfredl Aug 27, 2016 edited Member

The documentation says it will be set to <VAR>-NOTFOUND but I will try and see what happens...

@fwalch
fwalch Aug 27, 2016 edited Member

result is stored in the variable and the search will not be repeated unless the variable is cleared [..] If nothing is found, the result will be -NOTFOUND

Should be fine..

@fwalch fwalch commented on an outdated diff Aug 27, 2016
src/nvim/CMakeLists.txt
@@ -208,6 +225,21 @@ add_custom_command(OUTPUT ${GENERATED_EX_CMDS_ENUM} ${GENERATED_EX_CMDS_DEFS}
DEPENDS ${EX_CMDS_GENERATOR} ${EX_CMDS_DEFS_FILE}
)
+if (EXISTS "${GENERATED_FUNCS_CACHED}")
+ list(APPEND NEOVIM_GENERATED_SOURCES
+ "${GENERATED_FUNCS_CACHED}")
+else()
+ add_custom_command(OUTPUT ${GENERATED_FUNCS} ${FUNCS_DATA}
@fwalch
fwalch Aug 27, 2016 edited Member

Add the following before this?

if(NOT GPERF_PRG)
  message(FATAL_ERROR "gperf was not found.")
endif()
@bfredl
Member
bfredl commented Aug 28, 2016

As there are working gperf packages on all platforms we support I agree with @fwalch before, we can just skip the caching of gperf output. We probably will get more complaints from distro people for the release tarball being subtly different from the git tag (this happened in the python-client at least...), than for just needing to adding "gperf" in the "makedepends" field.

ZyX-I and others added some commits Jan 1, 2016
@ZyX-I @bfredl ZyX-I eval: Move VimL functions list to a lua file
Removes all kinds of problems with sorting, provides a ready-to-use function
list representation for genvimvim.lua, does not require specifying function name
twice (VimL function name (string) + f_ function name).
8fb7273
@ZyX-I @bfredl ZyX-I eval: Use generated hash to look up function list
Problems:
- Disables cross-compiling (alternative: keeps two hash implementations which
  need to be synchronized with each other).
- Puts code-specific name literals into CMakeLists.txt.
- Workaround for lua’s absence of bidirectional pipe communication is rather
  ugly.
5e59916
@bfredl bfredl api: rename "msgpack_rpc/defs.h" to "api/private/dispatch.h" and use …
…the header generator.
de3a515
@bfredl bfredl eval: add new function entries abe00d5
@bfredl bfredl api: auto generate api function wrappers for viml 3bd3b3b
@bfredl bfredl api: When calling get/set_lines from vimL, don't convert between "\n"…
… and "\0".
87e054b
@bfredl bfredl api: unify buffer numbers and window ids with handles
also allow handle==0 meaning curbuf/curwin/curtab
a2d25b7
@bfredl bfredl eval: use gperf to generate the hash of builtin functions
make api functions highlighted as builtins in vim.vim
7e2348f
@bfredl bfredl api: Allow blacklist functions that shouldn't be accesible from eval
Blacklist deprecated functions and functions depending on channel_id
e536abc
@bfredl bfredl api: consistently use nvim_ prefix and update documentation 1c22cab
@bfredl bfredl api: add tests for calling the api from vimscript 98a08c3
@bfredl bfredl api: fix leak when a api function is incorrectly called with a list.
This applies both to msgpack-rpc and eval.
acb7c82
@bfredl
Member
bfredl commented Aug 31, 2016

Marking as RDY, if no more comments of the code I will merge this, as there are a few PRs wanting to add API functions. (docs I'm not happy with, but as said that will be another PR with auto generating docs for methods, which also will make it easier to reason the surrounding docs)

@bfredl bfredl changed the title from [RFC] making the API callable from vimscript to [RDY] making the API callable from vimscript Aug 31, 2016
@marvim marvim added RDY and removed RFC labels Aug 31, 2016
@oni-link oni-link commented on an outdated diff Sep 1, 2016
src/nvim/lib/khash.h
-#define kmalloc(Z) xmalloc(Z)
-#endif
-#ifndef krealloc
-#define krealloc(P,Z) xrealloc(P,Z)
-#endif
-#ifndef kfree
-#define kfree(P) xfree(P)
+#ifdef USE_LIBC_ALLOCATOR
+# ifndef kcalloc
+# define kcalloc(N,Z) calloc(N,Z)
+# endif
+# ifndef kmalloc
+# define kmalloc(Z) malloc(Z)
+# endif
+# ifndef krealloc
+# define krealloc(P,Z) realloc(P,Z)
@oni-link
oni-link Sep 1, 2016 Contributor

As a note:
There is a difference in behavior for xrealloc(not NULL ,0) and realloc(not NULL, 0).
realloc() acts here as free(), xrealloc() will always allocate at least one byte.

@bfredl bfredl merged commit c6ac4f8 into neovim:master Sep 1, 2016

4 checks passed

QuickBuild Build pr-4934 finished with status SUCCESSFUL
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
coverage/coveralls Coverage increased (+0.09%) to 70.413%
Details
@jszakmeister jszakmeister removed the RDY label Sep 1, 2016
@justinmk
Member
justinmk commented Sep 1, 2016

Thanks for this, @bfredl

@piotrpalek

I came here from twitter (no idea about neovim codebase) and just noticed that wordcount doesn't have and equivalent entry in the lua file.. just saying in case this is a typo ¯_(ツ)_/¯

Member

Sorry about that. It was easier to rebase this thing and keeping track by having the corrections separately, but I guess I should have squashed right before merging...

@justinmk justinmk added a commit to justinmk/neovim that referenced this pull request Oct 27, 2016
@justinmk justinmk NVIM v0.1.6
FEATURES:
0b5a7e4 #4432 API: external UIs can render custom popupmenu
c6ac4f8 #4934 API: call any API method from vimscript
31df051 #4568 API: nvim_call_atomic(): multiple calls in a single request
b268ba3 #5424 API: nvim_win_get_number(), nvim_tabpage_get_number()
e7e2844 has("nvim-1.2.3") checks for a specific Nvim version
522b885 #5295, #5493 `:CheckHealth` checks tmux, terminfo, performance
719dae2 #5384 events: allow event processing in getchar()
f25797f #5386 API: metadata: Nvim version & API level
22dfe69 #5389 API: metadata: "since", "deprecated_since"
605e743 Added QuickFixLine highlight group

CHANGES:
4af6ec7 #5253 perf: Disable clipboard in do_cmdline()
6e9f329 #5299 perf: Skip foldUpdate() in insert-mode.
9d4fcec #5426 perf: Do not auto-update folds for some foldmethods.
eeec0ca #5419 tui: Default to normal-mode cursor shape.

FIXES:
e838452 #5436 tui: Fix "weird characters" / "bleeding termcodes"
10a54ad #5243 signal_init: Always unblock SIGCHLD.
bccb49b #5316 eval.c: Fix memory leak for detached pty job
626065d #5227 tchdir: New tab should inherit CWD.
cd321b7 #5292 getcwd(): Return empty string if CWD is invalid.
6127eae shada: Fix non-writeable ShaDa directory handling
ca65514 #2789 system(): Respect shellxescape, shellxquote
2daf54e #4874 Restore vim-like tab dragging
0c536b5 #5319 syntax.c: Support bg/fg special color-names.
3c53371 #4972 from justinmk/schedule-ui_refresh
68bcb32 #4789 tui.c: Do not wait for tui loop on teardown.
c8b6ec2 #5409 v:count broken in command-line window
6bc3bce #5461 fix emoji display
51937e1 #5470 fix :terminal with :argadd, :argu
79d77da #5481 external UIs: opening multiple files from command-line
657ba62 #5501 rplugin: resolve paths in manifest file
6a6f188 #5502 system('foo &', 'bar'): Show error, don't crash.
1ff162c #5515 os_nodetype: open fd with O_NONBLOCK
2a6c5bb #5450 modeline: Handle version number overflow.
0ade1bb #5225 CI tests now run against Windows!
962e876
@justinmk justinmk added a commit to justinmk/neovim that referenced this pull request Oct 28, 2016
@justinmk justinmk NVIM v0.1.6
FEATURES:
0b5a7e4 #4432 API: external UIs can render custom popupmenu
c6ac4f8 #4934 API: call any API method from vimscript
31df051 #4568 API: nvim_call_atomic(): multiple calls in a single request
b268ba3 #5424 API: nvim_win_get_number(), nvim_tabpage_get_number()
e7e2844 has("nvim-1.2.3") checks for a specific Nvim version
522b885 #5295, #5493 `:CheckHealth` checks tmux, terminfo, performance
719dae2 #5384 events: allow event processing in getchar()
f25797f #5386 API: metadata: Nvim version & API level
22dfe69 #5389 API: metadata: "since", "deprecated_since"
605e743 Added QuickFixLine highlight group

CHANGES:
4af6ec7 #5253 perf: Disable clipboard in do_cmdline()
6e9f329 #5299 perf: Skip foldUpdate() in insert-mode.
9d4fcec #5426 perf: Do not auto-update folds for some foldmethods.
eeec0ca #5419 tui: Default to normal-mode cursor shape.

FIXES:
e838452 #5436 tui: Fix "weird characters" / "bleeding termcodes"
10a54ad #5243 signal_init: Always unblock SIGCHLD.
bccb49b #5316 eval.c: Fix memory leak for detached pty job
626065d #5227 tchdir: New tab should inherit CWD.
cd321b7 #5292 getcwd(): Return empty string if CWD is invalid.
6127eae shada: Fix non-writeable ShaDa directory handling
ca65514 #2789 system(): Respect shellxescape, shellxquote
2daf54e #4874 Restore vim-like tab dragging
0c536b5 #5319 syntax.c: Support bg/fg special color-names.
3c53371 #4972 from justinmk/schedule-ui_refresh
68bcb32 #4789 tui.c: Do not wait for tui loop on teardown.
c8b6ec2 #5409 v:count broken in command-line window
6bc3bce #5461 fix emoji display
51937e1 #5470 fix :terminal with :argadd, :argu
79d77da #5481 external UIs: opening multiple files from command-line
657ba62 #5501 rplugin: resolve paths in manifest file
6a6f188 #5502 system('foo &', 'bar'): Show error, don't crash.
1ff162c #5515 os_nodetype: open fd with O_NONBLOCK
2a6c5bb #5450 modeline: Handle version number overflow.
0ade1bb #5225 CI tests now run against Windows!
cc1ec95
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment