New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extended Marks #5031

Open
wants to merge 55 commits into
base: master
from

Conversation

Projects
None yet
@timeyyy
Contributor

timeyyy commented Jul 10, 2016

These are marks designed for developers who want to build plugins or embed neovim. (they follow all column and line changes)
issue #4816

  • line insert
  • line delete
  • line join
  • line break
  • char inserts (WAHOO)
  • blockwise char inserts
  • char delete
  • delete with x
  • blockwise char deletes
  • char paste
  • blockwise char paste
  • replace
  • blockwise replace
  • undo of moved marks (undo of deleted marks is deferred)
  • redo
  • :move
  • insert tab
  • shiftwidth
  • merge undo info where possible
  • test multiwidth insert
  • kbtree merged
  • update since field in api
  • docs
  • auto-indent
  • baisc substitutions
  • substitute with backreferences (tentative)
  • substitute with newline in pattern and newlin in substition (tentative)
  • del_bytes bug
  • op_reindent
  • use ns pr
  • cleanup & refactor

In app visual testing can be done with

" Extended marks

function! LoadExtmarks()
  highlight extmark ctermbg=Blue guibg=Blue
  let g:mark_ns = nvim_create_namespace('myplugin')
  function! Testextmark(timer_id)
    " Get all the mark ids
    let a:all_marks = nvim_buf_get_marks(0, g:mark_ns, -1, -1, -1, 0)

    call clearmatches()

    for mark in a:all_marks
      let a:bytes = col([mark[1], mark[2]])
      let a:ma = matchaddpos('extmark', [[mark[1], mark[2]]])
    endfor
    call timer_start(100, 'Testextmark')
  endfunction
  call timer_start(1, 'Testextmark')
endfunction

nnoremap <leader>tm :call LoadExtmarks()<cr>
{
int fnum = 0;
buf_T *buf;
if (argvars[0].v_type != VAR_STRING){

This comment has been minimized.

@ZyX-I

ZyX-I Jul 10, 2016

Contributor

You should not check VAR_STRING, use one of get_tv_string* functions.

if (STRLEN(argvars[0].vval.v_string) > ARBMARK_MAXLEN){
EMSG(_("mark name is to large")); return;
}
if (argvars[1].v_type != VAR_NUMBER){

This comment has been minimized.

@ZyX-I

ZyX-I Jul 10, 2016

Contributor

get_tv_number*

static int _arbmark_delete(buf_T *buf, cstr_t *name);
static arbmark_T *get_arbmark(buf_T *buf, cstr_t *name);
static int pos_lt(pos_T *pos, pos_T *pos2);
static int pos_eq(pos_T *pos, pos_T *pos2);

This comment has been minimized.

@ZyX-I

ZyX-I Jul 10, 2016

Contributor

Everything here should be generated. Just if you add a new directory you need to add it to the directory list in src/nvim/CMakeLists.txt.

This comment has been minimized.

@justinmk

justinmk Jul 10, 2016

Member

@timeyyy IOW you can just delete these lines.

This comment has been minimized.

@ZyX-I

ZyX-I Jul 10, 2016

Contributor

@justinmk He is missing #include with generated functions. And he is missing it because until editor/ subdirectory is added to the list nothing is going to be generated.

This comment has been minimized.

@timeyyy

timeyyy Jul 11, 2016

Contributor

@justinmk the define should stay? do i need any includes in the c file? what does the auto generation.

This comment has been minimized.

@timeyyy

timeyyy Jul 11, 2016

Contributor

after deleting all these declarations i get implicit declaration of function errors. Am i suppose to silence these or does this mean generation some how failed?

This comment has been minimized.

@ZyX-I

ZyX-I Jul 11, 2016

Contributor

@timeyyy Check other files, they have include guarded by INCLUDE_GENERATED_DECLARATIONS, usually right after typedef’s. But as I said, as long as you have this in new editor/ subdirectory src/nvim/CMakeLists.txt needs to be edited for declarations generator to generate anything. This also applies to *.h file, and it also should not have function declarations.

This comment has been minimized.

@justinmk

justinmk Jul 11, 2016

Member

@timeyyy To be clear, CMakeLists.txt does not need to be edited if you move the file(s) out of editor/...

}
/* Will fail silently on missing name */
int arbmark_unset(buf_T *buf, cstr_t *name){

This comment has been minimized.

@ZyX-I

ZyX-I Jul 10, 2016

Contributor

{ should be on its own line. Everywhere { should be preceded by either space or newline.

#include "nvim/memory.h"
#include "nvim/map.h"
#include "nvim/lib/kvec.h"
#include "nvim/editor/arbmark.h"

This comment has been minimized.

@justinmk

justinmk Jul 10, 2016

Member

I would suggest not introducing editor/ namespace--it is redundant. We have api/ to separate non-"editor" functionality, all top-level modules are implicitly part of the core editor.

Also consider renaming the file mark_extended.{h,c} so that it sorts next to mark.{h,c}.

This comment has been minimized.

@ZyX-I

ZyX-I Jul 10, 2016

Contributor

@justinmk I also do not like name arbmark.c. But with mark_extended what should be the function prefix, mark_extended itself is too long? I can suggest extmark.

This comment has been minimized.

@justinmk

justinmk Jul 10, 2016

Member

extmark is fine with me for internal identifiers.

This comment has been minimized.

@justinmk

justinmk Jul 10, 2016

Member

Also I am wondering if this feature should be API-only (and exposed to VimL as nvim_extmark or whatever). We should perhaps make that our default choice.

This comment has been minimized.

@timeyyy

timeyyy Jul 11, 2016

Contributor

Do i need to do something to make it API only?

This comment has been minimized.

@ZyX-I

ZyX-I Jul 11, 2016

Contributor

@timeyyy “API only” means that you need public interface in src/nvim/api/(I guess)buffer.c, and this should be the only public interface: no Ex commands, no VimL functions. Tests in test/functional/api.

Note: public interface. Implementation in mark_extended.c.

This comment has been minimized.

@bfredl

bfredl Jul 11, 2016

Member

It might be worth pointing out here that API functions will soon be accessible from VimL, though with different naming and type-checking conventions than native VimL builtins. The benefit is that the generated wrappers will make (most of) the type-checking and conversion for you.

static arbmark_T *_find_pos(pos_t pos, bool FORWARD);
static int _arbmark_create(buf_T *buf, cstr_t *name, pos_T *pos);
static int _arbmark_update(buf_T *buf, arbmark_T *arbmark, pos_T *pos);
static int _arbmark_delete(buf_T *buf, cstr_t *name);

This comment has been minimized.

@ZyX-I

ZyX-I Jul 10, 2016

Contributor

Starting static functions with underscore is a useless action, making them static is enough.

@marvim marvim added the WIP label Jul 10, 2016

@@ -0,0 +1,18 @@
-- Sanity checks for buffer_* API calls via msgpack-rpc

This comment has been minimized.

@ZyX-I

ZyX-I Jul 10, 2016

Contributor

api/ is for RPC calls or events, this should probably be directly in functional/. Also new tests are normally written with one variable per line, not local clear, nvim, buffer = …. And I would suggest to never use nvim, curbuf, curwin and avoid eval: nvim('meth', …) is meths.meth(…), same for curbuf and curwin (curbufmeths, curwinmeths), nvim('eval', 'arbmark_index(haha)') is funcs.arbmark_index('haha') (and note that you are trying to refer to variable haha in your code, I guess you actually wanted string 'haha').

This comment has been minimized.

@justinmk

justinmk Jul 10, 2016

Member

Tests shouldn't live in functional/ top level, how about functional/viml.

(dict_notifications_spec.lua should be moved to functional/viml ...)

This comment has been minimized.

@ZyX-I

ZyX-I Jul 10, 2016

Contributor

I have no idea what is supposed to be placed into functional/viml. dict_notifications_spec should be probably in eval. viml/function_spec should be in eval (and I have no idea what api_info is doing in this file, I would place it separately because function_spec is too generic; this is fine if you are testing generic function-related features like MAX_FUNC_ARGS handling, but this is no place for testing specific functions), as well as viml/errorlist_spec: other functions are tested there. Not sure about viml/lang_spec and viml/completion_spec: first may be in eval, second is such a feature that can have its own directory.

This comment has been minimized.

@justinmk

justinmk Jul 11, 2016

Member

Moving viml/* to eval/ is fine with me. I think they were created by coincidence long ago.

kbitr_t = itr; \
arbmark_T = *arbmark; \
for (;kb_itr_valid(&itr) \
;kb_itr_next(arbmark_T, buf->b_arbmarks_tree; &itr)){

This comment has been minimized.

@ZyX-I

ZyX-I Jul 10, 2016

Contributor

; &itr? Code looks like you meant comma there.

@mhinz mhinz referenced this pull request Jul 10, 2016

Open

extended marks #4816

static int pos_eq(pos_T *pos, pos_T *pos2);
/* Create or update an arbmark, */
int arbmark_set(buf_T *buf, char_u *name, pos_T *pos)

This comment has been minimized.

@ZyX-I

ZyX-I Jul 11, 2016

Contributor

BTW, this ought to be char *. char_u * is deprecated, for strings you should use char *, for sequence of numbers uint8_t * (though mostly need to cast individual characters to uint8_t when appropriate). And, I guess, this should be const char *.

fmark_T fmark;
fmark_T *next;
fmark_T *prev;
} arbmark_T;

This comment has been minimized.

@ZyX-I

ZyX-I Jul 11, 2016

Contributor

CamelCase, _T suffix is legacy. Given the rename I guess this should be ExtendedMark.

@timeyyy timeyyy force-pushed the timeyyy:extended_marks branch from 50e4778 to 2b9f23a Jul 11, 2016

@timeyyy timeyyy changed the title from [WIP] arbitrary marks, to [WIP] Extended Marks Jul 11, 2016

// TODO start with current buffer
// TODO use a lru cache,
buf_T *extmark_buf_from_fnum(int fnum)

This comment has been minimized.

@bfredl

bfredl Jul 11, 2016

Member

after #4934 find_buffer_by_handle can be used for this

for (;kb_itr_valid(&itr); kb_itr_next(ExtendedMark, buf->b_extmarks_tree, &itr)){
/* Create or update an extmark, */
int extmark_set(buf_T *buf, cstr_t *name, pos_T *pos)

This comment has been minimized.

@ZyX-I

ZyX-I Jul 11, 2016

Contributor

This is rather strange to see cstr_t anywhere, but where it is absolutely required because char * cannot be part of the function/etc name. It should be char ** if you really meant cstr_t * (cstr_t is an alias to char *, not char), just char * otherwise.

This comment has been minimized.

@timeyyy

timeyyy Jul 11, 2016

Contributor

this name is going to be used as the key for the hash table, would you prefer i cast it to cstr just before using it as a key? Is it prefered that the function accept char * or cstr_t ?

This comment has been minimized.

@ZyX-I

ZyX-I Jul 11, 2016

Contributor

You don’t need to cast char * to cstr_t. Function should accept char *, cstr_t exists purely for technical reasons.

This comment has been minimized.

@ZyX-I

ZyX-I Jul 11, 2016

Contributor

I actually have no idea why *Map interface exists, khash.h is already fine and it does not require you to query hash twice to do “if KEY is in hash return VAL, otherwise put (KEY, VAL) to hash” . Also one map_new is two small mallocs alone, while my small changes to khash.h allowed to put hash on stack (or e.g. directly as a part of some structure, not as a pointer).

My code never uses map.h.

This comment has been minimized.

@bfredl

bfredl Jul 11, 2016

Member

“if KEY is in hash return VAL, otherwise put (KEY, VAL) to hash”

map_ref(map, key, true) supports this in most cases. (more precisely: it won't work when one wants to distinguish a nonexisting key from the initializer value, but one could easily add a bool return value to signal this if necessary)

This comment has been minimized.

@ZyX-I

ZyX-I Jul 11, 2016

Contributor

@bfredl Last part (nonexisting vs initializer) is exactly what I mean. Also this does not remove issue with double allocations vs no allocations and useless indirection (existing kh API is already fine, also allocated struct containing a single pointer value which is also allocated?!).

This comment has been minimized.

@ZyX-I

ZyX-I Jul 11, 2016

Contributor

I also have an impression that what I already said was not the only issue with map interface, but cannot remember anything else. Perhaps one of the issues was that public API is too complex: when you declare a new hash using KHASH_… you get nice names like kh_init_foo. When you declare a new hash using map interface you need to use map_new(Type1, Type2). Worse, you need to typedef Type1 and Type2 in some cases I would not typedef and this is absolutely not needed when using khash.

This comment has been minimized.

@bfredl

bfredl Jul 11, 2016

Member

We could refactor the Map wrapper functions (I think some of them are common enough to justify a wrapper, like putting a value in a single function (or macro) call) to operate directly on the khash_t types, not on the superfluous wrapper types.

This comment has been minimized.

@ZyX-I

ZyX-I Jul 11, 2016

Contributor

@bfredl There is no need in wrapper functions, just use khash directly. When needed khash.h can be adjusted, ability to work with values allocated on stack was my addition when I asked myself why should I need to allocate a hash struct itself.

Though there is additionally one thing I do not understand in khash.h: why keys/flags/vals and not one struct{key, flag, val} items? My best guess is that this has something to do with CPU cache, three allocations are definitely not going to be faster then one.

This comment has been minimized.

@bfredl

bfredl Jul 12, 2016

Member

I did't say wrappers must be used - just that some operations are common enough to justify it, ie a wrapper kh_putval(type, somestruct->map, key, val) instead of kh_val(somestruct->map, kh_put(type, somestruct->map, key, &ret) = val. This can just be inline functions or macros, no need for a separate compilation unit map.c.

My best guess is that this has something to do with CPU cache, three allocations are definitely not going to be faster then one.

The vectors are going to be used many times for each (re)allocation (though in principle a struct-of-arrays could also be allocated using a single alloc just fine, if one just places the most aligned type first) and as flags and keys will be accessed a lot more than values in lookup it is often beneficial to keep them close together. (though an another option is to keep the keys and values array compact, and have the sparse hash table just have indicies to them, like pypy:s implementation, though the approaches could also be combined )

@timeyyy timeyyy force-pushed the timeyyy:extended_marks branch from 2f1f6f1 to 09eaf67 Jul 12, 2016

@@ -33,6 +33,7 @@ typedef struct file_buffer buf_T; // Forward declaration
#define MODIFIABLE(buf) (!buf->terminal && buf->b_p_ma)

This comment has been minimized.

@ZyX-I

ZyX-I Jul 12, 2016

Contributor

Too many blank lines.

@@ -8,6 +8,7 @@
#include "nvim/memory.h"
#include "nvim/pos.h"
#include "nvim/os/time.h"
/* #include "nvim/ex_cmds_defs.h" // for exarg_T */

This comment has been minimized.

@ZyX-I

ZyX-I Jul 12, 2016

Contributor

?

return pmap_get(ExtendedMark)(buf->b_extmarks, name);
}
int _pos_cmp(pos_T a, pos_T b)

This comment has been minimized.

@ZyX-I

ZyX-I Jul 12, 2016

Contributor

This is FUNC_ATTR_CONST FUNC_ATTR_WARN_UNUSED_RESULT.

}
}
static int pos_lt(pos_T *pos, pos_T *pos2){

This comment has been minimized.

@ZyX-I

ZyX-I Jul 12, 2016

Contributor

And this should be bool and return true/false, not int with OK/FAIL. OK/FAIL is for statuses, it is not a failure if e.g. two positions are equal.

This comment has been minimized.

@ZyX-I

ZyX-I Jul 12, 2016

Contributor

Also I see no reason to have this and pos_eq as a separate functions since they are not used anywhere, pos_cmp is enough.

/// @param name The mark's name
/// @param[out] err Details of an error that may have occurred
/// @return The (row, col) tuple
ArrayOf(Integer, 2) mark_index(Buffer buffer, String name, Error *err)

This comment has been minimized.

@bfredl

bfredl Jul 16, 2016

Member

Please prefix api functions in buffer.c with buffer_ (Or, as I expect #4934 to be merged before this, you can just use nvim_buf_ upfront, but then you will need to write tests with helpers.request for the moment until #4934 gets merged)

/// @param name The mark's name
/// @param[out] err Details of an error that may have occurred
/// @return The (row, col) tuple
ArrayOf(Integer, 2) mark_next(Buffer buffer, String name, Error *err)

This comment has been minimized.

@bfredl

bfredl Jul 17, 2016

Member

While this is a good api for consumers (vimscript, lua) within the nvim process, for remote plugins it could be too slow as it requires a roundtrip (two context switches) per mark it passes by. I would imagine a function nvim_buf_get_extmarks(buffer, start, end, which) which would return marks in some range [start,end] where the boundaries ideally could be either explicit positions (including the start/end of the entire buffer), or the name of marks, and one could request to either get the N first or N last or all marks in that range. (which will cover the ability of these functions)

@bfredl

This comment has been minimized.

Member

bfredl commented Jul 17, 2016

Also, while out of scope of this PR, some time later I would imagine all api functions that take a line number or a position (line,col) pair to be polymorphic and also take

  • the name of existing marks 'a and named positions like the cursor position .
  • the name of an extended mark
  • the numerical id of an unnamed extended mark*

* I would imagine many plugins have no reason to name their private marks and would be forced to come up with dummy myfancyplugin0/myfancyplugin1 etc names. Ideally they will be able to pass the empty string to mark_set and get a unique numerical ID in return which could be passed in everywhere the name of an extended mark is expected.

It would also be useful to reuse the binary tree defined here for other kind of marks, like bufhl, manual folds etc. As I said likely out of scope of this PR (which could focus on delivering a useful MVP), but I'm pointing it out here as it might (or might not) influence the design.

@timeyyy

This comment has been minimized.

Contributor

timeyyy commented Jul 17, 2016

At the moment I am following the api for tk. Text tags have more functionality than text marks. Theyvprivide the functionalit you mentioned and more. The reason youbwould use tags for this is because they live in there own namespace, you can get a tag range for a specific tag type, while with marks you are limited to the one namespace. If later on one wanted ranges of different types of marks you have to switch to the tag api or handle some mapping yourself. That might be a reason the api doesnt exist, if people want I can add the range functionality to marks as well.

@timeyyy

This comment has been minimized.

Contributor

timeyyy commented Jul 19, 2016

I was thinking to introduce a namespace for the marks. This would mean bufhl and tags/ plugins create one mark that needs to be moved per position.

when doing mark_next / mark_names etc,
The results could also be filtered namespaces as well.

If we did not do this, queryibg marks will always give all marks, and not just those that a particular user of the api cares about.

@bfredl

This comment has been minimized.

Member

bfredl commented Jul 19, 2016

Yes, that would be very useful. That would in fact improve performance of "numerical marks". an rplugin could then increment a private counter itself, and doadd_extmark("myFancyPlugin",0,...), add_extmark("myFancyPlugin",1,...) etc without needing to wait for it to return an unique id.

@timeyyy timeyyy force-pushed the timeyyy:extended_marks branch from 5d6b1e1 to 47743ae Jul 19, 2016

@bfredl bfredl force-pushed the timeyyy:extended_marks branch from 931eb2e to b29898d Nov 29, 2018

@bfredl bfredl force-pushed the timeyyy:extended_marks branch from f94a901 to 08dcc03 Nov 29, 2018

bfredl added some commits Dec 1, 2018

API CHANGE: use 0-based indexing in api
only changing surface for now. We could change internals later, or
not...
@bfredl

This comment has been minimized.

Member

bfredl commented Dec 1, 2018

Changed to use zero-indexing in API, as is convention. Also nvim_buf_clear_namespace WIP, but needs more testing (maybe I already wrote this, but github requires like 10 clicks to read the end of this thread...)

bfredl added some commits Dec 14, 2018

extmark: change bounds handling in API
- remove "magic" -1 mark that means different things for upper and
lower, just use {0, 0} and {-1, -1} as extreme bounds

- make it end exclusive, in line with conventions
extmark: simplify API by removing "reverse" arg.
just specify an inverted range. Though this cannot be end-exclusive,
so revert that part
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment