vim9: Make Autoload Functions Easier to Define and Call

[rbong]

Is your feature request about something that is currently impossible or hard to do? Please describe the problem.

As the language is being overhauled I believe we should make autoload functions nicer and easier to convert to other languages.

I've included some quotations from the vim 9 doc to support this idea.

we can also make Vim script easier to use. In other words: "less weird"

Compared with other languages, prefixing the file name on every function call is weird and awkward.
The only other option for public functions is the global namespace, which practically also requires some name prefix.

It should be possible to convert code from other languages to Vim script.

For a project with a public API, my options with vim9script function naming are g:SomeLongPrefix or some#long#prefix#.
Naming prefixes like this are not required in any other language.
This makes conversion more awkward than anything else with vim9script.

This becomes worse the more deeply nested autoload files are.
Using the global namespace is no better.

Describe the solution you'd like

Add a command to define the current autoload file and automatically prefix this name.
Allow "importing" functions directly from autoload files without pre-loading them.

Example:

In Vim9 script:

vim9script

# This prefixes function names with "my_plugin#nested#"
setautoload my_plugin#nested#

def foo()
  echo "Foo"
enddef

# To avoid name conflicts, you can also manually define function names.
def my_plugin#nested#bar()
  echo "Bar"
enddef

This is equivalent to the following in legacy vimscript:

def my_plugin#nested#foo()
  echo "Foo"
enddef

def my_plugin#nested#bar()
  echo "Bar"
enddef

You could also import this function:

vim9script

# Replaces all calls to "foo" with "my_plugin#nested#foo"
import { foo } from autoload "my_plugin#nested#"

# Replaces all calls to "nested_foo" with "my_plugin#nested#foo"
import { foo as nested_foo } from autoload "my_plugin#nested#"

Using this, functions still would not be loaded until they were actually called.

Describe alternatives you've considered

Autoloading could be automatic, or autoload names could automatically be prefixed.

[brammool]

As the language is being overhauled I believe we should make autoload functions nicer and easier to convert to other languages.

Thanks for thinking about this. you appear to point out two things: - In the autoload script itself, all functions need to be prefixed with path and directory name, such as "some#long#prefix". - Repeating this prefix every time the function is called. Generally the names are in the global namespace and the prefix is used to locate the script. That can't be changed. We can look into how the prefix is used. Using a long name, including directory names, is not unusual. Most languages have an "import" or "include" mechanism that uses this. Usually then, after the import, only the last part is used, e.g. the class name, and the path leading to it does not need to be repeated. Some languages import a file and you never know what it defines. Such as "include" in C and "import" in Dart. I definitely want to avoid that, it makes debugging very hard: You never know where a symbol came from. Vim9 has an import/export mechanism. But so far this didn't support auto-loading. The advantage of loading it right away is that any imported items can be inspected, whether they actually exist and what types they use. This also allows for compiling the function call. We could try to add auto-loading to import/export somehow. First thing is that the import only specifies the path below the runtime/autoload directory. Thus "import... autoload 'some/long/prefix/script.vim'" would search for the script 'some/long/prefix/script.vim' in all "autoload" directories of entries in the 'runtime' option. This is similar to how compilers specify a list of directories where library headers can be found. Otherwise it could work like existing Vim import commands, thus allowing for importing a single item, multiple items, or everything with "* as That". In the autoload script we should explicitly export those things that it wants to publish. The rest is script-local. That is consistent with how Vim9 scripts work. Like in Java you export the class and nothing else. Using "export autoload ..." doesn't make sense, since the autoload argument would be either on none or all "export" commands. It would be needed to declare that the script is an autoload script once. And fairly early. We could use "vim9script autoload". We already require "vim9script" to be used once in the script, and it specifies the type of script. Vim can figure out the prefix from the path down from "runtime/autoload", there is no need to specify it. You can then even move the file elsewhere and rename it, the contents does not need to change. Still, the new name must be used where it is imported. I think that covers your suggestion. It is an addition to what we already have, I don't see a conflict.

Using this, functions still would not be loaded until they were actually called.

That works, but does have a disadvantage: Existence and types are not checked until the function is used. We do not want to load the whole autoload script, since that would have a large performance penalty. I was wondering if we can have some "header script" that defines the types. But it's hard to keep in sync. It could perhaps be generated, but that complicates things. Also, just searching for the header file through directories already has a performance penalty. It seems better to just the autoload-import to remember what the script name is, and postpone looking for it until an item is actually used. Perhaps we can have a way to force loading and compiling an autoload script the moment it is encountered, so that you can test for errors? Perhaps with the test_override() function.

…

-- If you feel lonely, try schizophrenia. /// Bram Moolenaar -- ***@***.*** -- http://www.Moolenaar.net \\\ /// \\\ \\\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ /// \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[rbong]

I think that covers your suggestion.

It does cover my suggestion, it's exactly what I'm looking for - a way not to type the autoload prefix every time.

But as you point out there are wider reaching problems as well.

That works, but does have a disadvantage: Existence and types are not
checked until the function is used.

I have some ideas on this that may or may not be practical.

You might not need to check autoload functions themselves every time Vim starts up, just when installing or updating them.
Maybe you can do something at install and update so that you can type-check calls to them as well.

I think ideally, Vim would have a way to install and update packages from different sources.
This would be consistent, I think, with making Vim script nicer to use.
Most modern languages have tooling like this.

This ties in with the next idea about this...

I was wondering if we can have some "header script" that defines the
types. But it's hard to keep in sync. It could perhaps be generated,
but that complicates things.

I was thinking ideally, Vim could cache compiled functions similar to __pycache__ in Python.
The cache could be invalidated whenever the script changes.
You could also auto-generate a "header file" while compiling the cache.
You could also cache errors, so that if an autoload function that compiled with errors and hasn't changed is imported, it gives a warning or error.

This would also solve the problem of recompiling autoload functions the first time they're used every time.

Like I said, you only generally need to check types when installing or updating scripts.
If you had package install/update commands, you could do the first compile/cache for scripts when you run these commands.
After that you could still recompile if the script changes.

You could also implement the cache before package management, and initially just have autoload functions compile for the first time when they are imported.
This could come with a one-time performance hit that is not as transparent as explicitly installing packages.

A real problem with this solution is that if you import autoload functions and the autoload script has changed, you can't check types.
That is, unless if you recompile the autoload functions with invalidated caches on import, which will cause a performance hit.
However autoload functions would still have the advantage that they don't need to be loaded into memory until they're actually called, and most won't need to be recompiled every time.

Another problem with this solution, similar to what you pointed out, is that header files will cause a performance hit.

However despite these performance hits, if the cache is used for everything, performance might overall be better.

Like you said, this complicates things, but this is my idealistic solution.
There are probably problems and practical considerations I'm not thinking of, but I think it's worth at least talking about better tooling and caching as long as Vim script is being overhauled.

[brammool]

[...]

>That works, but does have a disadvantage: Existence and types are not >checked until the function is used. I have some ideas on this that may or may not be practical. You might not need to check autoload functions themselves every time Vim starts up, just when installing or updating them. Maybe you can do something at install and update so that you can type-check calls to them as well. I think ideally, Vim would have a way to install and update packages from different sources. This would be consistent, I think, with making Vim script nicer to use. Most modern languages have tooling like this.

There are several plugin managers, and it appears they work well. At least one of them is actually an autoload plugin, so that you only have to get one file to make it work. Vim provides some support for updating packages, but the actual fetching is something that goes beyond Vim's core, it's better done in a plugin. We could add some mechanism to load an autoload file as soon as it is encountered, so that any errors are uncovered early, not at some unexpected moment later. This could be used by a plugin developer who wants to check the latest modifications. Or by a plugin manager that just downloaded a new version. [...]

I was thinking ideally, Vim could cache compiled functions similar to `__pycache__` in Python. The cache could be invalidated whenever the script changes. You could also auto-generate a "header file" while compiling the cache. You could also cache errors, so that if an autoload function that compiled with errors and hasn't changed is imported, it gives a warning or error.

This is a very complicated mechanism, full of pitfalls and will be source of bugs. I would only go this way if we actually solve a big problem. I don't think there is a performance problem big enough to take on this extra ballast. There are so many things to take into account besides normal use: - restoring a backup - copying the Vim files to another system - having the files reside on a removable device - the system time being changed (DST start and end) - someone deleting a plugin, how do we clean up? - using two Vim versions simultaniously - etc.

…

-- A computer program does what you tell it to do, not what you want it to do. /// Bram Moolenaar -- ***@***.*** -- http://www.Moolenaar.net \\\ /// \\\ \\\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ /// \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[lacygoill]

I've just noticed that my Vim startup time increased significantly. It's around 160ms. In the past, it was between 110ms and 120ms (while the system is mostly idle).

That's a big increase. It seems the culprits are all the import autoload statements.

For example, my slowest plugin contains these statements:

import autoload 'cmdline.vim'
import autoload 'cmdline/cL.vim'
import autoload 'cmdline/cycle/filter.vim' as FilterCycle
import autoload 'cmdline/cycle/generic.vim' as GenericCycle
import autoload 'cmdline/cycle/vimgrep.vim' as VimgrepCycle
import autoload 'cmdline/tab.vim'
import autoload 'cmdline/transform.vim'
import autoload 'cmdline/unexpand.vim'

Here is the average time (in milliseconds, measured during 10 startups) when I write finish above all of them:

0.028

And here is the average time when I move finish below the first import autoload, the 2nd one, the 3rd one, ...:

0.478
0.881
1.373
1.722
2.2
2.81
3.433
3.568

It seems each statement adds on average about 0.4 milliseconds. I have 116 import autoload across all of my plugins. If each of them adds 0.4ms, that makes 46.4ms in total; this matches the overall startup time increase which I noticed originally.

Did someone else notice that the Vim9 import autoload statement was slower than the legacy mechanism?

Is it working as intended, or has there been some recent regression?

Could the way Vim handles import autoload somehow be optimized?

Do I need to gather more information (possibly via gprof(1))?

[brammool]

Do you perhaps have many entries in 'runtimepath'? The "import autoload" does not load the script, but it does find its location.
That means going over entries 'runtimepath' to search for "autoload/scriptname". This involves scanning directories, which might be slow.

A profile might help to locate any bottlenecks in this process. We could add some kind of caching for finding autoload scripts. But only when it really helps, since caching can cause trouble. It could even make it slower.

[lacygoill]

Do you perhaps have many entries in 'runtimepath'?

I guess so: I have 104 entries in 'runtimepath'.

A profile might help to locate any bottlenecks in this process.

A profile of the C code, right? I tried to get one, here it is.

If that's not what is needed, I would need someone to tell me which commands to run to get the right data.

[lacygoill]

A profile of the C code, right? I tried to get one, here it is.

To get it, I compiled Vim like this (in fish):

$ git reset --hard $(git rev-parse HEAD) \
    ; make clean \
    ; make distclean \
    ; sed -i '
    /#PROFILE_CFLAGS = -pg -g -DWE_ARE_PROFILING/s/^#//
    ; /#PROFILE_LIBS = -pg -no-pie/s/^#//
    ' src/Makefile \
    ; make \
    ; tput bel

Then, I started Vim with my full config:

$ ./src/vim

Finally, I asked gprof(1) to produce a log from the gmon.out binary dump:

$ gprof ./src/vim ./gmon.out

[lacygoill]

Sorry, the previous log was obtained while the system was not idle. Here is a new log (this time, the system was mostly idle).

[brammool]

Sorry, the previous log was obtained while the system was not idle. [Here is a new log](https://github.com/vim/vim/files/8374427/log.txt) (this time, the system was mostly idle).

This one is indeed much more useful. Central appears to be do_in_path(), which loops over the entries in 'runtimepath'. That confirms my suspicion. do_in_path() is called 333 times, and it calls copy_option_part() 17192 times, which suggests that 'runtimepath' has 51 entries (on average) that are looked at. There can be more entries, since often only the first match is found. The calls come half from ex_packadd(), I guess you are installing many plugins. This actually may add an entry to 'runtimepath', thus every later call has to go through more entries. 120 come from find_script_in_rtp(), which is called from ex_import(). I can't think of a way to avoid this with just code changes, the loop through 'runtimepath' is needed for several reasons. Is it perhaps true that these autoload imports actually use the autoload directory relative to where the script is? Then we would not have to look through 'runtimepath'. E.g.: plugin/script.vim: import autoload '../autoload/lib.vim' autoload/lib.vim: export def TheFunction() ... enddef This currently isn't supported, but it could be made to work. Another advantage is that the autoload script will be found even when another plugin uses the same script name. Perhaps we should not put it in the "autoload" directory for that reason. However, that would be a new mechanism and might cause problems somehow.

…

-- Did you ever stop to think... and forget to start again? -- Steven Wright /// Bram Moolenaar -- ***@***.*** -- http://www.Moolenaar.net \\\ /// \\\ \\\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ /// \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[lacygoill]

Is it perhaps true that these autoload imports actually use the autoload
directory relative to where the script is?

I suspect that yes, the vast majority are probably relative to the plugin script. IOW, most of the time, there should be no need to look into the runtimepath, provided that we support the ../autoload/script.vim syntax.

[bfrg]

I think this issue can be closed since we have import autoload by now.

[brammool]

Right, there haven't been comments or discussions on the current implementation recently. I take that as an indication that most script writers are OK with how it works now.
If someone does come up with a problem or idea for improvement, please open a new issue.