Inconsistent loading order of packages and their parts

[sheerun]

Describe the bug

It seems there is different loading order of plugins and their parts if using empty vimrc / packload / packadd / packadd!

To Reproduce

I've created https://github.com/sheerun/vim-packages-bug that should better illustrate the problem. You can run there bash script.sh to produce following output:

1. Putting package in ~/.vim/pack/*/start/* without pack commands in ~/.vimrc

loading plugin/*.vim after native vim plugins
loading ftdetect/*.vim after native filetype.vim
loading after/plugin/*.vim after native vim plugins
Press ENTER or type command to continue

2. Putting package in ~/.vim/pack/*/start/* with packload

loading plugin/*.vim before native vim plugins
loading ftdetect/*.vim after native filetype.vim
loading plugin/*.vim before native vim plugins
loading after/plugin/*.vim after native vim plugins
Press ENTER or type command to continue

3. Putting package in ~/.vim/pack/*/start/* with packadd vim-packages-bug in vimrc

loading plugin/*.vim before native vim plugins
loading ftdetect/*.vim after native filetype.vim
loading plugin/*.vim before native vim plugins
loading plugin/*.vim after native vim plugins
loading ftdetect/*.vim after native filetype.vim
loading after/plugin/*.vim after native vim plugins
Press ENTER or type command to continue

4. Putting package in ~/.vim/pack/*/start/* with packadd! vim-packages-bug in vimrc

loading plugin/*.vim before native vim plugins
loading plugin/*.vim after native vim plugins
loading ftdetect/*.vim after native filetype.vim
loading after/plugin/*.vim after native vim plugins
Press ENTER or type command to continue

5. Putting package in ~/.vim/pack/*/opt/* with packadd vim-packages-bug in vimrc

loading plugin/*.vim before native vim plugins
loading ftdetect/*.vim after native filetype.vim
loading plugin/*.vim before native vim plugins
loading after/plugin/*.vim after native vim plugins
Press ENTER or type command to continue

6. Putting package in ~/.vim/pack/*/opt/* with packadd! vim-packages-bug in vimrc

loading plugin/*.vim before native vim plugins
loading after/plugin/*.vim after native vim plugins
Press ENTER or type command to continue

---

To summarize

1. plugins can be loaded before or after vim plugins, possibly multiple times
2. ftdetect is loaded always after vim filetypes, possibly multiple times (all other package managers always load ftdetect before vim native files..)
3. In some cases no part of plugin is loaded before vim parts

Things get even worse if multiple packages are used, because then they can be loaded in alphabetical order (no pack commands), in defined order (packadd commands), or reverse defined order (packadd! commands), and all combinations of these. Order of loading is also affected if some plugins are calling filetype plugin indent on within themselves (like vim-sensible). Also all other package managers are loading filetype.vim of plugin while vim8 packages do not, ever.

Honestly it's pretty overwhelming for someone who would like to depend on some order of loading in package manager (the vim-polyglot plugin I'm developing needs to be loaded before vim's filetype.vim, and other plugins overriding its autocommands).

Environment (please complete the following information):

VIM - Vi IMproved 8.2 (2019 Dec 12, compiled Sep 16 2020 23:47:44)
macOS version
Included patches: 1-1700
Compiled by Homebrew
Huge version without GUI.  Features included (+) or not (-):
+acl               -farsi             +mouse_sgr         +tag_binary
+arabic            +file_in_path      -mouse_sysmouse    -tag_old_static
+autocmd           +find_in_path      +mouse_urxvt       -tag_any_white
+autochdir         +float             +mouse_xterm       -tcl
-autoservername    +folding           +multi_byte        +termguicolors
-balloon_eval      -footer            +multi_lang        +terminal
+balloon_eval_term +fork()            -mzscheme          +terminfo
-browse            +gettext           +netbeans_intg     +termresponse
++builtin_terms    -hangul_input      +num64             +textobjects
+byte_offset       +iconv             +packages          +textprop
+channel           +insert_expand     +path_extra        +timers
+cindent           +ipv6              +perl              +title
-clientserver      +job               +persistent_undo   -toolbar
+clipboard         +jumplist          +popupwin          +user_commands
+cmdline_compl     +keymap            +postscript        +vartabs
+cmdline_hist      +lambda            +printer           +vertsplit
+cmdline_info      +langmap           +profile           +virtualedit
+comments          +libcall           -python            +visual
+conceal           +linebreak         +python3           +visualextra
+cryptv            +lispindent        +quickfix          +viminfo
+cscope            +listcmds          +reltime           +vreplace
+cursorbind        +localmap          +rightleft         +wildignore
+cursorshape       +lua               +ruby              +wildmenu
+dialog_con        +menu              +scrollbind        +windows
+diff              +mksession         +signs             +writebackup
+digraphs          +modify_fname      +smartindent       -X11
-dnd               +mouse             -sound             -xfontset
-ebcdic            -mouseshape        +spell             -xim
+emacs_tags        +mouse_dec         +startuptime       -xpm
+eval              -mouse_gpm         +statusline        -xsmp
+ex_extra          -mouse_jsbterm     -sun_workshop      -xterm_clipboard
+extra_search      +mouse_netterm     +syntax            -xterm_save
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
       defaults file: "$VIMRUNTIME/defaults.vim"
  fall-back for $VIM: "/usr/local/share/vim"
Compilation: clang -c -I. -Iproto -DHAVE_CONFIG_H   -DMACOS_X -DMACOS_X_DARWIN  -g -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: clang   -L. -fstack-protector-strong -L/usr/local/lib -L/usr/local/opt/libyaml/lib -L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/readline/lib  -L/usr/local/lib -o vim        -lncurses -liconv -lintl -framework AppKit  -L/usr/local/opt/lua/lib -llua5.3 -mmacosx-version-min=10.15 -fstack-protector-strong -L/usr/local/lib  -L/usr/local/Cellar/perl/5.32.0/lib/perl5/5.32.0/darwin-thread-multi-2level/CORE -lperl -lm -lutil -lc  -L/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/config-3.8-darwin -lpython3.8 -framework CoreFoundation  -lruby.2.7

[sheerun]

Also ftdetect can be not loaded if packadd! is used and plugin is in opt (but it works if it's in start)....

[sheerun]

Oh, plus vim and nvim have different behavior for each of these: k-takata/minpac#118 (comment) (minpac depends on vim8 / neovim packages to load plugins instead of modiifying rtp like vim-plug or vundle)

I mean in current form vim8 packages are pretty much unstable, and actually much harder to work with than any other solution available. I think they need v2.0 with much simpler logic, something like this:

1. No splitting of plugins of packages into start and opt, all plugins are opt, the tree of pack is simplified to: ~/.vim/pack/<packagename>/<pluginname>/{indent,plugin,ftdetect,after,...}
2. To load plugin, it needs to be explicitly called in ~/.vimrc (this ensures proper loading order), with some new command like pack vim-something
3. Directories of plugins are added to rtp at the front in chronological order, and after directories of plugins are added at the end of rtp, also in chronological order (i.e. defined by order of pack commands in ~/.vimrc).
4. No command like packload which can load plugins in unstable order (alphabetical, reverse alphabetical)
5. It's plugins that are responsible for loading their sub-dependencies, not vim (e.g. by calling pack command in their plugin/* scripts)
6. Plugins are loaded immediately when plug is called (with after directories deferred). This allows for easy overridding of settings set by plugins directly in ~/.vimrc

[brammool]

"the vim-polyglot plugin I'm developing needs to be loaded before vim's filetype.vim". That is a very difficult restriction you are making. The user may have "syntax on" somewhere you don't expect it. Why not remove this dependency?
Also, packages should not depend on the loading order. :packload first adds all the directories to 'runtimepath', so that when there are dependencies it should be possible to find them.

[sheerun]

Well, for me it's difference between 60ms and 10ms of loading time, because that's how much hundreds of au! calls take ( au calls are pretty fast), and I need to run them to override vim defaults if I load after vim. I can (and will) program polyglot to use au! instead of au if for some reason vim files loaded first (which will satisfy "packages should not depend on the loading order"), but it'll always have performance penalty. With current implementation of vim8 packages it's pretty much guaranteed that vim-polyglot loads later, while for all other plugin managers it's pretty much guaranteed that it'll always load first.

[brammool]

For filetypes it's basically "last one wins", so why do you need to remove what was added before?
Specifically, is this about the order in which autocommands are added, or the order of 'runtimepath', which changes in which order the plugins are found?

[sheerun]

runtimepath works well for all files except initial ftdetect files, or rather filetype.vim (e.g. ruby.vim of vim-polyglot is properly loaded always before vim's). Indeed there's "last wins", but changing filetype multiple times for loaded file has its own performance penalty, so it's best handled by ensuring correct filetype is applied in the first place (vim uses setf wchich is "first one wins").

[sheerun]

I mean I can hack anything as long as something is loaded before vim's runtime files. Current implementation doesn't load anything at all, so there's no way around it. I basically must live with vim runtime files being sourced first, and I cannot do anything about it, this is the gist of this issue.

[brammool]

runtimepath works well for all files except initial ftdetect files, or rather filetype.vim (e.g. ruby.vim of vim-polyglot is properly loaded always before vim's). Indeed there's "last wins", but changing filetype multiple for loaded file has its own performance penalty, so it's best handled by ensuring correct filetype is applied in the first place (vim uses `setf` wchich is "first one wins").

Yeah, that's right, using "setf" is different. And you don't really want to set one filetype and then overrule it, causing multiple filetype plugins to be loaded in sequence. I would think it works to overrule the autocommand from filetype.vim: au BufRead *.js --- Autocommands --- filetypedetect BufRead *.js setf javascript au! filetypedetect BufNewFile,BufRead *.js setf myJs au BufRead *.js --- Autocommands --- filetypedetect BufRead *.js setf myJs

…

-- How To Keep A Healthy Level Of Insanity: 6. In the memo field of all your checks, write "for sexual favors". /// Bram Moolenaar -- Bram@Moolenaar.net -- http://www.Moolenaar.net \\\ /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\ \\\ an exciting new programming language -- http://www.Zimbu.org /// \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[sheerun]

Yes indeed, I've mentioned I can use au! but it has heavy performance penalty. 60ms vs 10ms loading time

[sheerun]

I've decided to not use au! for each overridden filetype, but au! filetypedetect + basically copy all vim's filetype.vim into vim-polyglot, which finally I know will load last :P

As of now the issue is twofold:

1. vim's filetype.vim is unnecessairly loaded which takes 25ms (it is fully overridden by vim-polyglot)
2. au! can't be used by other plugins because it depends on load order of plugins, and also it requires that arguments are exactly the same which they often aren't. for example one plugin can use au! BufRead Makefile.am setf make, when second uses au! BufRead [Mm]akeifile.am setf makefile (second au! won't override first one, and filetype will be set to make)

This could all be solved by instructing vim to source second plugin first. which currently isn't encouraged ("packages should not depend on the loading order") or even possible to enforce with vim8 packages

[brammool]

Adam Stankiewicz wrote:

I've decided to not use `au!` for each overridden filetype, but `au! filetypedetect` + basically copy all vim's filetype.vim into vim-polyglot, which finally I know will load last :P As of now the issue is twofold: 1. vim's filetype.vim is unnecessairly loaded which takes 25ms (it is fully overridden by vim-polyglot)

I don't see how you can override all of filetype.vim. Do you mean to clear it and load it again?

2. `au!` can't be used by other plugins because it depends on load order of plugins, and also it requires that arguments are exactly the same which they often aren't. for example one plugin can use `au! BufRead Makefile.am setf make`, when second uses `au! BufRead [Mm]akeifile.am setf makefile`. This could all by instructing vim to source second plugin first. which currently isn't encouraged ("packages should not depend on the loading order") or even possible to enforce with vim8 packages

It's very tricky to make plugins load in the right order. Especially if you don't know what the right order is. Autocommands are executed in the order they are defined. If a later plugin adds an autocommand for filetype detection, it can overrule what happened earlier: au BufNewFile,BufRead *.mine set filetype=mine In filetype.vim ":setfiletype" is used, which won't change 'filetype' if it was already set, thus it doesn't matter if it's loaded before or after this. The only disadvantage is that 'filetype' might be set twice. Is that really a problem? If two plugins fight over the same rule, well, then there is a problem.

…

-- How To Keep A Healthy Level Of Insanity: 16. Have your coworkers address you by your wrestling name, Rock Hard Kim. /// Bram Moolenaar -- Bram@Moolenaar.net -- http://www.Moolenaar.net \\\ /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\ \\\ an exciting new programming language -- http://www.Zimbu.org /// \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[brammool]

Yes indeed, I've mentioned I can use `au!` but it has heavy performance penalty. 60ms vs 10ms loading time

I suppose that's because it has to go over all the autocommand to find the ones that need to be deleted. Perhaps we can make this faster, that is useful in many ways. So this delay happens if, after "filetype on", you use an "au!" command? Can you give an example?

…

-- How To Keep A Healthy Level Of Insanity: 15. Five days in advance, tell your friends you can't attend their party because you're not in the mood. /// Bram Moolenaar -- Bram@Moolenaar.net -- http://www.Moolenaar.net \\\ /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\ \\\ an exciting new programming language -- http://www.Zimbu.org /// \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[sheerun]

I don't see how you can override all of filetype.vim. Do you mean to
clear it and load it again?

In case vim's filetype.vim is sourced first, I'm running au! filetypedetect which quickly clears any autocommands set by filetype.vim, and then I "source" filetype.vim again (in reality I've just copied some code from filetype.vim)

In case vim's filetype.vim is sourced last, thanks to let did_load_filetypes = 1 set by vim-polyglot, it makes sourcing vim's filetype.vim a no-op 0ms, instead of 25ms (optimization which I cannot do if filetype.vim of vim is loaded first).

So this delay happens if, after "filetype on", you use an "au!" command?
Can you give an example?

It isn't one-time delay, it's difference of calling hundreds of au, vs calling analogical hundreds of au! commands (I'm no longer doing that, what I'm doing now is just single au! filetypedetect, which is fast and doesn't need to be improved).

Currently without vim-polyglot loading time is as follows:

1. filetype.vim = 25ms

With vim-polyglot loaded after filetype.vim loading time is as follows:

1. filetype.vim = 25ms
2. polyglot = 10ms

If vim-polyglot is loaded before filetype.vim, then loading time is as follows:

1. polyglot = 10ms
2. filetype.vim = 0ms

So using vim-polyglot makes startup time faster than native, if sourced in correct order, and slower than native if in incorrect order.