Create DSL for hooks

[ZyX-I]

This is hooks modifications I was talking about in #80.

Proposed DSL capabilites:

• conditional execution
• ERE regular expressions substitutions
• executing shell commands
• changing current directory
• file manipulations: read/write/copy/move/delete/test for existence
• directory manipulation: create/copy/move/delete/test for existence
• logging
• platform detection
• executable presence detection
• variables
• read-write access to environment variables.

Reasoning about ERE: ERE regexes are simple and can run without modifications on most regex engines.

Suggested syntax: a list of lines in the form COMMAND ARGS. Expressions syntax: sequence of tokens (values and functions) in suffix notation (as it is very easy to implement). E.g.

# It is assumed that before running this code PM changed directory to plugin root
IF PLATFORM IS windows
    READ install_windows.bat TO &variable
    SUBSTITUTE /[A-Z]:\\vim\\vimfiles/ WITH . IN &variable
    WRITE &variable TO install_windows.bat
ELSE
    READ install_linux_dev.sh TO &install_script_contents
    SUBSTITUTE (~ .vim SEPARATOR 2 JOIN) WITH . IN &install_script_contents
    WRITE &install_script_contents TO install_linux_dev.sh
END_IF
CREATE_DIRECTORY plugin
IF PLATFORM IS windows
    RUN_SHELL cmd .\install_windows.bat
ELSE
    RUN_SHELL sh ./install_linux_dev.sh
END_IF

. Expressions here:

1. Everything not starting with /, ', ", (, &, $, a digit or an uppercase letter is a raw string literal. It will end with either <Space>, ) or a newline/end of string.
2. If expression starts with ( it is a complex expression which is expected not to contain any ) except for inside quoted string. Such expressions consist of a sequence of expressions except for parenthesized ones.
3. Expression starting with /, ' or " is a string literal with different style of quoting: / uses repeated quoting (///http://abc.def///): it starts with any positive amount of slashes and ends with the same amount. ' uses regular vim quoting style: 'abc''def'. " uses JSON escapes (note: only JSON escapes).
4. & denotes a variable name. This name ends in the same fashion as raw string literal.
5. $ denotes an environment variable name.
6. Uppercase letter denotes a function or built-in constant. Unless function does not use the stack it is only allowed inside parenthesis.
7. Digit denotes an integer type.

Built-in types: string, integer, boolean.

Note about IF: its syntax is IF {expr} [{operator} {expr}]. IF PLATFORM IS windows does not use infix notation. Also accepted IF (make command EXISTS).

Complete list of commands:

# Comment
IF {expr} [{operator} {expr}]
ELSE_IF {expr} [{operator} {expr}]
ELSE
END_IF
# Operators: string comparison: IS[_NOT]; number comparisons: =, /=, <=, >=, <, >.
READ {expr} TO &var
# Note: TO &var, not TO {expr}
WRITE {expr} TO {expr}
SUBSTITUTE {expr} WITH {expr} IN &var
SUBSTITUTE {expr} WITH {expr} IN $var
CREATE_DIRECTORY {expr}
MOVE {expr} (TO[_DIRECTORY] {expr}|HERE)
# MOVE {expr} HERE is like MOVE {expr} TO_DIRECTORY .
COPY {expr} (TO[_DIRECTORY] {expr}|HERE)
DELETE[[_EMPTY]_DIRECTORY] {expr}
# DELETE_EMPTY_DIRECTORY refuses to delete non-empty directories
SET &var TO {expr}
EXPORT {expr} AS $var
CHANGE_DIRECTORY_TO {expr}
RUN_SHELL {expr}…[PIPING_TO &var] [PIPING_FROM &var] [EXPECTING_EXIT_CODE {expr} | IGNORING_EXIT_CODE] [IN_DIRECTORY {expr}]
PRINT {type} {expr}…
# types (sorted by level): DEBUG_INFO, MESSAGE, WARNING, ERROR
ABORT_WITH_MESSAGE {expr}…
REQUIRE {version}
# Aborts if current implementation is older then {version} or if it has different major version. In the latter case developer may choose to run in compatibility mode though.

Complete list of functions:

PLATFORM
SEPARATOR
CURRENT_DIRECTORY
{path} {type} EXISTS
# takes two values from the stack. Used like this:
# IF (make command EXISTS)
# IF (plugin directory EXISTS)
# IF (addon-info.json file EXISTS)
{component}… {separator} {number_of_components} JOIN

Note about VimL implementation: current directory and environment variables manipulation capabilities should not alter user environment (i.e. restore state afterwards).

---

Reasoning: it is assumed that hooks will be used in cases when vim is hard to use: e.g. if you want parallel installation you either have to spawn a separate vim instance or parse hook with python/ruby/etc, same for installation using command-line utilities.

As hooks are meant to be cross-platform you may not expect sed or coreutils being there. Thus main function of the DSL is to contain cross-platform implementation of SUBSTITUTE and file/directory manipulation functions.

[MarcWeber]

I don't like creating a new DSL right now. If you have time and want to maintain it I don't mind. I'd like to understand how I'd like see Vim's future first which will take a couple of month - I have more important stuff to be done at the moment. I agree that we need some "declarative" way to specify some hooks, so that hooks can be used by multiple manager solutions.

[Shougo]

Yes, the current hooks features are only for VAM. But I don't want to use new DSL...

[ZyX-I]

This variant is not declarative.

[MarcWeber]

To keep the long story short:
If there is something to be setup - please tell me why its the task of
the plugin manager to run make. Why don't plugins check themselves and
contain all the logic they need - beacuse they know best.

Then we could introduce a new plugin "build-utils" which runs make or
such - or introduces a DSL.

But why do we as plugin manager maintainer have to care about all
pieces?

Eg vim-addon-async does compile its sourcefile itself - all you need is
gcc (thus on win you're in trouble usually).

But the real problem is VimL - its not suited to do such cross platform
tests. Eg how to calc md5sum of a distributed or downloaded executable
to suggest using a newer version from authors homepage - because
installing gcc is cumbersome ?

Another solution would be installing a binary version of the plugin / or
a binary extension - such as vim-addon-async-binaries-for-windows and
have vim-addon-async look for that - or suggest installing it as needed.
(I just never did it)

Of course this does not work for tools which interface with setup such
as python/ruby/.. libraries on the target pc, there you have to build
from source.

I feel we should try to hard to keep our workload as small as possible

• and keep things separate - if we have time left we can still support
  and contribute to external additional solutions ?

I got it, there is one reason why the "plugin manager" must do all the
work: "async installation ..." ? Is it worth the effort - I don't use
many plugins which I have to compile.

[ZyX-I]

Yes, the current hooks features are only for VAM. But I don't want to use new DSL...

There is a reasoning behind DSL. Personally I am fine with the current state, but for the reasons explained it does not work well as a universal format. I would expect alternative solution, “don’t want” is not constructive.

[Shougo]

Well, but we must make parser for parsing DSL.

[ZyX-I]

If there is something to be setup - please tell me why its the task of
the plugin manager to run make. Why don't plugins check themselves and
contain all the logic they need - beacuse they know best.

1. You suggest to violate responsibilities. There is no good place for such kind of logic in vim directory structure.
2. Users do not expect a simple mapping taking 1 minute to run because it runs gcc -O3. They expect that installation is the heaviest process except for some rare kinds of plugins which need to e.g. index your project.
3. Plugins are not supposed to have write access to the place where they are installed. The fact that most plugins are installed local to user is not going to make plugins in the portage tree (and other similar repositories) disappear. Note that pip is perfectly fine with installing system-wide, so are most of the other language-local package managers.
4. “Checking themselves” means more code. More code means more bugs. make knows when file was updated. VAM knows when it updates plugin. Once you put this logic into the plugin it is easy to make “added a file, but forgot to add a check whether this file changed” mistakes in logic that checks whether plugin was updated. Running make always is not an option.
5. Plugins do not have this functionality. Why portage tree contains patches? Because it is portage for installing software, not software for being installed by portage.

I got it, there is one reason why the "plugin manager" must do all the
work: "async installation ..." ? Is it worth the effort - I don't use
many plugins which I have to compile.

It does not matter how many plugins are supposed to be compiled. Once you have said async the whole installation process should be async. It does not matter whether or not there is more then one process requiring hooks (_WARNING_ hooks are not only doing compilation), in any case there is no good way to use vim from background. I should also add one feature to #82: automatic creation of specs for RPM/deb and ebuilds.

[ZyX-I]

Well, but we must make parser for parsing DSL.

I designed it simple enough for it to take at most a few hours in any language I have ever written any projects with (including C, though “few hours” is much greater in this case).

[Shougo]

OK.

[MarcWeber]

Excerpts from ZyX-I's message of Thu Jan 09 15:10:38 +0100 2014:

If there is something to be setup - please tell me why its the task of
the plugin manager to run make. Why don't plugins check themselves and
contain all the logic they need - beacuse they know best.

1. You suggest to violate responsibilities. There is no good place for
   such kind of logic in vim directory structure.
   Why not create a new location? Just because it doesn't exist does not
   mean it doesn't make sense to start such.

But let's summarize:

• There are 2 steps for each plugin: 1) install 2) setup (compile)
• when using gentoo portage setup should happen directly after 1),
  when using VAM being lazy would be fine, too. (Why compile if you
  don't use plugin at the moment?)

The gentoo/portage pypi whatsoever thing clearly indicates that it would
make sense to provide kind of declarative standard, such as

'windows-setup': {
  requires: {
    'cygwin': { 'virtual-package':  'gcc' }
  }
}
'linux-setup': {
  requires: { 'gcc' }
}

(This is not even a draft ..), then both VimL VAM like managers or code
generators generating gentoo ebuilds could benefit eventually.
You could add special keys meaning "this requires python packages A,B,C
.." (from PyPi) etc.

1. Users do not expect a simple mapping taking 1 minute to run because it runs gcc -O3.
   Well, you could show them a message such as "compile it [y/n]", then
   they know what will happen. Users don't expect Vim to compile package
   A,B,C if they don't use it at the moment (eventually)
2. Plugins are not supposed to have write access to the place where
   they are installed.
   Good point, we have two use cases: User installs and system wide
   installs. That's why we eventually should think about "decalarative
   approaches" to finally end all this the right way.
3. “Checking themselves” means more code.
   How much?

map call supporter#IfReady(g:plugin.declarative_list_of_things_to_be_done_first, 'then execute this viml code)

but g:plugin is bad, beacuse tools creating gentoo ebuilds cannot read
it. Thus I vote for addon-info.json.

make knows when file was updated. VAM knows when it updates plugin.
So this means we need different commands for updating and first
installation? You might need "make uninstall && update && make build &&
make install" - so that no traces of old versions are left? ...
It heavily depends on the plugin eventually.

if the plugins had update and install hooks they could do what they
think is best. We could make them know when they got updated.
(or they could check the git hash themselves - thus we could introduce
an api telling plugins about their versions ..)

Again - the addon-info.json would solve it (its the only sane way IMHO),
because the plugin or plugin managers or gentoo ebuild (or nix
derivation) tools can use that information.

Does it make sense to start a list of packages and patches to get a sane
idea what we need?

Marc Weber

[MarcWeber]

Bottom of this page:
http://l/vim-wiki/wiki/topic/vim%20plugin%20managment.html

I've tried describing the problem briefly, let's collect what we need to
find the best solution - there is no hurry.

I'll add a pointer to vim-pi's readme so that we can keep a list of
ongoing discussions (on bitbucket)

[ZyX-I]

• when using gentoo portage setup should happen directly after 1),
  when using VAM being lazy would be fine, too. (Why compile if you
  don't use plugin at the moment?)

Because otherwise it is taking way too long time. Why download if you don’t use plugin ATM?

(This is not even a draft ..), then both VimL VAM like managers or code
generators generating gentoo ebuilds could benefit eventually.

Are we discussing transfer of the responsibility of compiling package from VAM to plugin or discussing a new hooks format?

Well, you could show them a message such as "compile it [y/n]", then
they know what will happen. Users don't expect Vim to compile package
A,B,C if they don't use it at the moment (eventually)

They do expect it when they install. When they instruct plugin manager to install. Every package manager does compilation during installation.

Adding compilation time to download time is acceptable, especially for vim plugins where compilation (if any) is usually faster then download. Even if it is 1 minute gcc -O3 on ten C files it is still acceptable because installation is expected to be rather slow, even by binary distributions users.

Once PM asks me whether to compile it I will start looking for alternatives. It forces user to do heavy context switches in this case.

Good point, we have two use cases: User installs and system wide
installs. That's why we eventually should think about "decalarative
approaches" to finally end all this the right way.

We have one use case: just installs. The only difference are paths. I do not see any arguments for splitting installs: it is overcomplicated without benefits (lazy loading is not the one).

How much?

 map <f2> call supporter#IfReady(g:plugin.declarative_list_of_things_to_be_done_first, 'then execute this viml code)

You have not added supporter#IfReady() code. Note: if supporter is not in vim distribution then you loose rather big piece of auditory: it will be neither advertised by vim user manual and other docs nor accepted by developers that think that adding dependency for one function is too much.

So this means we need different commands for updating and first
installation? You might need "make uninstall && update && make build &&
make install" - so that no traces of old versions are left? ...
It heavily depends on the plugin eventually.

We already have different commands for updating and first installation.

Does it make sense to start a list of packages and patches to get a sane
idea what we need?

Using vim search in scmsources.vim and reading section in patchinfo.vim was fine for me. I did not add hooks actively enough to have all packages requiring hooks have them and nobody else was adding hooks.

[ZyX-I]

I'll add a pointer to vim-pi's readme so that we can keep a list of
ongoing discussions (on bitbucket)

If we agree to move there I will export all issues and shut down any discussion here.