improve support of a scripting language

[ghost]

The main target of this issue is to collect ideas around improving the scripting ability of mutt.

make mutt completely scriptable by a scripting language

It would be great if mutt would allow configuration in a scripting language. I think the main advantages of this point is readability for complex things and making mutt configs more powerful.

• Mutt does kind of allow functions by using external shell commands/scripts (this way, even perl/lua oneliners are possible, i think.)
• Neomutt has support for if statements

what is the problem with the existing solutions?

• if we want to use functions, we need to call bash/perl/whatever, which is not very resource efficient
• either we have to use one liners, or we have to create a sperate file which contains those external functions. I personally would love to see everything in one file.
• we have no arrays, no type system (e.g. no integer variables) in mutt.
• we have glue code like: !zsh -ic FunctionToExecute<enter> which consumes just space without
  poviding information.

Lua

advantages

• to cite the lua.org website:

64-bit Linux, the Lua interpreter built with all standard Lua libraries takes 245K and the Lua library takes 420K.

• higher order functions maybe can provide elegant solutions to solve complex problems

Python

advantages

• way more libraries

disadvantages

• slow

@flatcap worked already on integrating lua

• github repo for lua samples
• devel/lua-demo branch

some general ideas

• lua usage

Questions

• Are there other languages we could use? What about Python, ruby, etc. ?
• Why is lua currently our best approach?
• What use cases can you think of?
• should we directly replace the current muttrc with a lua config? or should we add a option embedding e.g. lua into the current muttrc ?

lets discuss :)

[flatcap]

Why LUA?

It's a simple language to understand.
It was very easy to integrate into C.

By contrast, Perl required me to define interfaces and create wrappers.
I couldn't find a simple way to do simple things.

What can LUA do?

It's a programming language like all the others: variables, loops, etc.
It comes with libraries to read files, talk to databases, etc.

What does the demo version do?

To be useful for scripting, I decided I needed to be able to:

• Call a LUA script from C
• Pass variables from C to LUA
• Read LUA variables from C
• Call a C function from LUA

The last item makes the scripting really useful.
We can write wrapper functions, to manipulate emails, and export them -- allowing a simple script to automate mutt.

[pickfire]

@flatcap Yeah, I notice Lua have the speed (almost comparable with C/C++), not sure why everyone choose python. (I use python here but really hoping to use Lua)

python seems nice as most systems have it by default but I hate having 2 different versions of python with their respectives libraries which wasted me around 200MB each.

Not sure about ruby and perl, from what I know, I think they are mostly used to do something related to text and strings.

Fun fact: Neovim port most of the vim scripts to lua https://github.com/neovim/neovim/wiki/FAQ#design

[flatcap]

This is a list of everything I could think to script.
There's a lot of overlap / recurring themes.
Some of the ideas are sensible, some not :-)

General

All scripts would be passed:

• neomutt version string
• current date

Have access to functions:

• patch_supported()

Config

Allow script to be embedded in normal Mutt config files, e.g. like PHP:

Pass in:

• username
• user's home dir
• current config file
• line number

<?lua
    if (MUTT_VERSION > "20160916") then
        mutt_config("set sidebar_visible")
    end
?>

Lua's print() function could be overridden to call mutt_message()

Index

A one-pass processing of the mailbox.
Call the script once for each email.

Add new expandos, %L1, %L2, etc to be filled by the script.

Pass in:

• Copy of all the fields
• Current account
• Current mailbox
• Current email

Define functions to:

• Colour:
  • All of the line
  • One field
  • Regex on field, or line
• Modify a field (‡)
• Hide / Delete an email (‡)
• Set the fold level of thread
• Set score of email
• Lookup address -- is_alias()
• Add/remove icon for email
  • Flag GitHub notifications
  • Add coloured stars

Can optionally disable Mutt's colouring

Pager

A one-pass processing of an email.
Call the script once for each line.

Define functions to:

• Query the headers
• Colour all, or part, of the line
  • Look for markers like "^diff"
  • Look for a signature: "-- "
  • Heuristics for languages
• Modify a line (‡)
• Hide / Delete a line (‡)
• Insert a new line (‡)
• Set the fold level

‡ Actions are non-destructive

Sidebar

Sidebar mailbox entries
Define functions to:

• Colour all, or part, of the line
• Visibility - hide a mailbox
• Modify a mailbox name (‡)
• Change Counts of: flagged, read, total emails (‡)

‡ Actions are non-destructive

e.g.

• Hide folders with no new mail
• Emails in "drafts" are always unread

Hooks

These hooks might be useful to override with a script:

• account-hook
• close-hook
• folder-hook
• reply-hook
• send-hook
• timeout-hook

Most useful, would be a new-mail-hook.

Pass it:

• Account info
• Mailbox info
• Email headers
• Current date
• Size
• Number of attachments
• List of attachments

Allow it a small set of functions:

• set_flag: read, flagged
• set_score
• email: move, copy, delete
• search body
• search attachment
• stop processing

Status Bar

Generate: $status_format, $ts_status_format

Pass info about current:

• account
• mailbox
• email
• max length

Testing framework

Need to expose:

• index functions
• sidebar functions
• variables

Be able to:

• call functions
• send keystrokes
• get error codes returned from functions
• save the results

Automation:

• Run tests on Travis
• Need public access to test mailboxes

Create a test rig, built from Mutt code, that takes a mailbox and a script.

• test-rig test.mbox config.lua
• machine-parseable output

[guyzmo]

About the lua integration in the config files, I'd rather prefer to follow vim/neovim's way of adding a mutt command: lua (or maybe something like ex/exec or script) to execute discrete commands:

lua mutt_echomsg("Hello world!")

that would work on a single environment, so that:

lua x, y = hello, world
lua mutt_echomsg(x .. y)

then we could extend using the following UNIX standard syntax pretty easily:

lua << END
mutt_echomsg("Hello world!")
END

because mutt config is not a tag based language, I'd rather see the above syntax instead of suggested:

<?lua
?>

[guyzmo]

I have had a little thought on lua integration, and I've read a bit on the neovim approach.

New mutt commands

• lua: execute a line of command in the rc file or the enter-command contexts
• luasource: source a lua file

eventually make it possible to introduce a new syntax in neomutt to parse multiline lua scripts inline of an rc file:

lua << END
…
END

API

As a start we could consider having only a couple of APIs

• mutt_parse_exec which will run any command passed as a string parameter into the current context.
• mutt_parse_rcline which will run any command passed as a string parameter as a config file line (or enter-command context).

to make it possible to run context-aware commands, extending the API to support arguments and prevent any interactive enquiring will be necessary, cf #294.

Keep the API close to the C code

• exposing functions and globals using. That would mean exposing all mutt functions and settings from init.h's Commands[] and globals.h variables defined with WHERE, using a mutt_<function_name>() or mutt_<setting_name> symbol.
• exposing contextualised functions using mutt_<context>_<function_name>() and context dependent settings (if possible) using mutt_<context>_<setting>.

eventually turn it into a nicer lua API using lua itself

we could then have some code to expose a nice OO-like approach to the API using a mutt singleton with a full API exposing the globals and rcline-level functions as methods, and the context aware ones as a mutt.context member.

Something in the idea of lua-client

Add the ability to create new commands and hooks

Finally, the toughest part will be to have mutt integrate lua code from mutt, by adding the ability to register new:

• commands — either globals ones, or context-dependent ones,
• settings
• context hookss — though registery a new command should make it hookable.

[guyzmo]

I have implemented the first step in this branch:

• two new commands:
  • :lua to run one line of lua script,
  • :lua-source to load a lua script file
• two mutt API methods exposed in lua:
  • mutt.message() and
  • mutt.error()

as a next step exposing existing init.h functions in lua.

[somini]

This is the branch: https://github.com/guyzmo/neomutt/tree/feature/lua-scripting, not the one in the neomutt organization.

I tested this and got mutt.error to work. What's the mutt.print signature?

I also tested the lua-source with this script:

print(1+2)

And mutt entered pager mode, even though the terminal borked and I had to <C-l> it.
Great work, @guyzmo

[guyzmo]

Oops, my bad, it's print.message() updating my former post.

Here's a few examples:

:lua mutt.message("Hello World!")
→ Hello World!
:lua mutt.error("Fubar!")
→ Fubar!
:lua print("Get out!") → Get out! (on real stdout)

Basically if you set in your muttrc:

color error brightred default
color message brightcyan default

error messages will be in red, and the others in cyan.

[guyzmo]

just added the mutt.call() API, so that you can do:

:lua out = mutt.call("set visual")
:lua mutt.message(out)
→ visual=vim
:lua out = mutt.call("set visual="nvim"")
:set visual
→ visual=nvim

😎

[somini]

I can confirm this, it works!

[guyzmo]

updates to the API:

former mutt.call is now mutt.enter (which is the : binding to enter-command in mutt):

mutt.enter("color quoted brightblue default")

and new mutt.call API is as follows:

mutt.call("color", "quoted", "brightblue", "default")

but it is also exposed as straight lua methods:

mutt.command.color("quoted", "brightblue", "default")

---

the set API now supports all internal types, cf this gist:

lua mutt.set("visual", "less")
lua mutt.set("connect_timeout",42)
lua mutt.set("arrow_cursor", true)
lua mutt.set("abort_noattach", mutt.QUAD_ASKNO)
lua mutt.set("alias_file", "~/muttrc")
lua mutt.set("mbox_type", "Maildir")
lua mutt.set("sort", "date")
lua mutt.set("mask", "!^\\\\.[^.]")
lua mutt.set("to_chars", "+TCFL")
lua mutt.set("from", "fubar@example.org")

and it will throw exceptions at you if you do it wrong!

[guyzmo]

Added support for myvars.

lua mutt.set("my_fubar", "Zaphod Beeblebrox")

[guyzmo]

Added -B parameter for batch mode (so you don't get the ncurses UI up when running a script, so far it's really useful for testings).

It's exiting before the part sending mails in main.c so I don't think it's optimal.

[guyzmo]

I'm pretty much done with the first part of the implementation.

Please review the code, test the features, break it, and come back commenting and cristicising it!
I'm all ears

🤗

[somini]

So, I spent the afternoon playing around with this. Once we can integrate hooks into executing LUA code, the sky is the limit!

Some context first.

I use mutt with various accounts, so instead of manually managing the "mailboxes" for new accounts, I put them in a hierarchy and wrote a Python script to create the mailboxes command for me. It's ugly, but it works for my setup. Here's the hierarchy

• Work
  • Group1
    • Employer1
      • INBOX
      • ...
    • Employer2
      • INBOX
      • ...
• Personal
  • GroupA
    • AccountA1
      • INBOX
      • ...
    • AccountA2
      • INBOX
      • ...

This lets me call my mutt wrapper mutt-profile Work that sets the "folder" to that path and I only get work emails in that window. Notmuch works automagically, since the ".notmuch" folders are separate. Avoids finding personal emails in work-related searches.

The Python script receives a profile and creates the mutt "mailboxes" command. Here it is: https://gist.github.com/somini/84c5d624d0f14784e60da3817530ab48

I converted it to LUA (minus some improvements) and it works perfectly. Since I use the mutt bindings, it can be much improved and doesn't suffer from string escaping bugs. Here: https://gist.github.com/somini/c13f73d60d3a550ffa69314048daa2ec

[somini]

I do have a bug report, if I set mutt.set("sidebar_visible", "yes") in a script, the variable is on but the sidebar is not visible.

[flatcap]

mutt.set("sidebar_visible", "yes") in a script, the variable is on but the sidebar is not visible.

The options in init.h have various refresh flags.
When $sidebar_visible is changed it needs to trigger a redraw.

[ghost]

Regarding the PR from @guyzmo :

in general nice work :) I don't use mutt much, but i think it would provide some great readability advantages when we can replace our conditional features with lua. What do you think about that?

[guyzmo]

@somini indeed, the mutt_option_set() function is not caring for redrawings and resorting. That will need to be added.

[guyzmo]

@toogley yes, I believe that the if construct or the conditional formats could be replaced by lua stuff, in a similar fashion to the hooks implementation. But we'd need to be careful not to offload too much to lua at risks of slowing mutt down! Especially when running a callback once for each mail (for each displayed mail, it should be fine, though).

[guyzmo]

hurra 🙌👍♥