Add source command

[alexherbo2]

Source file.

[ALSchwalm]

Is -source what your looking for? I think it's just not documented yet (I'm not sure how complete it is at the moment).

[xiaq]

Yes, we have -source :) The reason that it is marked experimental is that it does not interact well with the static checker. For instance, if you have the following in a.elv:

x = haha

then you should be able to do -source a.elv; echo $x. But you cannot yet.

I will close this issue when this problem with -source is resolved, and it will be renamed source at that time.

[krader1961]

FWIW, I stumbled across this issue because I found I needed source to implement a function in the top level scope but without putting it in my rc.elv config script. Namespaces are great and should be used for anything meant to be used by others or that is non-trivial. But for defining personal functions to be used interactively it would be nice if there was something analogous to fish's autoloaded function mechanism. Without that I have to either put those functions in my rc.elv or put a -source statement in that script. The latter is preferable to keep things organized.

[krader1961]

I'm wondering if we really need a source command. What if the use command supported the equivalent of Python's from module import *? That is, import the module with its members qualified by the namespace but also alias all the members into the current scope? That is, if I did the hypothetical use nx as * I could reference the embedded function as either nx:nx or nx and $nx:status or $status. Obviously this is dangerous but no more so than source and is one less command that needs to be implemented. Obviously, source is slightly more general in that you can give it an arbitrary path. But that could potentially be solved by allowing absolute pathnames or explicitly relative (i.e., they begin with ./) with use to mean don't search the usual locations.

[zzamboni]

@krader1961 I really like your idea of being able to specify which symbols to import in the use statement. Something like Clojure's :refer argument to require. For example:

use re           # default, everything is imported under the re: namespace
use re [replace] # `replace` is also "linked" into the current namespace
use re [:all]    # or some other special value, to specify that all symbols should be linked into the current namespace

[xiaq]

Importing names from modules is not hard to implement, but I doubt its usefulness beyond being able to put your wrapper functions ("aliases") in a module and pouring that into rc.elv. Go does not have this function and I like its absence, because it is now always clear which names are imported because they are always namespace-qualified. It is also the style recommended by Google Python Style Guide.

Also, importing names that are not known beforehand (use some-mod [:all], borrowing @zzamboni's syntax) poses a unique challenge for Elvish.

To start with, Elvish always compiles a chunk of code before evaluating it, and part of the compilation phase is checking variable names. In dynamic languages like Python, use of undefined variables is a runtime error. In Elvish, it is a compilation error and will prevent the whole chunk from executing. For instance, the following code won't do anything (provided that you didn't define $undefined-var):

echo before
echo $undefined-var
echo after

whereas the following Python code will print out before and then a traceback for a NameError exception:

print('before')
print(undefined_var)
print('after')

Now, to compile code like the following

use some-mod [:all]
echo $some-var

Elvish has to load some-mod in compilation phase in order to check whether $some-var has been defined. Currently, Elvish only loads the module in the evaluation phase (which implies that access to names like $some-mod:some-var are not checked, but let's not talk about that for now). It seems that moving the loading procedure to the compilation phase will solve the problem. Now let me talk about Elvish's editor.

If you type something like echo $undefined in the editor, Elvish will complain that $undefined is undefined right away, before you hit Enter (in fact, it complains about $u, $un, $und, ... all the way up). That error message is driven by the compiler: Elvish parses and compiles your input on every keystroke, and notifies you of any errors.

Now if the module loading is to be moved to the compilation phase, that means that when you type use some-mod, Elvish will try to load all of s, so, som, ... up until some-mod. This makes the compilation phase much more expensive because module loading usually involves reading from the disk.

To summarize the challenge in supporting use some-mod [:all]:

• Elvish always checks variable names in a compilation phase ("statically") before actually evaluating the code.

• Supporting "importing all names" means that the compiler now needs to look at the module file. Currently this is only done during the evaluation phase.

• Making the compiler look at the module file is expensive because the editor compiles the input on each keystroke.

Code references:

• For the static checking of variable names, see here.

• For the implementation of use, see here.

---

On the other hand, there is no problem with use some-mod [name1 name2], because that gives enough information to the compiler. But again, I doubt its usefulness beyond being able to put aliases in a separate module.

---

So how can the current -source work?

The way -source now works is that it does not inform the compiler of new names. That means that if you have the following in a.elv:

foo = bar

The following code won't compile:

-source a.elv
echo $foo

because -source makes no effort in telling the compiler that $foo might be defined in a.elv. (If you run this from the editor, you need to use Alt-Enter to insert a literal newline.)

However, if you run those two lines as separate commands from the editor, it does work. This is because Elvish creates a new compiler for every new command, and the new compiler knows all names that are defined.

This badness of behavior is why I chose to mark source as internal (by calling it -source).

Code references:

• For the realtime compilation from the editor, see here.

• For the implementation of -source, see here.

[zzamboni]

@xiaq thank you for the comprehensive explanation. As you say, the only benefit for importing names would be to allow defining "top-level functions". But the usefulness of this can be debated. At the moment, Elvish forces the user to be explicit about top-level functions. For example, to use my cd library as the default cd command, I have to do the following:

use dir
fn cd [@dir]{ dir:cd $@dir }

Which makes my configuration more explicit about what is being overriden. This is good for clarity at the expense of verbosity and exposing some more "implementation details" in my top-level configuration.

If dir.elv was allowed to define top-level functions, it could automatically define that function. One could argue this is useful as it allows more automated configuration (just load the package) at the expense of being less explicit about what the module is doing.

I think being able to specify the names to import would hit the right balance: abstract away implementation details (such as how to wrap a function) while still making the configuration explicit about what names are being imported:

use dir [ cd ]

[xiaq]

For uniformity, it is better if we can import variables, not just named functions. Named functions are really just variables who names start with '&', so it makes more sense to support importing variables.

Also, I find it clearer to separate the loading of modules (already supported by use) and the importing of names from that module (the functionality being discussed). I think it will be nicer to have a separate (say) refer command, so that what you propose as use dir [ cd ] will be written as:

use dir # loads module "dir"
refer dir ['&cd'] # aliases $dir:&cd as $&cd

[krader1961]

For uniformity, it is better if we can import variables, not just named functions....

Agreed. But do we really need a new command? I don't see the reason to implement refer dir ['&cd', 'pwd'] rather than use dir ['&cd', 'pwd']. Also, I want to make sure we're all on the same page in that by "aliasing" the name we're talking about binding a new name, in the local scope, to an object (var or function) in a namespace. Such that changing the value assigned to $pwd or $dir:pwd is reflected in the other var name.

[xiaq]

@krader1961 We have the same definition for aliasing.

There are three reasons for a separate refer command:

1. Module loading and name aliasing are two orthogonal concepts. Giving them separate constructs promotes composibility in general.

2. It should also be possible to alias names from pre-loaded namespaces like e: for external commands, or E: for environment variables. For instance, refer E [PATH] aliases $E:PATH to $PATH. Since E: is not really a module, saying use E [PATH] is confusing because there is no module loading involved. It will complicate the logics for use.

3. The semantics of use mod [name1 name2] is non-obvious. It loads the module named by mod and imports $name1 and $name2 from that module, but is $mod:name1 and $mod:name2 or any other name $mod:name3 also valid? In other languages, the import construct only does either of importing individual names into the current module (in Python, from mod import name1, name2) and importing the whole module as one namespace (in Python, import mod), but not both.

[krader1961]

Good points, @xiaq. Your first two points in particular are worth noting since composition is a useful feature in general. I hadn't considered the case of aliasing objects from semi-magic namespaces like E: or e: into the local scope. It's not clear doing that should be encouraged but it should be supported if we support aliasing objects from other namespaces imported by use.

The refer namespace [name1 name2...nameN] syntax and the refer command name seem confusing to me as a native English speaker. What about import [name1 name2...nameN] from namespace? Or alias [...] from namespace? Using either import or alias is probably clearer to a native English speaker. However, alias may be a poor choice due to its behavior in other shells like bash, zsh, and fish.

As for your point 3 I recognized that difference from Python behavior but deemed it acceptable given that Elvish is only influenced by Python. So the dual behavior of my proposal being equivalent to Python's import mod and from mode import name1, name2 could be justified. Albeit the ambiguity may not be ideal.

[xiaq]

The namesake of refer is from Clojure, although Lisp people are not exactly known for giving things friendly names :)

As you pointed out, alias is not a good choice for its potential confusion with alias in traditional shells. I avoided import because ImageMagick provides a command named import.

[krader1961]

In light of commit ff7c500 that formally deprecates the experimental -source builtin this issue should probably be closed as "will not implement".

[rdw20170120]

I need this capability, to customize my Elvish shell to configure my PATH, to activate a Python virtual environment, to configure other tools, etc. If Elvish intentionally lacks a source command or equivalent, then how is such shell session configuration supposed to be achieved.

Please note that my (and many other's) use case involves putting all the relevant shell configuration into a code repository as part of a project. Therefore, it MUST NOT involve modifying the user's configuration (in XDG directories and files).

Can this be done in Elvish, or must I look for another shell?

[hanche]

@rdw20170120 Instead of source ⟨file⟩, you can do eval (slurp < ⟨file⟩). The only caveat being that variable assignments and functions definitions made in ⟨file⟩ will not be reflected in the calling namespace. But there are edit:add-var and edit:add-vars to manipulate the interactive namespace from a script. (Environment variables are different: They are not lexically scoped, so no problem there.)

[rdw20170120]

@hanche, thank you for that insight; I will have to experiment with it.

My objectives are primarily two-fold: I must set crucial environment variables to configure this shell session for work on the current project, and I must define important functions to facilitate that project work.

However, I wonder whether I actually have to make the functions (modules) "stick" during my activation process. I am starting to think that I may not have to do so. Perhaps it will be enough to set PATH to find the project's scripts and tools, plus set a few supporting environment variables as needed. Then perhaps the individual scripts can use modules as necessary, relying upon relative references (or absolute references if that becomes available).

This may also "fix" the issue of module cacheing. If the individual scripts are executed in their own Elvish process (the default), then the modules that they use will be cached only for that process. If I edit a module, then rerun a script that will use it, then the script will get a fresh copy in the new process.

Where this becomes problematic is where I want to use functions directly from the shell, rather than merely in scripts. I could use the defining module directly from the shell, but then I am back to simply use ./activate to get all of what I want. It is certainly worth experimenting to find the best balance.

[hanche]

@rdw20170120 I think perhaps the best thing for a project like that would be to provide a module to be installed in, say, $E:HOME/.config/elvish/lib/the-project.elv (or in $E:XDG_CONFIG_HOME/elvish/lib/the-project.elv if XDG_CONFIG_HOME is set in the environment – using elvish conventions for environment variables here). It could offer, optionally, to insert a line use the-project in the user's rc.elv file. If the project is likely very central to the user's workflow, you could suggest a shorter module name, as in use the-project p so the user can invoke the project's functionality just using the p: prefix. And finally, the module could contain a function that will run a suitable edit:add-vars command to munge the interactive name space should the user prefer this. And of course, the module could easily manipulate the environment to its heart's content.

In short, better not to pollute the user's interactive name space needlessly, without the user's express consent.

[rdw20170120]

@hanche: I appreciate your comments. A crucial part of my design is to not compromise the user's account and configuration at all. Anything that my project/tool (or any other project/tool) might install within XDG directories could compromise the user, and so is out of the question.

My approach is to have the user open a shell, change directory to the root of the project repository that they cloned onto their machine, then invoke an activation script of some kind. Having done that, the user has certainly given their express consent to manipulate the environment. But it is essential that ONLY that shell SESSION has been reconfigured and might be "compromised". As soon as the user invokes exit to leave that shell session, all impact from my project/tool is GONE. I hope that you can see that this is far more cautious and protective of the user than injecting a module and configuration within their XDG directories.

Actually, I would never expect to put any code into the users $E:XDG_CONFIG_HOME/elvish/lib directory; maybe it could be done with great care and some appropriate conventions. Many programming language platforms have run into namespacing issues because numerous authors wrote extensions that eventually collided, e.g. Java, Python, etc. Some platforms established naming conventions to avoid the collisions, sometimes involving creating whole directory trees oriented towards subdividing the namespace. For example, what happens to the user when I deploy $E:XDG_CONFIG_HOME/elvish/lib/the-project.elv and then you deploy another $E:XDG_CONFIG_HOME/elvish/lib/the-project.elv? Obviously, we could do something like $E:XDG_CONFIG_HOME/elvish/lib/rdw20170120/the-project.elv and $E:XDG_CONFIG_HOME/elvish/lib/hanche/the-project.elv. But there are numerous conventions that we might individually adopt, and we might still collide if our conventions do not sync up. This is, of course, the challenge that the Elvish Package Manager (EPM) would presumably solve.

As @krader1961 and others have said elsewhere in these issues, most of us have learned to avoid any tool or project that is rude enough to directly mess with our user configuration such as in rc.elv. Beyond that, my goal is to also minimize and isolate any manual changes that the user must make to use my project/tool. A primary design goal is that relevant changes/configuration are kept within the repository, and therefore are isolated and versioned.

I realize that you have only abstract information about my project, and that makes it very difficult to comment. Another crucial part of my "tool" is that an individual user will (ideally) use multiple copies of it, one in each of their project repositories. God willing, someday everybody's project repository might contain a copy of my tool (I can dream, right?). So an individual user's account on their workstation might have a dozen copies of my tool, embedded in a dozen project repository clones, with various versions involved. Each such use of my tool empowers the user to create a custom activation script, suited to the content and tools for that project. Trying to manage such multiple copies and configurations within $E:XDG_CONFIG_HOME/elvish--all from the same tool provider--just does not make any sense to me.

[krader1961]

As @krader1961 and others have said elsewhere in these issues, most of us have learned to avoid any tool or project that is rude enough to directly mess with our user configuration such as in rc.elv.

There might be a misunderstanding here. My point was that installing a shell extension should not automatically modify my shell config file(s). There is absolutely nothing wrong with requiring the user to manually add a block of code to their shell config file(s). Having said that, there is nothing wrong with expecting the user to cd to the root of your project and do something like use ./activate. Note: There is nothing stopping the user from adding those statements to their rc.elv so that activation happens every time they start an interactive Elvish shell and they therefore don't have to run those two commands interactively.

I also don't really understand what you're getting at with respect to namespace collisions since that can happen even with your preferred approach. There is absolutely nothing that keeps the user from defining env vars or shell functions that conflict with those defined by your project.

[rdw20170120]

@krader1961 has rightly suggested that I open a new issue for my various questions.

[xiaq]

The summarize, the use cases satisfied by source in traditional shells are satisfied by different mechanisms in Elvish:

• Execute code in a file for side effect: eval (slurp < file)
• Add names into the local namespace: the general case is intentionally not supported; adding names to the REPL namespace can be done with edit:add-var
• Reusing code: use use to import a module