ShellJS

[piranna]

http://documentup.com/arturadib/shelljs
https://github.com/arturadib/shelljs

Don't know if it should/could be a dependency of nsh, but definitelly it should be on the tool-box... :-D

[groundwater]

I tried shelljs once. At the time, it shelled out to bash a lot. So if you don't have bash on the system, it doesn't work 😦

[piranna]

I've reviewed all shelljs code and probably due to being multiplatform (Mac, Windows & Linux) only dependences are fs,os &path modules, that I doubt are using bash in the background... ;-)

As a minor critical, commands are not exposed as bins on package,json so they could be used by regular shell scripts, but it wouldn't be difficult.

[groundwater]

This line here

cmd += ' > '+stdoutFile+' 2>&1'; // works on both win/unix

The use of > will cause a shell process to spawn to interpret the redirect. I think this is a feature built into whatever libc exec call that libuv is using.

[piranna]

I was suspicious of the code block under that, maybe it's a bug since they already have the to() function that works as the redirection. If so, the patch is easy to do.

Anyway, good catch ;-)

[piranna]

Seems the problem is related with the redirection of stderr, to() doesn't allow to define any parameter, something logical because it replace the redirection, but in the other hand it would allow them since it's no a standard command. It seems shelljs commands only works with stdout, would need to take a look in deeper detail, maybe stderr is stored somewhere so this can be fixed...

[groundwater]

I've tried a few pre-made shells for node, and so far they all fail this test:

1. start the shell
2. start the shell again (shell within a shell)
3. start vim, write, and save some text
4. exit the second shell with ^D
5. tail -f the file you made (see its content)
6. hit ^C to exit the tail command
7. exit the first shell with ^D

You should now be logged out.

The hardest part is keeping signals going to the right place, and I haven't seen any that do this correctly. I think even nsh has some corner cases, but last I tried, it can pass the above test.

[piranna]

Challenge accepted, it will be funny :-D

[groundwater]

and remember, it needs to be installable via npm 😄

[piranna]

First version using REPL:

[piranna@Latitude:~/Dropbox/Proyectos/nsh]
 (master) > ./nsh.js 
> pwd
/home/piranna/Dropbox/Proyectos/nsh
undefined
> node ./nsh.js
> cd
undefined
> pwd
/home/piranna
undefined
> ^D
undefined
> pwd
/home/piranna/Dropbox/Proyectos/nsh
undefined
> 
(^C again to quit)
> 
^D

[piranna@Latitude:~/Dropbox/Proyectos/nsh]
 (master) > 

Profit! 😄

There are some things broken like the prompt since both REPL and the underlying readline doesn't support to define a function, only a string (something stupid, I find it more interesting to call a function to define the prompt each time it will be printed, but anyway...) and some hacks about how REPL works (it's CLEARLY intended to process only Javascript queries, what a shame...), but definitely it works and has potential :-) I'll do a pull-request so code can be reviewed and discussed while I work on this :-)

[piranna]

I have been reviewing both shelljs and bashful projects and don't finish to be satisficied with any of them. bashful is very mature (and @SubsTrack is almost a genious :-) ) but doesn't have almost commands and is very streams oriented, that I think is a generic but bloated design here, has very few commands and seems abandoned. I was thinking about making it compatible with shelljs, it has a lot of commands in a function-oriented design, but they are incomplete and primitive, and the port of bashful seems difficult.

So, I'm thinking to develop both my own bash parser and bash-like commands in Node.js using both projects as basis, each command working as a regular function (stdout by returning, stderr by throwing) and a wrapper function to catch each of them independently, making them autonomous executables, but also being regular function being able to require and use them from other programs as a library. This way we can isolate their behaviour and be sure they work correctly, and more than this, being actual functions we can return objects that can be used by others in a higher level, being able to build an object-oriented-shell :-) What do you think? :-)

[groundwater]

You should check out what @grncdr has been working on. I think he's put a lot of work into https://github.com/grncdr/js-shell-parse

I would love to hook that up to nsh

[piranna]

It's f*cking cool!!! :-D Seems to be a little bit abandoned, but if the tests are really passing, currently it's really powerful! :-D The AST is a little bit verbose to my taste, but it could be really useful to integrate bash commands with Javascript functions and statements if I develop the built-ins library I told you before, thanks! :-)

[grncdr]

Hi @piranna glad you like it 😄.

Seems to be a little bit abandoned, but if the tests are really passing, currently it's really powerful!

It's only a little abandoned 😉. I got the parser to the point I was satisfied with it and started working on an actual shell, but real life got in the way and I haven't been working on it very much recently.

The AST is a little bit verbose to my taste

I agree, but I haven't found it to be too big of a deal. You may also want to look at https://github.com/grncdr/js-shell-frontend for an interface that's more suitable for implementing an actual shell.

A big part of my motivation for writing a shell was to have something that I could integrate with npm itself and have proper portable "scripts" in package.json.

I have a halfway working interpreter for the AST produced by js-shell-parse so if you're interested I could put it up on Github somewhere later this evening. (I don't have access to that code at the moment).

[piranna]

Hi @grncdr :-) Yeah, reading the tests seems it's really cool and manage very good the bash commands, @SubsTrack said it was a nighmare when he was developing bashful... :-P

My idea is about the AST could help to identify the commands and their relationship, so later I could merge them with the regular Javascript functions and statements instead of build a plain shell, so I think I'll need to stick to the lower level library ;-) My comment about being too much verbose is of the empty arrays and objects and null atributes, but definitely they will make easier to work with them since you don't need to do checks if they exists.

If you want you can upload the interpreter code so we can check and comment on it :-)

[piranna]

I'm having too much problems to be able to identify and merge bash commands and Javascript statements so they can be merged, and I'm thinking: does it makes sense? My idea was to be able to detect external commands and be able to assing their result to variables and work with it, but at the same time being able to work with the usuals bash functions (redirections, pipes and so), so maybe I should focus only on this basic bash/prompt functionality, and left the more advanced ones to plain Javascript. What do you think? The point is that I'm not too much in set explicitly when I'm using one language or the other by using backticks or similar, but if it's the only solution to be able to merge them... :-/

[groundwater]

Whenever I hit a roadblock like this I think "what is the least I can do, such that I can publish my changes", and that's usually the right answer. It's much easier to iterate on working code that non-working code.

I guess what I'm saying is, if you can improve the bash support, start there. If you can later figure out how to support the JavaScript REPL, do that second. Of course you can try this the other way too, get JS support rolling and then come back and try for basic bash support 😄

[piranna]

Whenever I hit a roadblock like this I think "what is the least I can do, such that I can publish my changes", and that's usually the right answer. It's much easier to iterate on working code that non-working code.

Occam's razor applied to software development, lol :-P

I guess what I'm saying is, if you can improve the bash support, start there. If you can later figure out how to support the JavaScript REPL, do that second. Of course you can try this the other way too, get JS support rolling and then come back and try for basic bash support

REPL support is very advanced and working, so maybe I'll do a step
back regarding bash support and only allow by the moment to launch
commands, since it was working. I'll see later how to integrate them,
but I'm thinking I would need to use esprima,

"Si quieres viajar alrededor del mundo y ser invitado a hablar en un
monton de sitios diferentes, simplemente escribe un sistema operativo
Unix."
– Linus Tordvals, creador del sistema operativo Linux

[grncdr]

I'll just chime in here with my 2 cents:

If you want to combine javascript and shell scripts into one file, you will need a way of switching parsers inline. This is something I've daydreamed about for my js-shell toolchain, but it isn't implemented yet. My general idea was to do this:

# shell syntax is the default
echo Welcome to my mixed-language script

# heredoc shebangs switch interpreters
<<<!js
  // The built-in `js` interpreter would be a custom node environment with some convenient globals
  var enclosingBashScript = readFileSync(path.join(process.cwd(), $0));
  console.log('Script length:', enclosingBashScript.length) 
js

# or you can use the name of an executable instead
<<<!ruby-1.8
  puts ENV['PATH']
ruby-1.8

The point is that the <<<!{interpreter} syntax is equivalent to cat <<{interpreter} | {interpreter} but the shell would be designed in such a way that you could extend it with interpreters that provide a bit more convenience for embedding in a shell. For example the js interpreter would probably be something like this:

#!/usr/bin/env node
var concat = require('concat-stream');
global.env = process.env
for (var k in global.env) if (!global.hasOwnProperty(k)) global[k] = global.env[k];
process.stdin.pipe(concat(function (script) {
  eval(script.toString());
}))

[piranna]

My idea of integration was far beyond, be able to detect commands and call them as functions

> ls
blah
> var a = ls()
undefined
> a
blah
> ls('-a')
.
..
blah

and also be able to detect if and for statements and "compile" them to Javascript, but seems it's not an easy task, so probably the best and only solution I would get is to use esprima to parse the Javascript command line, detect backticks and use bash strings as a domain-specific language embebed in the Javascript source code lines:

> var a = `ls`
undefined
> a
blah

Detect heuristically if a line is Javascript or Bash or a merge of both? A compiler from Bash to Node.js? Maybe have I been too ambitious? :-/