'nydp

'nydp is a new LISP dialect, much inspired by Arc, and hence indirectly by all of 'arc's ancestors, and implemented in Ruby.

'nydp is "Not Your Daddy's Parentheses", a reference to Xkcd 297 (itself a reference to Star Wars), as well as to the meme Not Your Daddy's Q, where Q is a modern, improved Q quite unlike the Q your daddy used. 'nydp also shamelessly piggypacks on the catchiness and popularity of the NYPD abbreviation ("New York Police Department").

We do not wish to suggest by "Not Your Daddy's Parentheses" that Common Lisp, Scheme, Racket, Arc, Clojure or your favourite other lisp are somehow old-fashioned, inferior, or in need of improvement in any way.

Further, we deplore and lament the gender bias and the age bias implicit in the use of the word "Daddy". We hereby declare that 'nydp is also not your mommy's parentheses, and, in fact, is not the parentheses of any other family member, including gender-neutral and nonbinary members, deceased ancestors and as-yet unconceived descendants. 'nydp is equivalently and unequivocally not their parentheses either, and neither more or less so than the parentheses of any male or masculine or father-role family member. It is possible that in due course this project may be renamed "Not Your Parents' Parentheses" even if 'nypp doesn't have quite the same ring to it and the 6-tuple "Parent" appears twice in the name.

Yet more deplorable and lamentable is the fact that "Daddy" appears to suggest that the age-gender axis is the only relevant axis under consideration. We hereby assert and declare that 'nydp is also not any parentheses you may obtain (in any manner legal or otherwise) from any race, religion, nationality, sexual orientation, or choice of text editor.

The goal of 'nydp is to allow untrusted users run sandboxed server-side scripts. By default, 'nydp provides no system access :

• no file functions
• no network functions
• no IO other than $stdin and $stdout
• no process functions
• no threading functions
• no ruby calls

Peruse 'nydp's features here in the tests directory.

Pronunciation guide: there is no fixed canonical pronunciation of 'nydp. Just keep in mind that if you find yourself wading knee-deep through raw sewage, it's still better than working on a java or C# project in a bank. Or insurance company, if that's your thing.

Running

Get a REPL :

$ bundle exec bin/nydp
welcome to nydp
^D to exit
nydp >

The REPL uses the readline library so you can use up- and down-arrows to navigate history. IRB also uses readline and command history may get a little bit mixed-up if you start nydp inside an irb session.

Invoking from Ruby

Suppose you want to invoke the function named question with some arguments. Do this:

ns     = Nydp.build_nydp     # keep this for later re-use, it's expensive to set up

answer = ns.apply :question, :life, ["The Universe", and_also(everything)]

==> 42

ns is a Nydp::Namespace object, mapping ruby symbols to nydp symbols for quick lookup at nydp compile-time. The nydp symbols maintain the values of global variables, including all builtin functions and any other functions defined using def.

Nydp is designed to operate in a multi-tenant architectural context, so you can instantiate multiple Namespace instances, where each tenant applies their own custom scripts ; such scripts will not interfere with one another (unless you've specifically arranged it to be so by duplicating namespaces or some such sorcery).

Facing the Truth

In conditional statements, nil is false, anything else is true

(if)           ;; same as (if nil)
(if a)         ;; same as a
(if a b)       ;; same as (if a b nil)
(if a b c)     ;; if a is nil, return c, otherwise return b
(if a b c d)   ;; same as (if a b (if c d))
(if a b c d e) ;; same as (if a b (if c d e))

;; and so on

Different from Arc :

1. Macro-expansion runs in lisp

After parsing its input, 'nydp passes the result as an argument to the pre-compile function. This is where things get a little bit circular: initially, pre-compile is a builtin function that just returns its argument. pre-compile bootstraps itself into existence in core-010-precompile.nydp.

You can override pre-compile to transform the expression in any way you wish. By default, the core-010-precompile.nydp implementation of pre-compile performs macro-expansion by searching for keywords in the macs global variable and applying the corresponding function to the given expression if a function is found.

(def pre-compile (expr)
  (map pre-compile
    (if (mac-names (car expr))
        (pre-compile (mac-expand (car expr) (cdr expr)))
        expr)))

(mac yoyo (thing) `(do-yoyo ,thing))

nydp > (pre-compile '(yoyo 42))

==> (do-yoyo 42)

2. Special symbol syntax

The parser detects syntax embedded in smybol names and emits a form whose first element names the syntax used. Here's an example:

nydp > (parse "x.y")

==> (dot-syntax x y)

nydp > (parse "$x x$ x$x $x$ $$")

==> (dollar-syntax || x)     ; '|| is the empty symbol.
==> (dollar-syntax x ||)
==> (dollar-syntax x x)
==> (dollar-syntax || x ||)
==> (dollar-syntax || || ||)

nydp > (parse "!foo")

==> (bang-syntax || foo)

nydp > (parse "!x.$y")

==> (bang-syntax || (dot-syntax x (dollar-syntax || y)))

Nydp provides macros for some but not all possible special syntax

nydp > (pre-compile 'x.y)

==> (hash-get x 'y) ; 'dot-syntax is a macro that expands to perform hash lookups

nydp > (pre-compile 'x.y.z)

==> (hash-get (hash-get x 'y) 'z)


nydp > (pre-compile '!eq?)

==> (fn args (no (apply eq? args)))

nydp > (pre-compile '(!eq? a b))

==> ((fn args (no (apply eq? args))) a b) ; equivalent to (no (eq? a b))

Ampersand-syntax - for example &foo, expands to a function which performs a hash-lookup on its argument.

nydp > (parse "&foo")
((ampersand-syntax || foo))

nydp > (pre-compile (parse "&foo"))
((fn (obj) (hash-get obj (quote foo))))

nydp > (assign hsh { foo 1 bar 2 })
nydp > (&bar hsh)
2

nydp > (&lastname (car german-composers))
Bach

nydp > (map &lastname german-composers)   ; ampersand-syntax creates a function you can pass around
(Bach Beethoven Wagner Mozart)

As with all other syntax, you can of course override the ampersand-syntax macro to handle your special needs.

You can combine special syntaxes ("%td" comes from nydp-html gem)

nydp > (map %td:&lastname german-composers)

"<td>Bach</td><td>Beethoven</td><td>Wagner</td><td>Mozart</td>"

So, %td expands to (percent-syntax || td), &lastname to (ampersand-syntax || lastname), and the whole %td:&lastname to (colon-syntax (percent-syntax || td) (ampersand-syntax || lastname)). Luckily for you, there's a fine colon-syntax macro that knows how to build a function out of these bits and pieces.

Look for SYMBOL_OPERATORS in parser.rb to see which syntax is recognised and in which order. The order of these definitions defines special-syntax-operator precedence.

Any character that is not special syntax will be recognised as part of a symbol. At time of writing, this includes the plus sign, hyphen, and slash.

;; nonsense code illustrating the use of certain
;; characters as function and variable names
(def //-+ (x y z)
  (let -*- (x y)
    (if z (//-+ x y -*-) -*-)))

3. Special list syntax

The parser detects alternative list delimiters

nydp > (parse "{ a 1 b 2 }")

==> (brace-list a 1 b 2)

brace-list is a macro that expands to create a hash literal. It assumes every item 0 is a literal key (symbol, string or number), item 1 is the corresponding value which is evaluated at run time, and so on for each following item-pair.

nydp > { a 1 b (author-name) }

==> {a=>1, b=>"conanite"}

4. Sensible, nestable string interpolation

The parser detects lisp code inside strings. When this happens, instead of emitting a string literal, the parser emits a form whose car is the symbol string-pieces.

nydp > (parse "\"foo\"")

==> "foo"

nydp > (let bar "Mister Nice Guy" "hello, ~bar")

==> "hello, Mister Nice Guy"

; this is a more tricky example because we need to make a string with an interpolation token in it

nydp > (let s "\"hello, \~world\"" (parse s))

==> (string-pieces "hello, " world) ; "hello, ", followed by the interpolation `world`

; It is possible to nest interpolations. Note that as with many popular language features, just because you can do something, does not mean you should:

nydp > (def also (str) "\nAND ALSO, ~str")
nydp > (with (a 1 b 2)
         (p "Consider ~a : the first thing,
              ~(also "Consider ~b : the second thing,
              ~(also "Consider ~(+ a b), the third (and final) thing")")"))

==> Consider 1 : the first thing,
==> AND ALSO, Consider 2 : the second thing,
==> AND ALSO, Consider 3, the third (and final) thing

By default, string-pieces is a function that just concatenates the string value of its arguments. You can redefine it as a macro to perform more fun stuff, or you can detect it within another macro to do extra-special stuff with it. The 'nydp-html gem detects 'string-pieces and gives it special powers in order to render haml and textile efficiently, and also to capture and report errors inside interpolations and report them correctly.

5. No continuations.

Sorry. While technically possible ... why bother?

6. No tail-call optimisation / tail-call elimination

This isn't completely true. For performance reasons, the nydp stack maps directly to the underlying ruby stack, and it is possible to enable TCO in your ruby VM. Nydp doesn't enable it for you though, you'll need to take care of this yourself. So basically nydp doesn't guarantee you TCO, but if you're in control of your deployment environment, you can still have it.

As a result, nydp has an extra keyword, 'loop, so that recursive solutions can be rewritten as iterative ones. For example

(def elegant-recursive-solution (n f things)
  (if (> n 0)
      (elegant-recursive-solution (- n 1) (cdr things))
      (f things)))

;; can be rewritten as
(def stupid-iterative-solution (n f things)
  (loop (> n 0)
    (= things (cdr things)
       n (- n 1)))
  (f things))

Besides that, what can Nydp do?

1. Functions and variables exist in the same namespace.

2. Macros are maintained in a hash called 'macs in the main namespace.

3. Loop construct (see above under 'no tail-call optimisation')

4. 'if like Arc:

(if a b c d e) ; equivalent to ruby :

if    a
      b
elsif c
      d
else  e

5. Lexically scoped, but dynamic variables possible, using thread-locals

nydp> (dynamic foo) ;; declares 'foo as a dynamic variable

nydp> (def do-something () (+ (foo) 1))

nydp> (w/foo 99 (do-something)) ;; sets 'foo to 99 for the duration of the call to 'do-something, will set it back to its previous value afterwards.

==> 100

nydp> (foo)

==> nil

6. Argument destructuring

This is not built-in to the language, but is available via the 'fun macro. Use 'fun as a drop-in replacement for 'fn and you get destructuring.

(def funny (a (b c) . others)
     xxx)

Equivalent to, and effectively pre-compiled to:

(def funny (a d . others)
     (with (b (nth 0 d)
            c (nth 1 d))
       xxx))

Note that d in this example will in real-life be a gensym and will not clobber your existing namespace.

In this example, others is either nil, or a list containing the third and subsequent arguments to the call to funny. For many examples of this kind of invocation, see invocation-tests in the tests directory. See also destructuring-examples

Nested destructuring lists work as expected.

def, 'with', and let all expand to forms using fun.

7. Basic error handling

nydp> (on-err (p "error")
        (ensure (p "make sure this happens")
          (/ 1 0)))

make sure this happens
error

8. Intercept comments

nydp > (parse "; blah blah")

==> (comment "blah blah")

Except in 'mac and 'def forms, by default, comment is a macro that expands to nil. If you have a better idea, go for it. Any comments present at the beginning of the body argument to mac or def are considered documentation. (See "self-documenting" below).

9. Prefix lists

The parser emits a special form if it detects a prefix-list, that is, a list with non-delimiter characters immediately preceding the opening delimiter. For example:

nydp > (parse "%w(a b c)")

==> (prefix-list "%w" (a b c))

This allows for preprocessing lists in a way not possible for macros. nydp uses this feature to implement shortcut monoexpression functions with single-letter arguments, as in

nydp > (parse "λx(len x)")

==> ((prefix-list "λx" (len x)))

nydp > (pre-compile '(prefix-list "λx" (len x)))

==> (fn (x) (len x))

Each character after 'λ becomes a function argument:

nydp > (parse "λxy(* x y)")

==> ((prefix-list "λxy" (* x y)))

nydp > (pre-compile '(prefix-list "λxy" (* x y)))

==> (fn (x y) (* x y))

Example:

nydp > (map λx(* x x) '(1 2 3 4))

==> (1 4 9 16)

Use 'define-prefix-list-macro to define a new handler for a prefix-list. Here's the code for the 'λ shortcut:

```lisp
(define-prefix-list-macro "^λ.+" vars expr
  (let var-list (map sym (cdr:string-split vars))
    `(fn ,var-list ,expr)))

In this case, the regex matches an initial 'λ ; there is no constraint however on the kind of regex a prefix-list-macro might use.

10. Self-documenting

Once the 'dox system is bootstrapped, any further use of 'mac or 'def will create documentation.

Comments immediately prior to the start of the form body will be used to generate help text. For example:

nydp > ; return the foo of x and y
       (def foo (x y)
         (* x y))

nydp > (dox foo)

Function : foo
args : (x y)
return the foo of x and y

source
======
(def foo (x y)
    (* x y))

'dox is a macro that generates code to output documentation to stdout. 'dox-lookup is a function that returns structured documentation.

nydp > (dox-lookup 'foo)
((foo def ("return the foo of x and y") (x y) (def foo (x y) (* x y))))

Not as friendly, but more amenable to programmatic manipulation. Each subsequent definition of 'foo (if you override it, define it as a macro, or define it again in some other context) will generate a new documentation structure, which will simply be preprended to the existing list.

11. Pretty-Printing

'dox above uses the pretty printer to display code source. The pretty-printer is hard-coded to handle some special cases, so it will unparse special syntax, prefix-lists, quote, quasiquote, unquote, and unquote-splicing.

You can examine its behaviour at the repl:

nydp > (p:pp '(string-pieces "hello " (bang-syntax || (dot-syntax x y (ampersand-syntax foo bar))) " and welcome to " (prefix-list "%%" (a b c d)) " and friends!"))

==> "hello ~!x.y.foo&bar and welcome to ~%%(a b c d) and friends!"

nydp > (p:pp:dox-src 'pp/find-breaks)

(def pp/find-breaks (form)
    (if (eq? 'if (car form))
      (let if-args (cdr form)
        (cons (list 'if (car if-args)) (map list (cdr if-args))))
      (or
        (pp/find-breaks/mac form)
        (list form))))

The pretty-printer is still rather primitive in that it only indents according to some hard-coded rules, and according to argument-count for documented macros. It has no means of wrapping forms that get too long, or that extend beyond a certain predefined margin or column number.

12. DSLs

The pre-compile system (described earlier in "Macro-expansion runs in lisp") is available for implementing local, mini-languages. To do this, use pre-compile-with in a macro. pre-compile-with expects a hash with expansion rules, and an expression to expand using these rules. For example, to build a "describe" dsl :

  (= describe-dsl (hash))
  (= describe-dsl.def (fn (a b c) `(a-special-kind-of-def ,a ,b ,c)))
  (= describe-dsl.p   (fn (w)     `(alternative-p ,w)))

  (mac describe (name expr outcome)
    `(add-description ',name
       (pre-compile-with describe-dsl ',expr)
       ',outcome))

In this example, describe-dsl establishes rules for expanding def and for p which will shadow their usual meanings, but only in the context of a describe form.

This technique is useful to avoid cluttering the global macro-namespace with rules that are used only in a specific context. A word of warning: if you shadow a popular name, for example let, map or do, you are likely to run into trouble when some non-shadowed macro inside your dsl context expands into something you have shadowed; in this case your shadowy definitions will be used to further expand the result.

For example, if you shadow with, but not let, and then you use a let form within a sample of the dsl you are specifying ; the let form expands into a with form, thinking the with will expand in the usual way, but in fact no, your private dsl is going to expand this with in your private special way, possibly in a way that's incompatible with the form produced by let.

If, notwithstanding the aforementioned, you do this anyway and the outcome works the way you expected it to, then you have the wrong kind of genius and your license to program should be revoked.

Extending NYDP

You can add any object implementing #call (for example, a ruby Proc) to your ns instance:

ns.assign(:"send-mail", -> {|props| Net::SMTP.deliver_the_letter(props).the_sooner_the_better }) # imaginary example

In your lisp,

(send-mail { from "myself@example.com" to "yourself@example.com" subject "SPECIAL OFFER" body (mail-body blah-blah) })

Installation

Add this line to your application's Gemfile:

gem 'nydp'

And then execute:

$ bundle

Or install it yourself as:

$ gem install nydp

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request