Skip to content
nydp - a modern, more dangerous kind of lisp
Ruby
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bin
lib
spec
.gitattributes
.gitignore
.rspec
.zeiger.yml
Gemfile
LICENSE.txt
README.md
Rakefile
nydp.gemspec

README.md

'nydp

'nydp is a new LISP dialect, much inspired by Arc, and hence indirectly by all of 'arc's ancestors, and implemented in Ruby.

'nydp is "Not Your Daddy's Parentheses", a reference to Xkcd 297 (itself a reference to Star Wars), as well as to the meme Not Your Daddy's Q, where Q is a modern, improved Q quite unlike the Q your daddy used. 'nydp also shamelessly piggypacks on the catchiness and popularity of the NYPD abbreviation ("New York Police Department", for those who have no interest in popular US politics or TV).

We do not wish to suggest by "Not Your Daddy's Parentheses" that Common Lisp, Scheme, Racket, Arc, Clojure or your favourite other lisp are somehow old-fashioned, inferior, or in need of improvement in any way.

The goal of 'nydp is to allow untrusted users run sandboxed server-side scripts. By default, 'nydp provides no system access :

  • no file functions
  • no network functions
  • no IO other than $stdin and $stdout
  • no process functions
  • no threading functions
  • no ruby calls

Peruse 'nydp's features here in the tests directory.

Pronunciation guide: there is no fixed canonical pronunciation of 'nydp. Just keep in mind that if you find yourself wading knee-deep through raw sewage, it's still better than working on a java project in a bank.

Running

Get a REPL :

$ bundle exec bin/nydp
welcome to nydp
^D to exit
nydp >

The REPL uses the readline library so you can use up- and down-arrows to navigate history.

Invoking from Ruby

Suppose you want to invoke the function named question with some arguments. Do this:

ns     = Nydp.build_nydp     # keep this for later re-use, it's expensive to set up

answer = Nydp.apply_function ns, :question, :life, ["The Universe", and_also(everything)]

==> 42

ns is just a plain old ruby hash, mapping ruby symbols to nydp symbols for quick lookup at nydp compile-time. The nydp symbols maintain the values of global variables, including all builtin functions and any other functions defined using def.

You can maintain multiple ns instances without mutual interference. In other words, assigning global variables while one ns is in scope will not affect the values of variables in any other ns (unless you've specifically arranged it to be so by duplicating namespaces or some such sorcery).

Different from Arc :

1. Macro-expansion runs in lisp

After parsing its input, 'nydp passes the result as an argument to the pre-compile function. This is where things get a little bit circular: initially, pre-compile is a builtin function that just returns its argument. pre-compile bootstraps itself into existence in core-010-precompile.nydp.

You can override pre-compile to transform the expression in any way you wish. By default, the core-010-precompile.nydp implementation of pre-compile performs macro-expansion.

(def pre-compile (expr)
  (map pre-compile
    (if (mac-names (car expr))
        (pre-compile (mac-expand (car expr) (cdr expr)))
        expr)))

(mac yoyo (thing) `(do-yoyo ,thing))

nydp > (pre-compile '(yoyo 42))

==> (do-yoyo 42)

2. Special symbol syntax

The parser detects syntax embedded in smybol names and emits a form whose first element names the syntax used. Here's an example:

nydp > (parse "x.y")

==> (dot-syntax x y)

nydp > (parse "$x x$ x$x $x$ $$")

==> (dollar-syntax || x)     ; '|| is the empty symbol.
==> (dollar-syntax x ||)
==> (dollar-syntax x x)
==> (dollar-syntax || x ||)
==> (dollar-syntax || || ||)

nydp > (parse "!foo")

==> (bang-syntax || foo)

nydp > (parse "!x.$y")

==> (bang-syntax || (dot-syntax x (dollar-syntax || y)))

Nydp provides macros for some but not all possible special syntax

nydp > (pre-compile 'x.y)

==> (hash-get x 'y) ; 'dot-syntax is a macro that expands to perform hash lookups

nydp > (pre-compile 'x.y.z)

==> (hash-get (hash-get x 'y) 'z)


nydp > (pre-compile '!eq?)

==> (fn args (no (apply eq? args)))

nydp > (pre-compile '(!eq? a b))

==> ((fn args (no (apply eq? args))) a b) ; equivalent to (no (eq? a b))

Ampersand-syntax - for example &foo, expands to a function which performs a hash-lookup on its argument.

nydp > (parse "&foo")
((ampersand-syntax || foo))

nydp > (pre-compile (parse "&foo"))
((fn (obj) (hash-get obj (quote foo))))

nydp > (assign hsh { foo 1 bar 2 })
nydp > (&bar hsh)
2

nydp > (&lastname (car german-composers))
Bach

nydp > (map &lastname german-composers)   ; ampersand-syntax creates a function you can pass around
(Bach Beethoven Wagner Mozart)

As with all other syntax, you can of course override the ampersand-syntax macro to handle your special needs.

Look for SYMBOL_OPERATORS in parser.rb to see which syntax is recognised and in which order. The order of these definitions defines special-syntax-operator precedence.

3. Special list syntax

The parser detects alternative list delimiters

nydp > (parse "{ a 1 b 2 }")

==> (brace-list a 1 b 2)

brace-list is a macro that expands to create a hash literal. It assumes every item 0 is a literal symbol key, item 1 is the corresponding value which is evaluated at run time, and so on for each following item-pair.

nydp > { a 1 b (author-name) }

==> {a=>1, b=>"conanite"}

4. Sensible, nestable string interpolation

The parser detects lisp code inside strings. When this happens, instead of emitting a string literal, the parser emits a form whose car is the symbol string-pieces.

nydp > (parse "\"foo\"")

==> "foo"

nydp > (let bar "Mister Nice Guy" "hello, ~bar")

==> "hello, Mister Nice Guy"

; this is a more tricky example because we need to make a string with an interpolation token in it

nydp > (let s (joinstr "" "\"hello, " '~ "world\"") (parse s))

==> (string-pieces "hello, " world "") ; "hello, ", followed by the interpolation 'world, followed by the empty string after 'world

; It is possible to nest interpolations. Note that as with many popular language features, just because you can do something, does not mean you should:

nydp > (def also (str) "\nAND ALSO, ~str")
nydp > (with (a 1 b 2)
         (p "Consider ~a : the first thing,
              ~(also "Consider ~b : the second thing,
              ~(also "Consider ~(+ a b), the third (and final) thing")")"))

==> Consider 1 : the first thing,
==> AND ALSO, Consider 2 : the second thing,
==> AND ALSO, Consider 3, the third (and final) thing

By default, string-pieces is a function that just concatenates the string value of its arguments. You can redefine it as a macro to perform more fun stuff, or you can detect it within another macro to do extra-special stuff with it. The 'nydp-html gem detects 'string-pieces and gives it special treatment in order to render haml and textile efficiently, and also to capture and report errors inside interpolations and report them correctly.

5. No continuations.

Sorry. While technically possible ... why bother?

Besides that, what can Nydp do?

1. Functions and variables exist in the same namespace.

2. Macros are maintained in a hash called 'macs in the main namespace.

3. General tail call elimination allowing recursion without stack overflow in some cases.

4. 'if like Arc:

(if a b c d e) ; equivalent to ruby :
if    a
      b
elsif c
      d
else  e

5. Lexically scoped, but dynamic variables possible, using thread-locals

nydp> (dynamic foo) ;; declares 'foo as a dynamic variable

nydp> (def do-something () (+ (foo) 1))

nydp> (w/foo 99 (do-something)) ;; sets 'foo to 99 for the duration of the call to 'do-something, will set it back to its previous value afterwards.

==> 100

nydp> (foo)

==> nil

6. Argument destructuring

This is not built-in to the language, but is available via the 'fun macro. Use 'fun as a drop-in replacement for 'fn and you get destructuring.

(def funny (a (b c) . others)
     xxx)

Equivalent to, and effectively pre-compiled to:

(def funny (a d . others)
     (with (b (nth 0 d)
            c (nth 1 d))
       xxx))

Note that d in this example will in real-life be a gensym and will not clobber your existing namespace.

In this example, others is either nil, or a list containing the third and subsequent arguments to the call to funny. For many examples of this kind of invocation, see invocation-tests in the tests directory. See also destructuring-examples

Nested destructuring lists work as expected.

def, 'with', and let all expand to forms using fun.

7. Basic error handling

nydp> (on-err (p "error")
        (ensure (p "make sure this happens")
          (/ 1 0)))

make sure this happens
error

8. Intercept comments

nydp > (parse "; blah blah")

==> (comment "blah blah")

Except in 'mac and 'def forms, by default, comment is a macro that expands to nil. If you have a better idea, go for it. Any comments present at the beginning of the body argument to mac or def are considered documentation. (See "self-documenting" below).

9. Prefix lists

The parser emits a special form if it detects a prefix-list, that is, a list with non-delimiter characters immediately preceding the opening delimiter. For example:

nydp > (parse "%w(a b c)")

==> (prefix-list "%w" (a b c))

This allows for preprocessing lists in a way not possible for macros. nydp uses this feature to implement shortcut oneline functions, as in

nydp > (parse "λx(len x)")

==> ((prefix-list "λx" (len x)))

nydp > (pre-compile '(prefix-list "λx" (len x)))

==> (fn (x) (len x))

Each character after 'λ becomes a function argument:

nydp > (parse "λxy(* x y)")

==> ((prefix-list "λxy" (* x y)))

nydp > (pre-compile '(prefix-list "λxy" (* x y)))

==> (fn (x y) (* x y))

Use 'define-prefix-list-macro to define a new handler for a prefix-list. Here's the code for the 'λ shortcut:

(define-prefix-list-macro "^λ.+" vars expr
  (let var-list (map sym (cdr:string-split vars))
    `(fn ,var-list ,expr)))

In this case, the regex matches an initial 'λ ; there is no constraint however on the kind of regex a prefix-list-macro might use.

10. Self-documenting

Once the 'dox system is bootstrapped, any further use of 'mac or 'def will create documentation.

Any comments at the start of the form body will be used to generate help text. For example:

nydp > (def foo (x y)
         ; return the foo of x and y
         (* x y))

nydp > (dox foo)

Function : foo
args : (x y)
return the foo of x and y

source
======
(def foo (x y)
    (* x y))

'dox is a macro that generates code to output documentation to stdout. 'dox-lookup is a function that returns structured documentation.

nydp > (dox-lookup 'foo)
((foo def ("return the foo of x and y") (x y) (def foo (x y) (* x y))))

Not as friendly, but more amenable to programmatic manipulation. Each subsequent definition of 'foo (if you override it, define it as a macro, or define it again in some other context) will generate a new documentation structure, which will simply be preprended to the existing list.

11. Pretty-Printing

'dox above uses the pretty printer to display code source. The pretty-printer is hard-coded to handle some special cases, so it will unparse special syntax, prefix-lists, quote, quasiquote, unquote, and unquote-splicing.

You can examine its behaviour at the repl:

nydp > (p:pp '(string-pieces "hello " (bang-syntax || (dot-syntax x y (ampersand-syntax foo bar))) " and welcome to " (prefix-list "%%" (a b c d)) " and friends!"))

==> "hello ~!x.y.foo&bar and welcome to ~%%(a b c d) and friends!"

nydp > (p:pp:dox-src 'pp/find-breaks)

(def pp/find-breaks (form)
    (if (eq? 'if (car form))
      (let if-args (cdr form)
        (cons (list 'if (car if-args)) (map list (cdr if-args))))
      (or
        (pp/find-breaks/mac form)
        (list form))))

The pretty-printer is still rather primitive in that it only indents according to some hard-coded rules, and according to argument-count for documented macros. It has no means of wrapping forms that get too long, or that extend beyond a certain predefined margin or column number.

12. DSLs

The pre-compile system (described earlier in "Macro-expansion runs in lisp") is available for implementing local, mini-languages. To do this, use pre-compile-with in a macro. pre-compile-with expects a hash with expansion rules, and an expression to expand using these rules. For example, to build a "describe" dsl :

  (= describe-dsl (hash))
  (= describe-dsl.def (fn (a b c) `(a-special-kind-of-def ,a ,b ,c)))
  (= describe-dsl.p   (fn (w)     `(alternative-p ,w)))

  (mac describe (name expr outcome)
    `(add-description ',name
       (pre-compile-with describe-dsl ',expr)
       ',outcome))

In this example, describe-dsl establishes rules for expanding def and for p which will shadow their usual meanings, but only in the context of a describe form.

This technique is useful to avoid cluttering the global macro-namespace with rules that are used only in a specific context. A word of warning: if you shadow a popular name, for example let, map or do, you are likely to run into trouble when some non-shadowed macro inside your dsl context expands into something you have shadowed; in this case your shadowy definitions will be used to further expand the result.

For example, if you shadow with, but not let, and then you use a let form within a sample of the dsl you are specifying ; the let form expands into a with form, thinking the with will expand in the usual way, but in fact no, your private dsl is going to expand this with in your private special way, possibly in a way that's incompatible with the form produced by let.

If, notwithstanding the aforementioned, the outcome works the way you expected it to, then you have the wrong kind of genius and your license to program should be revoked.

Installation

Add this line to your application's Gemfile:

gem 'nydp'

And then execute:

$ bundle

Or install it yourself as:

$ gem install nydp

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request
You can’t perform that action at this time.