Skip to content
Small Clojure interpreter and linter
Go Other
  1. Go 98.6%
  2. Other 1.4%
Branch: gostd
Clone or download
Pull request Compare This branch is 873 commits ahead, 2 commits behind candid82:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci Update circleci image. Jul 21, 2019
core Merge from upstream and regen docs Nov 13, 2019
docs Merge from upstream and regen docs Nov 13, 2019
snippets Add a FileMode snippet Oct 26, 2019
std Merge from upstream and regen docs Nov 13, 2019
tests Merge from upstream and regen docs Nov 13, 2019
tools
.gitattributes PR feedback Jan 10, 2019
.gitignore Move gostd2joker here as gostd; refactor Dec 18, 2018
DEVELOPER.md Merge from upstream, reorder go generate, clarify gostd docs Nov 11, 2019
GOSTD.md Replace (Go var := expr) with GoVar support in (var-set var expr) Oct 26, 2019
LICENSE Update readme, add LICENSE file Dec 11, 2016
README.md Merge from upstream, reorder go generate, clarify gostd docs Nov 11, 2019
all-tests.sh [scripts/all-tests] Exit all-tests with rc. Apr 18, 2019
build-all.sh Improve build-all.sh Mar 22, 2017
build-arm.sh
build.sh Fix build.sh for Go v1.12. Apr 29, 2019
clean.sh Have to delete docs/index.html after re-running Joker to clean up Oct 20, 2019
custom.go Revert some of what was accidentally included in previous commit Oct 20, 2019
epl-v10.html legal stuff May 3, 2016
eval-tests.sh Pass actual stdin to cmd for :stdin :pipe Jun 13, 2019
flag-tests.sh add working-dir flag Nov 24, 2017
go.mod Support for Go 1.13 Oct 22, 2019
go.sum Support for Go 1.13 Oct 22, 2019
linter-tests.sh Update circle.yml Apr 17, 2017
main.go Merge from upstream Nov 8, 2019
release.joke Generate docs in release.joke. Oct 20, 2018
repl.go
repl_plan9.go Plan9 support Oct 31, 2019
run.sh
shadow.sh Create shadow.sh script May 2, 2019
test-linter.sh Ignore project.clj and user.clj in test-linter.sh Nov 26, 2017

README.md

CircleCI

Joker

Joker is a small Clojure interpreter and linter written in Go.

This gostd experimental fork extends official Joker by reading Golang source code (the Go standard library) and "wrapping" some of their functions, types, constants, and variables so that code written in Joker can access them. See below for information on the go.std.* namespaces thereby provided.

Installation

On macOS, the easiest way to install Joker is via Homebrew:

brew install candid82/brew/joker

The same command can be used on Linux if you use Linuxbrew.

If you use Arch Linux, there is AUR package.

If you use Nix, then you can install Joker with

nix-env -i joker

On other platforms (or if you prefer manual installation), download a precompiled binary for your platform and put it on your PATH.

You can also build Joker from the source code.

Usage

joker - launch REPL

joker <filename> - execute a script. Joker uses .joke filename extension. For example: joker foo.joke. Normally exits after executing the script, unless --exit-to-repl is specified before --file <filename> in which case drops into the REPL after the script is (successfully) executed. (Note use of --file in this case, to ensure <filename> is not treated as a <socket> specification for the repl.)

joker --eval <expression> - execute an expression. For example: joker -e '(println "Hello, world!")'. Normally exits after executing the script, unless --exit-to-repl is specified before --eval, in which case drops into the REPL after the expression is (successfully) executed.

joker --lint <filename> - lint a source file. See Linter mode for more details.

joker --lint --working-dir <dirname> - recursively lint all Clojure files in a directory.

joker - - execute a script on standard input (os.Stdin).

Documentation

Standard library reference

Joker slack channel

Project goals

These are high level goals of the project that guide design and implementation decisions.

  • Be suitable for scripting (lightweight, fast startup). This is something that Clojure is not good at and my personal itch I am trying to scratch.
  • Be user friendly. Good error messages and stack traces are absolutely critical for programmer's happiness and productivity.
  • Provide some tooling for Clojure and its dialects. Joker has linter mode which can be used for linting Joker, Clojure and ClojureScript code. It catches some basic errors. For those who don't use Cursive, this is probably already better than the status quo. Joker can also be used for pretty printing EDN data structures (very basic algorithm at the moment). For example, the following command can be used to pretty print EDN data structure (read from stdin):
joker --hashmap-threshold -1 -e "(pprint (read))"

There is Sublime Text plugin that uses Joker for pretty printing EDN files. Here you can find the description of --hashmap-threshold parameter, if curious. Tooling is one of the primary Joker use cases for me, so I intend to improve and expand it.

  • Be as close (syntactically and semantically) to Clojure as possible. Joker should truly be a dialect of Clojure, not a language inspired by Clojure. That said, there is a lot of Clojure features that Joker doesn't and will never have. Being close to Clojure only applies to features that Joker does have.

Project non-goals

  • Performance. If you need it, use Clojure. Joker is a naive implementation of an interpreter that evaluates unoptimized AST directly. I may be interested in doing some basic optimizations but this is definitely not a priority.
  • Have all Clojure features. Some features are impossible to implement due to a different host language (Go vs Java), others I don't find that important for the use cases I have in mind for Joker. But generally Clojure is a pretty large language at this point and it is simply unfeasible to reach feature parity with it, even with naive implementation.

Differences with Clojure

  1. Primitive types are different due to a different host language and desire to simplify things. Scripting doesn't normally require all the integer and float types, for example. Here is a list of Joker's primitive types:
Joker type Corresponding Go type
BigFloat big.Float
BigInt big.Int
Boolean bool
Char rune
Double float64
Int int
Keyword n/a
Nil n/a
Ratio big.Rat
Regex regexp.Regexp
String string
Symbol n/a
Time time.Time

Note that Nil is a type that has one value nil.

  1. The set of persistent data structures is much smaller:
Joker type Corresponding Clojure type
ArrayMap PersistentArrayMap
MapSet PersistentHashSet (or hypothetical PersistentArraySet, depending on which kind of underlying map is used)
HashMap PersistentHashMap
List PersistentList
Vector PersistentVector
  1. Joker doesn't have the same level of interoperability with the host language (Go) as Clojure does with Java or ClojureScript does with JavaScript. It doesn't have access to arbitrary Go types and functions. There is only a small fixed set of built-in types and interfaces. Dot notation for calling methods is not supported (as there are no methods). All Java/JVM specific functionality of Clojure is not implemented for obvious reasons.
  2. Joker is single-threaded with no support for concurrency or parallelism. Therefore no refs, agents, futures, promises, locks, volatiles, transactions, p* functions that use multiple threads. Vars always have just one "root" binding.
  3. The following features are not implemented: protocols, records, structmaps, chunked seqs, transients, tagged literals, unchecked arithmetics, primitive arrays, custom data readers, transducers, validators and watch functions for vars and atoms, hierarchies, sorted maps and sets.
  4. Unrelated to the features listed above, the following function from clojure.core namespace are not currently implemented but will probably be implemented in some form in the future: subseq, iterator-seq, reduced?, reduced, mix-collection-hash, definline, re-groups, hash-ordered-coll, enumeration-seq, compare-and-set!, rationalize, load-reader, find-keyword, comparator, resultset-seq, file-seq, sorted?, ensure-reduced, rsubseq, pr-on, seque, alter-var-root, hash-unordered-coll, re-matcher, unreduced.
  5. Built-in namespaces have joker prefix. The core namespace is called joker.core. Other built-in namespaces include joker.string, joker.json, joker.os, joker.base64 etc. See standard library reference for details.
  6. Miscellaneous:
  • case is just a syntactic sugar on top of condp and doesn't require options to be constants. It scans all the options sequentially.
  • slurp only takes one argument - a filename (string). No options are supported.
  • ifn? is called callable?
  • Map entry is represented as a two-element vector.
  • resolving unbound var returns nil, not the value Unbound. You can still check if the var is bound with bound? function.

Linter mode

To run Joker in linter mode pass --lint --dialect <dialect> flag, where <dialect> can be clj, cljs, joker or edn. If --dialect <dialect> is omitted, it will be set based on file extension. For example, joker --lint foo.clj will run linter for the file foo.clj using Clojure (as opposed to ClojureScript or Joker) dialect. joker --lint --dialect cljs - will run linter for standard input using ClojureScript dialect. Linter will read and parse all forms in the provided file (or read them from standard input) and output errors and warnings (if any) to standard output (for edn dialect it will only run read phase and won't parse anything). Let's say you have file test.clj with the following content:

(let [a 1])

Executing the following command joker --lint test.clj will produce the following output:

test.clj:1:1: Parse warning: let form with empty body

The output format is as follows: <filename>:<line>:<column>: <issue type>: <message>, where <issue type> can be Read error, Parse error, Parse warning or Exception.

Integration with editors

Here are some examples of errors and warnings that the linter can output.

Reducing false positives

Joker lints the code in one file at a time and doesn't try to resolve symbols from external namespaces. Because of that and since it's missing some Clojure(Script) features it doesn't always provide accurate linting. In general it tries to be unobtrusive and error on the side of false negatives rather than false positives. One common scenario that can lead to false positives is resolving symbols inside a macro. Consider the example below:

(ns foo (:require [bar :refer [def-something]]))

(def-something baz ...)

Symbol baz is introduced inside def-something macro. The code is totally valid. However, the linter will output the following error: Parse error: Unable to resolve symbol: baz. This is because by default the linter assumes external vars (bar/def-something in this case) to hold functions, not macros. The good news is that you can tell Joker that bar/def-something is a macro and thus suppress the error message. To do that you need to add bar/def-something to the list of known macros in Joker configuration file. The configuration file is called .joker and should be in the same directory as the target file, or in its parent directory, or in its parent's parent directory etc up to the root directory. When reading from stdin Joker will look for a .joker file in the current working directory. The --working-dir <path/to/file> flag can be used to override the working directory that Joker starts looking in. Joker will also look for a .joker file in your home directory if it cannot find it in the above directories. The file should contain a single map with :known-macros key:

{:known-macros [bar/def-something foo/another-macro ...]}

Please note that the symbols are namespace qualified and unquoted. Also, Joker knows about some commonly used macros (outside of clojure.core namespace) like clojure.test/deftest or clojure.core.async/go-loop, so you won't have to add those to your config file.

Joker also allows you to specify symbols that are introduced by a macro:

{:known-macros [[riemann.streams/where [service event]]]}

So each element in :known-macros vector can be either a symbol (as in the previous example) or a vector with two elements: macro's name and a list of symbols introduced by this macro. This allows to avoid symbol resolution warnings in macros that intern specific symbols implicitly.

Additionally, if you want Joker to ignore some unused namespaces (for example, if they are required for their side effects) you can add the :ignored-unused-namespaces key to your .joker file:

{:ignored-unused-namespaces [foo.bar.baz]}

Sometimes your code may refer to a namespace that is not explicitly required in the same file. This is rarely needed, but if you face such situation you can add that namespace to :known-namespaces list to avoid "No namespace found" or "Unable to resolve symbol" warnings:

{:known-namespaces [clojure.spec.gen.test]}

If your code uses tagged literals that Joker doesn't know about, add them to :known-tags list:

{:known-tags [db/fn]}

If you use :refer :all Joker won't be able to properly resolve symbols because it doesn't know what vars are declared in the required namespace (i.e. clojure.test). There are generally three options here:

  1. Refer specific symbols. For example: [clojure.test :refer [deftest testing is are]]. This is usually not too tedious, and you only need to do it once per file.
  2. Use alias and qualified symbols:
(:require [clojure.test :as t])
(t/deftest ...)
  1. "Teach" Joker declarations from referred namespace. Joker executes the following files (if they exist) before linting your file: .jokerd/linter.cljc (for both Clojure and ClojureScript), .jokerd/linter.clj (Clojure only), .jokerd/linter.cljs (ClojureScript only). The rules for locating .jokerd directory are the same as for locating .joker file.

    • ⚠️ Joker can be made aware of any additional declarations (like deftest and is) by providing them in .jokerd/linter.clj[s|c] files. However, this means Joker cannot check that the symbols really are declared in your namespace, so this feature should be used sparingly.
    • If you really want some symbols to be considered declared in any namespace no matter what, you can add (in-ns 'joker.core) to your linter.clj[s|c] and then declare those symbols. (see issues 52 and 50 for discussion).

I generally prefer first option for clojure.test namespace.

Linting directories

To recursively lint all files in a directory pass --working-dir <dirname> parameter. Please note that if you also pass file argument (or --file parameter) Joker will lint that single file and will only use --working-dir to locate .joker config file. That is,

joker --lint --working-dir my-project

lints all Clojure files in my-project directory, whereas

joker --lint --working-dir my-project foo.clj

lints single file foo.clj but uses .joker config file from my-project directory.

When linting directories Joker lints all files with the extension corresponding to the selected dialect (*.clj, *.cljs, *.joke, or *.edn). To exclude certain files specify regex patterns in :ignored-file-regexes vector in .joker file, e.g. :ignored-file-regexes [#".*user\.clj" #".*/dev/profiling\.clj"].

When linting directories Joker can report globally unused namespaces and public vars. This is turned off by default but can be enabled with --report-globally-unused flag, e.g. joker --lint --working-dir my-project --report-globally-unused. This is useful for finding "dead" code. Some namespaces or vars are intended to be used by external systems (e.g. public API of a library or main function of a program). To exclude such namespaces and vars from being reported as globally unused list them in :entry-points vector in .joker file, which may contain the names of namespaces or fully qualified names of vars. For example:

:entry-points [my-project.public-api
               my-project.core/-main]

Optional rules

Joker supports a few configurable linting rules. To turn them on or off set their values to true or false in :rules map in .joker file. For example:

:rules {:if-without-else true
        :no-forms-threading false}

Below is the list of all configurable rules.

Rule Description Default value
if-without-else warn on if without the else branch false
no-forms-threading warn on threading macros with no forms, i.e. (-> a) true
unused-as warn on unused :as binding true
unused-keys warn on unused :keys, :strs, and :syms bindings true
unused-fn-parameters warn on unused fn parameters false
fn-with-empty-body warn on fn form with empty body true

Note that unused binding and unused parameter warnings are suppressed for names starting with underscore.

Building

Joker requires Go v1.12 or later. Below commands should get you up and running.

go get -d github.com/candid82/joker
cd $GOPATH/src/github.com/candid82/joker
./run.sh --version && go install

Cross-platform Builds

After building the native version (to autogenerate appropriate files, "vet" the source code, etc.), set the appropriate environment variables and invoke go build. E.g.:

$ GOOS=linux GOARCH=arm GOARM=6 go build

(The run.sh script does not support cross-platform building.)

Coding Guidelines

  • Dashes (-) in namespaces are not converted to underscores (_) by Joker, so (unlike with Clojure) there's no need to name .joke files accordingly.
  • Avoid :refer :all and the use function, as that reduces the effectiveness of linting.

The go.std.* Namespaces

On this experimental branch, Joker is built against a Golang tree in order to pull in and "wrap" functions, types, constants, and variables provided by Go std packages.

NOTE: Only Joker versions >= 0.12 are now supported by this branch.

Quick Start

To make this "magic" happen:

  1. Ensure you can build the "canonical" version of Joker
  2. go get -u -d github.com/candid82/joker (This will download and update dependent packages as well, but not build Joker itself.)
  3. cd $GOPATH/src/github.com/candid82/joker
  4. git remote add gostd git@github.com:jcburley/joker.git
  5. git fetch gostd
  6. git co gostd
  7. ./run.sh, specifying optional args such as --version, -e '(println "i am here")', or even:
-e "(require '[go.std.net :as n]) (print \"\\nNetwork interfaces:\\n  \") (n/Interfaces) (println)"
  1. ./joker invokes the just-built Joker executable.
  2. go install installs Joker (and deletes the local copy).

Sample Usage

Assuming Joker has been built as described above:

$ joker
Welcome to joker v0.12.6. Use EOF (Ctrl-D) or SIGINT (Ctrl-C) to exit.
user=> (require '[go.std.net :as n])
nil
user=> (sort (map #(key %) (ns-map 'go.std.net)))
(AddrError. CIDRMask DNSConfigError. DNSError. Dial DialIP DialTCP DialUDP DialUnix FlagBroadcast FlagLoopback FlagMulticast FlagPointToPoint FlagUp Flags. IPConn. IPv4 IPv4Mask IPv4len IPv6len InterfaceAddrs InterfaceByIndex InterfaceByName Interfaces InvalidAddrError. JoinHostPort Listen ListenIP ListenMulticastUDP ListenPacket ListenTCP ListenUDP ListenUnix ListenUnixgram LookupAddr LookupCNAME LookupHost LookupIP LookupMX LookupNS LookupPort LookupSRV LookupTXT MX. NS. ParseCIDR ParseError. ParseIP ParseMAC Pipe ResolveIPAddr ResolveTCPAddr ResolveUDPAddr ResolveUnixAddr SRV. SplitHostPort TCPConn. TCPListener. UDPConn. UnixAddr. UnixConn. UnixListener. UnknownNetworkError.)
user=> (n/Interfaces)
[[{:Index 1, :MTU 65536, :Name "lo", :HardwareAddr [], :Flags 5} {:Index 2, :MTU 1500, :Name "eth0", :HardwareAddr [20 218 233 31 200 87], :Flags 19} {:Index 3, :MTU 1500, :Name "docker0", :HardwareAddr [2 66 188 97 92 58], :Flags 19}] nil]
user=>
$

Further Reading

See GOSTD Usage for more information.

Developer Notes

See DEVELOPER.md for information on Joker internals, such as adding new namespaces to the Joker executable.

Formalities

Contributors

(Generated by Hall-Of-Fame)

License

Copyright (c) Roman Bataev. All rights reserved.
The use and distribution terms for this software are covered by the
Eclipse Public License 1.0 (http://opensource.org/licenses/eclipse-1.0.php)
which can be found in the LICENSE file.

Joker contains parts of Clojure source code (from clojure.core namespace). Clojure is licensed as follows:

Copyright (c) Rich Hickey. All rights reserved.
The use and distribution terms for this software are covered by the
Eclipse Public License 1.0 (http://opensource.org/licenses/eclipse-1.0.php)
which can be found in the file epl-v10.html at the root of this distribution.
By using this software in any fashion, you are agreeing to be bound by
the terms of this license.
You must not remove this notice, or any other, from this software.
You can’t perform that action at this time.