OpenRPC for Common Lisp

This framework is built on top of JSON-RPC and Clack. Comparing to JSON-RPC library, it provides these key features:

• Automatic OpenRPC spec generation.
• Automatic JSON-RPC client building by OpenRPC spec. This includes creation of Common Lisp classes and methods for making RPC requests and returning native CL objects.
• On both server and client sides your code looks very lispy, all JSON marshalling is done under the hood.

Server

OPENRPC-SERVER ASDF System Details

• Description: OpenRPC server implementation for Common Lisp.
• Licence: BSD
• Author: Alexander Artemenko
• Homepage: https://40ants.com/openrpc/
• Source control: GIT
• Depends on: alexandria, clack-cors, clack-prometheus, closer-mop, jsonrpc, lack-request, lambda-fiddle, local-time, log4cl, log4cl-extras, serapeum, str, websocket-driver, yason

Defining Methods

Here is an example of openrpc-server ASDF system allows to define JSON-RPC methods and data-structures they return.

Let's see how we can define an api for usual PetShop example.

Simple Example

First, we will operate on usual Common Lisp class:

(defclass pet ()
  ((id :initarg :id
       :type integer
       :reader pet-id)
   (name :initarg :name
         :type string
         :reader pet-name)
   (tag :initarg :tag
        :type string
        :reader pet-tag)))

Now we can define an RPC method to create a new pet:

(openrpc-server:define-rpc-method create-pet (name tag)
  (:summary "Short method docstring.")
  (:description "Lengthy description of the method.")
  (:param name string "Pet's name"
          :description "This is a long description of the parameter.")
  (:param tag string "Old param, don't use it anymore." :deprecated t)
  (:result pet)
  (let* ((new-id (get-new-id))
         (pet (make-instance 'pet
                             :id new-id
                             :name name
                             :tag tag)))
    (setf (gethash new-id *pets*)
          pet)
    pet))

Here we should explicitly specify type for each parameter and result's type.

Pay attention, the result type is PET. openrpc-server system takes care on serializing objects and you can retrieve an OpenRPC spec for any type, using type-to-schema generic-function:

CL-USER> (serapeum:toggle-pretty-print-hash-table)
T
CL-USER> (openrpc-server:type-to-schema 'pet)
(SERAPEUM:DICT
  "type" "object"
  "properties" (SERAPEUM:DICT
                 "id" (SERAPEUM:DICT
                        "type" "integer"
                       )
                 "name" (SERAPEUM:DICT
                          "type" "string"
                         )
                 "tag" (SERAPEUM:DICT
                         "type" "string"
                        )
                )
  "required" '("tag" "name" "id")
  "x-cl-class" "PET"
  "x-cl-package" "COMMON-LISP-USER"
 )

This method is used to render response requests to /openrpc.json handle of your api.

There is also a second generic-function which transform class instance into simple datastructures according to a scheme. For example, here is how we can serialize our pet:

CL-USER> (openrpc-server:transform-result
          (make-instance 'pet :name "Bobik"))
(SERAPEUM:DICT
  "name" "Bobik"
 ) 
CL-USER> (openrpc-server:transform-result
          (make-instance 'pet
                         :name "Bobik"
                         :tag "the dog"))
(SERAPEUM:DICT
  "name" "Bobik"
  "tag" "the dog"
 )

Returning Lists

To return result as a list of objects of some kind, use (:result (list-of pet)) form:

(openrpc-server:define-rpc-method list-pets ()
  (:result (list-of pet))
  (retrieve-all-pets))

Paginated Results

Sometimes your system might operate on a lot of objects and you don't want to return all of them at once. For this case, framework supports a keyset pagination. To use it, your method should accept LIMIT argument and PAGE-KEY argument. And if there are more results, than method should return as a second value the page key for retrieving the next page.

In this simplified example, we'll return (list 1 2 3) for the first page, (list 4 5 6) for the second and (list 7 8) for the third. Pay attention how VALUES form is used for first two pages but omitted for the third:

(openrpc-server:define-rpc-method list-pets (&key (limit 3) page-key)
  (:param limit integer)
  (:param page-key integer)
  (:result (paginated-list-of integer))

  (cond
    ((null page-key)
     (values (list 1 2 3)
             3))
    ((= page-key 3)
     (values (list 4 5 6)
             6))
    (t
      (list 7 8))))

Of cause, in the real world application, you should use PAGE-KEY and LIMIT arguments in the WHERE SQL clause.

Using Clack to Start Server

Framework is based on Clack. Use make-clack-app to create an application suitable for serving with CLACK:CLACKUP.

Then just start the web application as usual.

(clack:clackup (make-clack-app)
               :address interface
               :port port)

Also, you might use any Lack middlewares. For example, here is how "mount" middleware can be used to make api work on /api/ URL path, while the main application is working on other URL paths:

(defparameter *app*
  (lambda (env)
    '(200 (:content-type "text/plain") ("Hello, World!"))))

(clack:clackup
 (lambda (app)
   (funcall (lack.util:find-middleware :mount)
            app
            "/api"
            (make-clack-app)))
 *app*)

OpenRPC Spec

The key feature of the framework, is an automatic OpenRPC spec generation.

When you have your api up and running, spec will be available on /openrpc.json path. For our example project it will looks like:

{
  "methods": [
    {
      "name": "rpc.discover",
      "params": [],
      "result": {
        "name": "OpenRPC Schema",
        "schema": {
          "$ref": "https://raw.githubusercontent.com/open-rpc/meta-schema/master/schema.json"
        }
      }
    },
    {
      "name": "list-pets",
      "params": [
        {
          "name": "page-key",
          "schema": {
            "type": "integer"
          }
        },
        {
          "name": "limit",
          "schema": {
            "type": "integer"
          }
        }
      ],
      "result": {
        "name": "list-pets-result",
        "schema": {
          "type": "object",
          "properties": {
            "items": {
              "type": "array",
...

API

class openrpc-server/api:api ()

variable openrpc-server/api:*current-api* -unbound-

Points to a current api object when processing any RPC method.

macro openrpc-server/api:define-api (NAME &KEY (TITLE "Default API") (VERSION "0.1.0"))

reader openrpc-server/api:api-methods (api) (= (make-hash-table :test 'equal))

Returns a hash-table containing meta-information about all api methods.

Returned hash-table has key strings having methods names and internal objects of class method-info as values. I'm not sure if we need to export functions to manipulate with method info objects manually. Use openrpc-server/method:define-rpc-method macro to add or update RPC methods.

reader openrpc-server/api:api-title (api) (:TITLE = "Default API")

Returns a title of the api.

reader openrpc-server/api:api-version (api) (:version = "0.1.0")

Returns a version of the api.

macro openrpc-server/method:define-rpc-method name args &body body

Macro to define RPC method.

All arguments should have corresponding (:param arg type) form in the BODY.

Also, there should be one (:result type) form in the BODY.

generic-function openrpc-server/interface:type-to-schema type

This method is called for all types for which primitive-type-p generic-function returns NIL.

It should return as hash-table with JSON-SCHEMA corresponding to type. Keys of the dictionary should be strings. It is convenient to use SERAPEUM:DICT for building the result.

generic-function openrpc-server/interface:transform-result object

Prepares object for serialization before responding to RPC call.

Result should be list, hash-map or a value of primitive type.

generic-function openrpc-server/interface:primitive-type-p type

Should return t for type if it's name matched to simple types supported by JSON-SCHEMA.

Argument TYPE is a symbol.

generic-function openrpc-server/interface:make-info api

Returns a basic information about API for info section of OpenRPC spec.

function openrpc-server/errors:return-error message &key (code -1) (error-class 'jsonrpc/errors:jsonrpc-callback-error)

Raises an error to interrupt processing and return status to the caller.

generic-function openrpc-server/clack:make-clack-app api &key http websocket indent-json

Should return an app suitable for passing to clackup.

You can define a method to redefine application. But to add middlewares it is more convenient to define a method for app-middlewares generic-function.

generic-function openrpc-server/clack:app-middlewares api

Should return an plist of middlewared to be applied to the Clack application.

Keys should be a keyword with middleware name. And value is a function accepting a Clack application as a single argument and returning a new application.

Middlewares are applied to the app from left to right. This makes it possible to define an :around method which will inject or replace a middleware or original app.

Default method defines two middlewares with keys :CORS and :PROMETHEUS. (Prometheus middleware works on SBCL but not on CCL). To wrap these middlewares, add your middlewares to the end of the list. To add your middleware inside the stack - push it to the front.

function openrpc-server/clack:debug-on

function openrpc-server/clack:debug-off

generic-function openrpc-server/interface:slots-to-exclude type

You can define a method for this generic function to exclude some slots from being shown in the JSON schema.

Pay attention that this generic-function is called with class not with objects to be serialized. We need this because at the moment of generation api methods and OpenRPC spec we know nothing about objects except their classes.

Methods of this function should return a list of strings. Given slots will be excluded from the spec and will not be serialized. Strings are compared in case-insensitive mode.

Client

OPENRPC-CLIENT ASDF System Details

• Description: OpenRPC client implementation for Common Lisp.
• Licence: BSD
• Author: Alexander Artemenko
• Homepage: https://40ants.com/openrpc/
• Source control: GIT
• Depends on: alexandria, dexador, jsonrpc, kebab, log4cl, serapeum, str, usocket, yason

openrpc-client ASDF system provides a way to build CL classes and methods for working with JSON-RPC API. All you need is to give it an URL and all code will be created in compile-time as a result of macro-expansion.

Generating

For example, this macro call:

(generate-client petshop
                 "http://localhost:8000/openrpc.json")

Will generate the whole bunch of classes and methods:

(defclass petshop (jsonrpc/class:client) nil)

(defun make-petshop () (make-instance 'petshop))

(defmethod describe-object ((openrpc-client/core::client petshop) stream)
  (format stream "Supported RPC methods:~2%")
  (format stream "- ~S~%" '(rpc-discover))
  (format stream "- ~S~%"
          '(list-pets &key (page-key nil page-key-given-p)
            (limit nil limit-given-p)))
  (format stream "- ~S~%" '(create-pet (name string) (tag string)))
  (format stream "- ~S~%" '(get-pet (id integer))))

(defclass pet nil
  ((id :initform nil :initarg :id :reader pet-id)
   (name :initform nil :initarg :name :reader pet-name)
   (tag :initform nil :initarg :tag :reader pet-tag)))

(defmethod print-object ((openrpc-client/core::obj pet) stream)
  (print-unreadable-object (openrpc-client/core::obj stream :type t)
    (format stream " ~A=~S" 'id (pet-id openrpc-client/core::obj))
    (format stream " ~A=~S" 'name (pet-name openrpc-client/core::obj))
    (format stream " ~A=~S" 'tag (pet-tag openrpc-client/core::obj))))

(defmethod rpc-discover ((openrpc-client/core::client petshop))
  ...)

(defmethod list-pets
    ((openrpc-client/core::client petshop)
     &key (page-key nil page-key-given-p) (limit nil limit-given-p))
  ...)

(defmethod create-pet
    ((openrpc-client/core::client petshop) (name string) (tag string))
  ...)

(defmethod get-pet ((openrpc-client/core::client petshop) (id integer))
  ...)

Using

When client is generated, you need to make an instance of it and to connect it to the server:

(let ((cl (make-petshop)))
    (jsonrpc:client-connect cl :url "http://localhost:8000/" :mode :http)
    cl)

You can use any transport, supported by JSONRPC library.

DESCRIBE-OBJECT method is defined for a client, so you might see which methods are supported right in the REPL:

OPENRPC-EXAMPLE/CLIENT> (defvar *client* (make-test-client))
#<PETSHOP {1007AB2B13}>

OPENRPC-EXAMPLE/CLIENT> (describe *client*)
Supported RPC methods:

- (RPC-DISCOVER)
- (LIST-PETS &KEY (PAGE-KEY NIL PAGE-KEY-GIVEN-P)
             (LIMIT NIL LIMIT-GIVEN-P))
- (CREATE-PET (NAME STRING) (TAG STRING))
- (GET-PET (ID INTEGER))

And then to call these methods as usually you do in Common Lisp. Pay attention, that the library returns not JSON dictionaries, but ready to use CL class instances:

OPENRPC-EXAMPLE/CLIENT> (create-pet *client* "Bobik" "the dog")
#<PET  ID=1 NAME="Bobik" TAG="the dog">

OPENRPC-EXAMPLE/CLIENT> (create-pet *client* "Murzik" "the cat")
#<PET  ID=2 NAME="Murzik" TAG="the cat">

OPENRPC-EXAMPLE/CLIENT> (create-pet *client* "Homa" "the hamster")
#<PET  ID=3 NAME="Homa" TAG="the hamster">

Now, pay attention how pagination does work.

OPENRPC-EXAMPLE/CLIENT> (list-pets *client* :limit 2)
(#<PET  ID=1 NAME="Bobik" TAG="the dog">
 #<PET  ID=2 NAME="Murzik" TAG="the cat">)
#<FUNCTION (FLET OPENRPC-CLIENT/CORE::RETRIEVE-NEXT-PAGE :IN LIST-PETS) {1006D1F3CB}>

This call has returned a list of objects as the first value and a closure, which can be called to retrive the next page. Let's retrieve it now!

OPENRPC-EXAMPLE/CLIENT> (funcall #v167:1)
(#<PET  ID=3 NAME="Homa" TAG="the hamster">)

Now this is the last page and there is now a closure to retrieve the next page. Learn more how to implement pagination on server-side in the Paginated Results section.

macro openrpc-client/core:generate-client class-name url-or-path &key (export-symbols t)

Generates Common Lisp client by OpenRPC spec.

CLASS-NAME is the name of a API class. Also, a corresponding MAKE- function is created.

URL-OR-PATH argument could be a string with HTTP URL of a spec, or a pathname if a spec should be read from the disc.

Our Ask...

If you use this or find value in it, please consider contributing in one or more of the following ways:

1. Sponsor project at Patreon or Boosty and make a contribution.
2. Star it!
3. Share posts about it in social networks!
4. Fix an issue.
5. Add a feature (post a proposal in an issue first!).

Contributors

These people have contributed to OpenRPC. I'm so grateful to them!

• Kilian M. Haemmerle

---

[generated by 40ANTS-DOC]