Library to marshal C style data structures. For use with legacy TCP socket communication protocols and file formats (e.g. dbf file) . Marshaling to/from clojure data structures: maps, vectors, lists, numbers, booleans, and strings using OutputStream and InputStream interfaces. Marshal big or little endian (default is little endian).
Primitive marshaling types:
ubyte
- unsigned bytesbyte
- signed byteushort
- unsigned 2 byte integersshort
- signed 2 byte integeruint32
- unsigned 4 byte integersint32
- signed 4 byte integeruint64
- unsigned 8 byte integersint64
- signed 8 byte integerfloat
- 4 byte floating point numberdouble
- 8 byte floating point numberbool8
- 1 byte booleanbool32
- 4 byte boolean
Functions to create composite marshalling types:
array
- fixed or variable length of homogenous marshaling types - marshal to/from Clojure vectors (or lists)struct
- ordered keyword/marshal type pairs - marshal to/from Clojure mapsascii-string
- fixed of variable length ascii string - marshal to/from Clojure strings ()vector
- unamed ordered list of marshal types
Marshaling functions (API):
read
- marshals a value from an InputStreamwrite
- marshals a value to an OutputStram
say we have a C language struct (no byte alignment padding) that has size 34 bytes (4+10+5*4) that we wish to marshal
//C or C++ header file declaration
struct {
int type;
char name[10];
int data[5];
};
=> (require '[marshal.core :as m])
nil
;;declare a marshaling struct in the same order and using the same types as the C struct
=> (def s (m/struct :type m/sint32 :name (m/ascii-string 10) :data (m/array m/sint32 5)))
#'user/s
;;create an output stream (in lieu of an outputstream from a socket)
=> (def os (java.io.ByteArrayOutputStream.))
#'user/os
;;marshal a clojure map to the outputstream
=> (m/write os s {:type 1 :name "1234567890" :data [1 2 3 4 5]})
34
;;create an input stream (in lieu of an inputstream from a socket)
=> (def is (clojure.java.io/input-stream (.toByteArray os)))
#'user/is
;;marshal the clojure map from the input stream
=> (m/read is s)
{:type 1, :name "1234567890", :data [1 2 3 4 5]}
the following, perhaps more realistic looking example, reads a dBase file format e.g. ESRI shapefile attribute format. NOTE: it's just an example i.e. only numeric and strings are supported
(ns marshal.examples.dbf
(:require [marshal.core :as m]
[clojure.java.io]))
(def dbheaderfield (m/struct :name (m/ascii-string 11)
:type m/ubyte
:offset m/uint32
:field-size m/ubyte
:field-dec m/ubyte
:res1 (m/array m/ubyte 2)
:work_area_id m/ubyte
:res2 (m/array m/ubyte 10)
:prod_index m/ubyte))
(def dbheader (m/struct :type m/ubyte
:mod-year m/ubyte
:mod-month m/ubyte
:mod-day m/ubyte
:rec-count m/uint32
:header-size m/ushort
:rec-size m/ushort
:res1 (m/array m/ubyte 2)
:incomplete-trans m/ubyte
:encryption m/ubyte
:multi-user (m/array m/ubyte 12)
:prod-index m/ubyte
:lang-id m/ubyte
:res2 (m/array m/ubyte 2)
:fields (m/array dbheaderfield (fn [this]
(/ (- (:header-size this) 32 1)
(m/sizeof dbheaderfield))))
:terminator m/ubyte))
(defn read-column [s m field]
(let [sz (:field-size field)
name (keyword (:name field))
val (m/read s (m/ascii-string sz))]
(condp = (char (:type field))
\N (assoc! m name (try
(Integer/parseInt val)
(catch java.lang.NumberFormatException e
(Double/parseDouble val))))
\C (assoc! m name val)
m)))
(defn read-record [s fields]
(if (= 0x20 (.read s))
(loop [coll (seq fields) res (transient {})]
(if coll
(let [[f & xs] coll]
(recur xs (read-column s res f)))
(persistent! res)))))
(defn read-records [s]
(let [header (m/read s dbheader)
fields (:fields header)
sz (:rec-count header)]
(loop [i 0 res (transient [])]
(if (< i sz)
(recur (inc i) (conj! res (read-record s fields)))
(persistent! res)))))
(defn dbf [filename]
(binding [m/*byte-order* java.nio.ByteOrder/LITTLE_ENDIAN]
(with-open [s (clojure.java.io/input-stream filename)]
(read-records s))))
- Github: http://github.com/russellc/Marshal
- Clojars: https://clojars.org/marshal
(defproject project-uses-marshal "1.0.0"
:dependencies [[marshal "1.0.0"]])
Copyright (c) 2011 Russell Christopher All rights reserved. The use and distribution terms for this software are covered by the Eclipse Public License 1.0 (http://opensource.org/licenses/eclipse-1.0.php) which can be found in the file epl-v10.html at the root of this distribution. By using this software in any fashion, you are agreeing to be bound by the terms of this license. You must not remove this notice, or any other, from this software.