Skip to content

Commit

Permalink
Updated readme, more details/etc
Browse files Browse the repository at this point in the history
  • Loading branch information
thebusby committed Nov 1, 2012
1 parent 3b6f3c6 commit eb91584
Showing 1 changed file with 18 additions and 9 deletions.
27 changes: 18 additions & 9 deletions README.md
@@ -1,30 +1,39 @@
# Iota

Iota is a simple library that applies a wrapper over Java NIO's mmap() functionalionality to produce a vector like object compatible with Clojure 1.5's reducers.
Iota is a simple library that applies a wrapper over Java NIO's mmap() functionalionality to produce a vector like object compatible with Clojure 1.5's reducers. Each element in the vector is associated with a single line in the file. O(1) lookup speed is maintained via a simple index.

This not only enables the use of reducers for greatly improved parallel performance, but allows text files to be stored in memory without java doubling or even tripling the amount of memory required (due to UTF-16 encoding, and String overhead).

Note: empty lines will return nil.

## Usage

(def mem-file (fvec "filename))
## Usage
```clojure
(def mem-file (iota/vec filename)) ;; Map the file into memory

(first mem-file)
(first mem-file) ;; Returns first line of file

(last mem-file)
(last mem-file) ;; Returns last line of file

(->> mem-file
(clojure.core.reducers/map #(->> (clojure.string/split % #"[\t]" -1)
(filter (fn [^String x] (not (.isEmpty x))))
;; Count number of non-empty fields in TSV file
(->> (iota/vec filename)
(clojure.core.reducers/filter identity) ;; filter out empty lines
(clojure.core.reducers/map #(->> (clojure.string/split % #"[\t]" -1)
(filter (fn [^String x] (not (.isEmpty x))))
count))
(clojure.core.reducers/fold +))

etc.
```



## License

CC0
http://creativecommons.org/publicdomain/zero/1.0/

I'd also like to thank my employer Gracenote, for allowing me to create this open source port.

NOTE: Relevant bits of iota/core.clj such as fjinvoke, fjfork, fjjoin, and various bits of iota/FileVector's implementation of Clojure interfaces were copied/modified from Rich Hickey's Clojure. These bits fall under the original owner's license (EPL at this time).


Expand Down

0 comments on commit eb91584

Please sign in to comment.