[idea] Nippy Backend

[harold]

https://github.com/ptaoussanis/nippy is a wicked serialization library that we use all over the place for storing edn.

The main trade-offs of nippy vs pr-str are:

• pr-str is plain-text and nippy is binary byte blob (opaque)
• nippy is smaller on disk (sometimes ridiculously so, ~20x?!)
• nippy reads and writes faster

Now that I am storing non-trivial data in .edn duratoms, it occured to me that nippy might make an interesting backend for this lib.

Any thoughts? Any interest in a PR?

[jimpil]

Hi Harold,

That is a neat idea indeed! Unfortunately, I will not be in a position to make this happen for at least another 2 weeks (maybe 3), as I'm on holidays abroad. If you feel like tackling it yourself, then by all means, have a go ;)

Kind regards,
Dimitris

[harold]

Enjoy your holidays.

I will try this when I reach performance limitations, for now I am still good to go.

If you do do this at some point I'd be happy to try it out.

Take care.

[Sandarr95]

Part of my logic relies on sorted-sets in a project, appearently edn silently transforms these into hash-sets. Nippy would be a solution and I think I could manage to implement it.

Any ideas to let sorted-sets work with edn are welcome too though.

[harold]

@Sandarr95 - interesting, I never thought of that.

It looks like you're right (duratom uses pr-str for edn):

> (pr-str (sorted-set 1 2 3))
#{1 2 3}

Some indication that nippy may help:

> (type #{1 2 3})
#<Class@1f1c7bf6 clojure.lang.PersistentHashSet>
> (type (sorted-set 1 2 3))
#<Class@3d62648d clojure.lang.PersistentTreeSet>
> (type (taoensso.nippy/thaw (taoensso.nippy/freeze (sorted-set 1 2 3))))
#<Class@3d62648d clojure.lang.PersistentTreeSet>

[Sandarr95]

So for anyone reading this that wants a quick fix, I saw someone forked and implemented it:
https://github.com/aaronblenkush/duratom

I'm gonna take some time to get the abstractions right, so both edn and nippy will be available and more serialization methods and storage backends can be added without reworking the logic.

[jimpil]

Hi guys,

I'm not in favour of the approach taken by the fork. Therefore, I spent a couple of hours last night and took a different approach. Basically, I'd like to be able to do something whereby people who want to carry on using plain edn can do so without changing anything. So what I've come up with so far for nippy users is something along these lines:

;; FILE-BACKED
(file-atom file-path lock initial-value 
  {:read (partial ut/read-edn-from-bytes! nippy/thaw-from-in!)
   :write (partial ut/write-edn-as-bytes! nippy/freeze-to-out!)})

;; POSTGRES-BACKED
(postgres-atom db-config table-name row-id lock initial-value 
  {:read nippy/thaw
   :write nippy/freeze
   :column-type :bytea})

;; S3-BACKED
(s3-atom creds bucket k lock initial-value 
  {:read nippy/thaw
   :write nippy/freeze})

So basically, nippy users will have to provide an extra argument when creating their duratoms.I think that's fair, and we avoid the dependency. I've not actually pushed any changes yet, so you won't be able to see the full solution yet, but the above should give you a clear idea of the direction I'm taking. Feedback is always welcome.

[jimpil]

duratom 0.3.5 is out and allows for custom encoders/decoders to be hooked on to the back-end. See the readme which specifically shows an example utilising nippy. Please do come back to me if you happen to find any problems. Closing this now. Thanks again...

[harold]

👏