serializable-whitelist from a JVM property?

[jarohen]

Hey @ptaoussanis 😄

Re #130 - would you be open to defaulting the dynamic var to the value of a JVM property, say? In Crux it's our users' data that's getting frozen, so we don't know what classes they're using, and our users aren't necessarily using Clojure.

We're adding a property ourselves (xtdb/xtdb#1044) that we're checking on startup but I wonder whether this would be better living in Nippy?

Cheers,

James

[ptaoussanis]

@jarohen Hi James! Yes, this is a nice idea - would totally be up for that.
Should have an opportunity tomorrow- will get this in, cut a new release, and ping you.

Cheers

[ptaoussanis]

@jarohen Quick update: have sketched out some ideas for this, but would like to sleep on it before making any final choices. How time sensitive are you? Could this wait another day or two?

[jarohen]

Another day or two should be fine - thanks :)

[ptaoussanis]

@jarohen Is there a specific reason that you're using ":" as a separator at https://github.com/juxt/crux/pull/1044/files#diff-a7947e61ae94e39666030e564721083dR23 as opposed to ","?

[jarohen]

Not particularly - I've been working with classpaths a lot recently so maybe it was fresh in my mind. Comma'd be fine :)

[ptaoussanis]

[com.taoensso/nippy "2.15.1"] now on Clojars.

Release info, relevant API docs.

Please let me know if this addresses your issue? Cheers!

[jarohen]

That's great, cheers @ptaoussanis, and thanks for the fast turnaround 😄

I'll upgrade and run our tests against it - will let you know

[jarohen]

Looking good here 😄

Thanks, will close.

[qwwwpp]

Hi, I am trying to upgrade crux-core and crux-jdbc from 20.06-1.9.1-beta to 20.09-1.11.0-beta. Then I got an error at compile time

Exception in thread "crux-polling-tx-consumer" clojure.lang.ExceptionInfo: Thaw failed: Decryption/decompression failure, or data unfrozen/damaged. {:opts {:compressor :auto, :encryptor :auto}}
	at taoensso.nippy$thaw$fn__9291$ex__9292.invoke(nippy.clj:1638)
	at taoensso.nippy$thaw$fn__9291$fn__9313.invoke(nippy.clj:1689)
	at taoensso.nippy$thaw$fn__9291$thaw_data__9294.invoke(nippy.clj:1664)
	at taoensso.nippy$thaw$fn__9291.invoke(nippy.clj:1688)
	at taoensso.nippy$call_with_bindings.invokeStatic(nippy.clj:1189)
	at taoensso.nippy$call_with_bindings.invoke(nippy.clj:1172)
	at taoensso.nippy$thaw.invokeStatic(nippy.clj:1631)
	at taoensso.nippy$thaw.invoke(nippy.clj:1608)
	at taoensso.nippy$thaw.invokeStatic(nippy.clj:1621)
	at taoensso.nippy$thaw.invoke(nippy.clj:1608)
	at crux.jdbc$eval8951$fn__8952.invoke(jdbc.clj:45)
	at clojure.lang.MultiFn.invoke(MultiFn.java:234)
	at crux.jdbc.JdbcDocumentStore$fn__9003.invoke(jdbc.clj:110)
	at clojure.core$juxt$fn__5822.invoke(core.clj:2594)
	at clojure.core$map$fn__5866.invoke(core.clj:2753)
	at clojure.lang.LazySeq.sval(LazySeq.java:42)
	at clojure.lang.LazySeq.seq(LazySeq.java:51)
	at clojure.lang.RT.seq(RT.java:535)
	at clojure.core$seq__5402.invokeStatic(core.clj:137)
	at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:24)
	at clojure.core.protocols$fn__8146.invokeStatic(protocols.clj:75)
	at clojure.core.protocols$fn__8146.invoke(protocols.clj:75)
	at clojure.core.protocols$fn__8088$G__8083__8101.invoke(protocols.clj:13)
	at clojure.core$reduce.invokeStatic(core.clj:6828)
	at clojure.core$into.invokeStatic(core.clj:6895)
	at clojure.core$into.invoke(core.clj:6887)
	at crux.jdbc.JdbcDocumentStore.fetch_docs(jdbc.clj:111)
	at crux.document_store.CachedDocumentStore.fetch_docs(document_store.clj:98)
	at crux.fork.ForkedDocumentStore.fetch_docs(fork.clj:188)
	at crux.tx.conform$tx_events__GT_docs.invokeStatic(conform.clj:168)
	at crux.tx.conform$tx_events__GT_docs.invoke(conform.clj:164)
	at crux.tx.InFlightTx.index_tx_events(tx.clj:283)
	at crux.tx$index_tx_log$fn__3630$fn__3635.invoke(tx.clj:422)
	at crux.tx$index_tx_log$fn__3630.invoke(tx.clj:413)
	at crux.tx$index_tx_log.invokeStatic(tx.clj:405)
	at crux.tx$index_tx_log.invoke(tx.clj:403)
	at crux.tx$__GT_polling_tx_consumer$fn__3655.invoke(tx.clj:445)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.base/java.lang.Thread.run(Thread.java:835)
Caused by: clojure.lang.ExceptionInfo: Thaw failed against type-id: 112 {:type-id 112}
	at taoensso.nippy$thaw_from_in_BANG_.invokeStatic(nippy.clj:1556)
	at taoensso.nippy$thaw_from_in_BANG_.invoke(nippy.clj:1415)
	at taoensso.nippy$thaw$fn__9291$thaw_data__9294.invoke(nippy.clj:1662)
	... 36 more
Caused by: clojure.lang.ExceptionInfo: Thaw failed against type-id: 46 {:type-id 46}
	at taoensso.nippy$thaw_from_in_BANG_.invokeStatic(nippy.clj:1556)
	at taoensso.nippy$thaw_from_in_BANG_.invoke(nippy.clj:1415)
	at taoensso.nippy$read_kvs_into$fn__9250.invoke(nippy.clj:1314)
	at clojure.lang.LongRange.reduce(LongRange.java:233)
	at clojure.core$reduce.invokeStatic(core.clj:6827)
	at clojure.core$reduce.invoke(core.clj:6810)
	at taoensso.encore$reduce_n.invokeStatic(encore.clj:879)
	at taoensso.encore$reduce_n.invoke(encore.clj:878)
	at taoensso.nippy$read_kvs_into.invokeStatic(nippy.clj:1314)
	at taoensso.nippy$read_kvs_into.invokePrim(nippy.clj)
	at taoensso.nippy$thaw_from_in_BANG_.invokeStatic(nippy.clj:1487)
	... 38 more
Caused by: clojure.lang.ExceptionInfo: Cannot thaw object: `*serializable-whitelist*` check failed. See docstring for details. {:class-name "java.time.OffsetDateTime"}
	at taoensso.nippy$read_serializable_depr1.invokeStatic(nippy.clj:1377)
	at taoensso.nippy$read_serializable_depr1.invoke(nippy.clj:1373)
	at taoensso.nippy$thaw_from_in_BANG_.invokeStatic(nippy.clj:1541)

I figured out that this issue is related to the error, but I still can not find a way to add java.time.OffsetDateTime to serializable-whitelist.

Anybody knows how to do that? (I am using [com.taoensso/nippy "2.15.3"])

[ptaoussanis]

Hi @qwwwpp!

Caused by: clojure.lang.ExceptionInfo: Cannot thaw object: *serializable-whitelist* check failed. See docstring for details. {:class-name "java.time.OffsetDateTime"}
at taoensso.nippy$read_serializable_depr1.invokeStatic(nippy.clj:1377)
at taoensso.nippy$read_serializable_depr1.invoke(nippy.clj:1373)
at taoensso.nippy$thaw_from_in_BANG_.invokeStatic(nippy.clj:1541)

I figured out that this issue is related to the error, but I still can not find a way to add java.time.OffsetDateTime to serializable-whitelist.

Have you tried checking the docstring of taoensso.nippy/*serializable-whitelist*? It would be helpful for me to know if something in the docstring was unclear.

You have a few options here, including:

1. Execute the following code before Nippy is used:

(alter-var-root #'taoensso.nippy/*serializable-whitelist*
  (fn [old-whitelist]
    (conj old-whitelist "java.time.OffsetDateTime")))

2. Set an environment variable:

#!/bin/bash
export TAOENSSO_NIPPY_SERIALIZABLE_WHITELIST_ADD='java.time.OffsetDateTime'
# or
export TAOENSSO_NIPPY_SERIALIZABLE_WHITELIST_ADD='java.time.OffsetDateTime,other.class.1,other.class.2'
# etc.

3. Set a JVM property:

-Dtaoensso.nippy.serializable-whitelist-add=java.time.OffsetDateTime
or
-Dtaoensso.nippy.serializable-whitelist-add=java.time.OffsetDateTime,other.class.1,other.class.2
etc.

All of these should do the trick, feel free to use whatever's most convenient.
In case it's helpful, you can also use wildcards ("*") in class names.

Apologies for the inconvenience, hope that helps!

Cheers :-)

[qwwwpp]

Thank you for your help.
I think I misunderstood this line - TAOENSSO_NIPPY_SERIALIZABLE_WHITELIST_ADD env var .

I was using yogthos/config to control all my configurations, so I tried to change env var in it.
Also I didn't realize alter-var-root is another solution
https://github.com/ptaoussanis/nippy/blob/v2.15.1/src/taoensso/nippy.clj#L427

[ptaoussanis]

Great, thanks for confirming!