Exception: namespace 'flambo.function' not found

[alilee]

Getting this error message requiring flambo.api:

clojure.lang.Compiler$CompilerException: java.lang.Exception: namespace 'flambo.function' not found

It seems to be resolved if I aot it:

:aot [flambo.function]

in the project.clj

Any thoughts?

[sorenmacbeth]

See the section on AOT in the README https://github.com/yieldbot/flambo#aot

[alilee]

The requiring namespaces are aot'd as per the readme.
This exception shows up when flambo.api refers in the flambo.function vars and seems to require flambo.function to be aot'd.

[sorenmacbeth]

since flambo.api requires flambo.function, AOTing the all namespaces that require flambo.api should be all that's required. If you've already done this, perhaps you can make me a gist of your project.clj and I might be able to help further.

[alilee]

https://gist.github.com/alilee/565d0097b4c36edcbd26

I'm starting to have issues when compiling/re-constructing serialised functions with closures (referencing let or param bindings inside the f/fn). Could this be related?

Thanks very much for your help.

[alilee]

I think this could be related to some subtleties in how serialised-fn works - there isn't much guidance in the docs. This is what I'm having trouble with:

(let [x {:f identity :g parse/parse-int}]
  (-> sc 
      (f/parallelize [1 2 3 4 5])
      (f/map (f/fn [i] x))
      (f/take 1)))

If you take :g parse/parse-int out, then it works fine. When I leave it in it gets clojure.lang.Compiler$CompilerException: java.lang.RuntimeException: Unable to resolve symbol: x in this context, compiling:(NO_SOURCE_PATH:0:0) coming back from the worker.

parse is :aot.

[sorenmacbeth]

parse/parse-int must also be a serializable-fn, otherwise it cannot be
serialized by your anonymous serializable-fn.

On Tuesday, December 16, 2014, Alister Lee notifications@github.com wrote:

I think this could be related to some subtleties in how serialised-fn
works - there isn't much guidance in the docs. This is what I'm having
trouble with:

(let [x {:f identity :g parse/parse-int}](-> sc
%28f/parallelize [1 2 3 4 5]%29
%28f/map %28f/fn [i] x%29%29
%28f/take 1%29))

If you take :g parse/parse-int out, then it works fine. When I leave it
in it gets clojure.lang.Compiler$CompilerException:
java.lang.RuntimeException: Unable to resolve symbol: x in this context,
compiling:(NO_SOURCE_PATH:0:0) coming back from the worker.

parse is :aot.

—
Reply to this email directly or view it on GitHub
#42 (comment).

http://about.me/soren

[alilee]

I get the same issue with this:

(let [x {:f identity :g (f/fn [s] (parse/parse-int s))}]
  (-> sc 
      (f/parallelize [1 2 3 4 5])
      (f/map (f/fn [i] x))
      (f/take 1)))

Is that what you were suggesting?

[alilee]

Hmmm. And same with this:

(let [x {:f identity :g (f/fn [s] (identity s))}]
  (-> sc 
      (f/parallelize [1 2 3 4 5])
      (f/map (f/fn [i] x))
      (f/take 1)))

[sorenmacbeth]

try moving your let statement directly around the anonymous fn (let [x ...] (f/fn [i] x))

[alilee]

(use 'serializable.fn)

(deserialize 
  (serialize 
    (let [x + 
          c {:f identity :g (f/fn [s] (identity s))}
      (fn [a b] (:f c)))))
;; clojure.lang.Compiler$CompilerException: java.lang.RuntimeException: Unable to resolve symbol: c in this context, compiling:(...)

Is this the issue? (from serializable-fn):
TODO
Recurse into collections with the deep function serialization

[sorenmacbeth]

No, that is an old TODO. Deep recursion works. I'll try your example when
I'm back at a computer.

On Tuesday, December 16, 2014, Alister Lee notifications@github.com wrote:

(use 'serializable.fn)

(deserialize
(serialize
(let [x +
c {:f identity :g (f/fn [s](identity s))}
(fn [a b](:f c)))))
;; clojure.lang.Compiler$CompilerException: java.lang.RuntimeException: Unable to resolve symbol: c in this context, compiling:(...)

Is this the issue? (from serializable-fn):
TODO https://github.com/yieldbot/serializable-fn#todo
Recurse into collections with the deep function serialization

—
Reply to this email directly or view it on GitHub
#42 (comment).

http://about.me/soren

[alilee]

Thanks a million. I think that is the issue though. Is your version ahead of yieldbot's?

(let [x  {:f identity :g (f/fn [s] (identity s))}   ;; x would break mapping fi but y works
      y  (f/fn [s] (parse/parse-int s))
      fi (f/fn [i] (y i))]
  (-> sc 
      (f/parallelize ["9" "2" "3" "4" "5"])
      (f/map fi)
      (f/take 1)))
;; [9]

[sorenmacbeth]

So-- it seems like there is a bug in the deep recursion where elements of a map are not recursed into for serialization properly. it is the :f identity key-value pair causing the issue. kryo is trying to serialize an instance of clojure.lang.AFunction, which is can't do. I'll see if I can fix the bug.

[alilee]

Fantastic - thanks very much again for all your help. I will test!

[sorenmacbeth]

Ok, so it's not really a bug per se. serializable.fn uses a library called carbonite, which is a library for using kryo to serialize clojure data structures. When serializable.fn serializes an fn and it's bindings, it simply calls kryo's serialize, which is turn checks the class and then in the case of a clojure map, calls carbonite here:

https://github.com/sritchie/carbonite/blob/master/src/clj/carbonite/serializer.clj#L62-L69

carbonite doesn't know how to serialize a standard clojure function, so it effectively just fails to do so.

Unfortunately there isn't a great fix, other than not using functions as keys or values in a map if you wish it to be serializable :/

[chrisbetz]

Or just try the stuff from PR #27. I did take it into my fork (https://github.com/chrisbetz/flambo) and it works. I could check for you later today or you give it a try yourself.

Cheers,
Chris

Am 18.12.2014 um 05:22 schrieb Soren Macbeth notifications@github.com:

Ok, so it's not really a bug per se. serializable.fn uses a library called carbonite, which is a library for using kryo to serialize clojure data structures. When serializable.fn serializes an fn and it's bindings, it simply calls kryo's serialize, which is turn checks the class and then in the case of a clojure map, calls carbonite here:

https://github.com/sritchie/carbonite/blob/master/src/clj/carbonite/serializer.clj#L62-L69

carbonite doesn't know how to serialize a standard clojure function, so it effectively just fails to do so.

Unfortunately there isn't a great fix, other than not using functions as keys or values in a map if you wish it to be serializable :/

—
Reply to this email directly or view it on GitHub.

[sorenmacbeth]

That works, but not in the repl. You'll have to AOT all your code, losing the ability to do ad-hoc or interactive spark stuff which may or may not matter to you.

@chrisbetz I would be quite curious to see if that works in your fork or not.

[chrisbetz]

Hi,

took me some time. You might want to check commit 99c998c in my fork for the test, which is "green".

Happy holidays,

Chris