Stream closed exception when multiple pod pools are initialized

[mecha1]

While working on boot-garden I noticed a Stream closed exception being thrown when multiple pod pools are created and initialized on multiple invocations of the same task. We've gotten around this issue in boot-garden, but the fact that the exception occurs at all suggests a potential problem with boot, so I am posting this issue to hopefully help identify a fix. The issue can be reproduced with the build.boot file below:

(set-env! :dependencies '[[garden "1.2.5"]])

(require '[boot.core :as boot]
         '[boot.pod :as pod]
         '[clojure.java.io :as io])

(defn create-pool []
  (println "Creating pod pool")
  (pod/pod-pool (update-in (boot/get-env) [:dependencies] conj '[garden "1.2.5"])
                :init (fn [pod]
                        (println "  START pool init" (pr-str pod))
                        (pod/require-in pod 'garden.core)
                        (println "  END pool init" (pr-str pod)))))

(deftask pool-bug
  "Demonstrates bug that occurs when multiple pod pools are created on multiple invocations of the same task."
  [o option VAL str "Some option"]
  (let [tmp (boot/tmp-dir!)
        out (io/file tmp (str option ".css"))

        ; pods pool is created for each invocation of the pool-bug task
        pods (create-pool)]
    (boot/with-pre-wrap fileset
      (println "Get pod from pool")
      (let [pod (pods :refresh)]
        (io/make-parents out)
        (pod/with-eval-in pod
          (println "Eval in pod" ~(pr-str pod))
          (garden.core/css {:output-to ~(.getPath out)} [:h1 {:color "blue"}])))
      (-> fileset (boot/add-resource tmp) boot/commit!))))

If multiple invocations of the pool-bug task are called like so:
boot pool-bug -o 1 -- pool-bug -o 2 -- pool-bug -o 3

This can result in the following output and exception. The exception happens pretty reliably on my system, but because this seems to have something to do with the :init function being called concurrently, on some occasions the exception does not appear. If the exception doesn't occur for you, try rerunning the above command ~10 times, or add more invocations of the pool-bug task.

Full stacktrace

[martinklepsch]

Great report. I stripped some stuff from the build.boot:

(require '[boot.core :as boot]
         '[boot.pod :as pod])

(defn create-pool [sym-to-require]
  (println "Creating pod pool")
  (pod/pod-pool (boot/get-env)
                :init (fn [pod]
                        (println "START pool init" (pr-str pod))
                        (pod/require-in pod sym-to-require)
                        (println "END pool init" (pr-str pod)))))

(deftask pool-using-task
  "Demonstrates bug that occurs when multiple pod pools are created on multiple invocations of the same task."
  [o option VAL str "Some option"
   s req SYM sym "symbol to require during pod init"]
  (let [tmp (boot/tmp-dir!)
        ;; pods pool is created for each invocation of the pool-bug task
        pods (create-pool req)]
    (boot/with-pre-wrap fileset
      (println "Get pod from pool")
      (let [pod (pods :refresh)]
        (pod/with-eval-in pod
          (println "Eval in pod" ~(pr-str pod))))
      (-> fileset (boot/add-resource tmp) boot/commit!))))

(deftask trigger-bug
  [n num NUM int "number of tasks to run"
   s req SYM sym "symbol to require during pod init"]
  (apply comp
         (for [i (range num)]
           (pool-using-task :option (str i) :req req))))

To exclude the issue being related to Garden I tested with a bunch of other libs could reproduce with all of the following:

• garden.core or garden.compiler
• compojure.core
• schema.core
• aws.sdk.s3
• ring.util.io

The build.boot file provides a trigger-bug task that you can call like this:

boot -d clj-aws-s3 trigger-bug --num 4 --req aws.sdk.s3

[martinklepsch]

@jeluard in martinklepsch/boot-garden#7 (comment)

I also see the stream closed issue pretty frequently depending on my boot configuration.