StackOverflowError on 1.3.186 but script properly working with 1.3.185

[ramonpin]

I have an script with a tail recursive function that works properly on babashka version 1.3.185 but fails with java.lang.StackOverflowError on 1.3.186.

version

1.3.186

platform

DISTRIB_ID=LinuxMint
DISTRIB_RELEASE=20
DISTRIB_CODENAME=ulyana
DISTRIB_DESCRIPTION="Linux Mint 20 Ulyana"

problem

The script brokens with java.lang.StackOverFlow but only on babashka 1.3.186.

repro

(ns empty
  (:require [clojure.string :as str]
            [babashka.fs :as fs]))

(defrecord FSEntry [permissions blocks user group size date time path])

(defn list-files! []
  (slurp "files.txt"))

(defn parse-files [files]
  (->> files
       (str/split-lines)
       (map #(re-seq #"\S+" %))
       (map #(apply ->FSEntry %))))

(defn path-components [file]
  (->> (fs/components file)
       (map #(str "/" %))
       (reductions #(str %1 %2))
       (apply hash-set)))

(defn dir? [entry]
  (str/starts-with? (.permissions entry) "d"))

(defn filter-empty-dirs
  ([entries]
   (let [groups (group-by #(dir? %) entries)
         dirs (map #(.path %) (groups true))
         files (map #(.path %) (groups false))]
     (filter-empty-dirs files dirs)))

  ([files dirs]
   (if (empty? files)
     dirs
     (let [[current-file & rest-files] files]
       (->> dirs
            (remove (path-components current-file))
            (recur rest-files))))))

(defn gen-minimal-deletes
  ([dirs]
   (gen-minimal-deletes (sort-by count dirs) []))

  ([pending to-remove]
   (if (empty? pending)
     to-remove
     (let [[current & more] pending
            current (str current "/")
            more (remove #(str/starts-with? % current) more)]
       (recur more (conj to-remove current))))))

;; Main clean process
(println "Prunning empty folders")
(->> (list-files!)
     (parse-files)
     (filter-empty-dirs)
     (gen-minimal-deletes)
     (run! println))

The files.txt file should be present at the current directory when the script gets executed.

expected behavior

When executed with babashka 1.3.185 it properly lists:

Prunning empty folders
/user/user2/tina/year=2023/month=10/day=25/
/user/user2/tina/year=2023/month=10/day=26/hour=03/

[ramonpin]

Strangely when I change the filter-empty-dirs function by adding a println at the start of its second header it works well with both babashka versions:

(defn filter-empty-dirs
  ([entries]
   (let [groups (group-by #(dir? %) entries)
         dirs (map #(.path %) (groups true))
         files (map #(.path %) (groups false))]
     (filter-empty-dirs files dirs)))

  ([files dirs]
   (println (count files) (count dirs))  # This line makes it work
   (if (empty? files)
     dirs
     (let [[current-file & rest-files] files]
       (->> dirs
            (remove (path-components current-file))
            (recur rest-files))))))

[borkdude]

I am repro-ing with docker with 1.3.185 and it also gives an SO:

docker run --rm -v $(pwd):/tmp babashka/babashka:1.3.185  bash -c 'cd /tmp; bb files.clj'

You can fix this by making remove strict before doing the recur:

(defn filter-empty-dirs
  ([entries]
   (let [groups (group-by #(dir? %) entries)
         dirs (map #(.path %) (groups true))
         files (map #(.path %) (groups false))]
     (filter-empty-dirs files dirs)))

  ([files dirs]
   (if (empty? files)
     dirs
     (let [[current-file & rest-files] files]
       (->> dirs
            (remove (path-components current-file))
            vec
            (recur rest-files))))))

[ramonpin]

Thnks!!!!

[ramonpin]

Strangely it does not fail for me when I use 1.3.185 docker image as you do. But it does when I use 1.3.186.

https://asciinema.org/a/1CqV7e3aLnpLMqeSvwIbccpmt

[borkdude]

It could be that the stack size is slightly different in your case, but lazy expression don't mix well with deeply recursive code and is known to cause such problems sooner or later.

[ramonpin]

Great!!! Thank you for your patience.