Reusing tasks in bb.edn by downloading from a URL

[weavejester]

I've been looking into replacing my use of Leiningen with a mix of tools.deps, tools.build and the Babashka task runner. It's a toolchain with a lot of advantages, but I find myself having to write a large bb.edn file that I have to copy between projects and manually update.

I may have missed something, but I don't believe there's currently a good solution to this?

One possible idea to solve this is to add a key that contains vector of base URLs that resolve to edn maps of tasks. The base tasks would be slurped, read and the local tasks merged onto it. e.g.

(defn get-tasks [{{:keys [bases] :as tasks} :tasks}]
  (let [base-tasks (map (comp edn/read-string slurp) bases)]
    (apply merge (conj base-tasks tasks)))

;; bb.edn
{:tasks
 {:bases ["https://raw.githubusercontent.com/example/bbedn/master/tasks.edn"]
  local  {:doc "A local task that overrides a base one of the same name."}}

It also seems reasonable to cache the bases, and to provide some sort of configurable timeout:

{:tasks
 {:bases [{:url "https://raw.githubusercontent.com/example/bbedn/master/tasks.edn"
           :cache-timeout 3600}]}

Let me know if this seems at all reasonable, and I can put together a PR.

[borkdude]

I may have missed something, but I don't believe there's currently a good solution to this?

True, but it sounds reasonable and convenient.

Any reason you're not using the already existing dependency mechanism in your proposal?

{:deps {tasks/base {:git/url "https://github.com/example/bbedn" ...}}
 :tasks
 {:bases [{:cp "tasks.edn"}]}

I like the idea of being able to specify the source of the bb.edn to not always have to use the classpath, e.g. a local file {:file "../bb.edn"} can also work.

[borkdude]

Forgot to mention: let's do an experimental PR and see how this pans out.

[borkdude]

More questions:

• Tasks can rely on dependencies. Should bases merge dependencies? Or should mergeable tasks not rely on dependencies?
• Similar for the :init block: should the init block of a base be pre-pended?
• Perhaps you can post an example of a tasks.edn you have in mind to evaluate the above questions

[weavejester]

Any reason you're not using the already existing dependency mechanism in your proposal?

Only because I didn't think of it! That looks reasonable, and would negate the need to handle caching, as the version or commit hash would be specified directly. Would :cp work better as a key, or would :resource or the fuller :classpath be more descriptive?

Tasks can rely on dependencies. Should bases merge dependencies? Or should mergeable tasks not rely on dependencies?

If we say that mergeable tasks cannot include dependencies, then we can delay that feature until later. Until we know someone needs it, I'm inclined to say that it's simpler if the first iteration does not merge dependencies, and then we sidestep the problem for now.

Same reasoning around the :init block. I think it's better just to cut the functionality for now, as I don't believe there's anything that would stop us from adding it later, at least as far as I can see...

Perhaps you can post an example of a tasks.edn you have in mind to evaluate the above questions

Sure! Here's part of a tasks.edn I'm currently working on. I won't post the whole thing because it gets quite large, but this is a good sample:

{clean
 {:requires ([babashka.fs :as fs]) 
  :doc      "Remove the target folder"
  :task     (fs/delete-tree "target")}
 lint
 {:requires ([babashka.pods :as pods]
             [clojure.edn :as edn])
  :doc      "Lint the source files"
  :task
  (do (pods/load-pod "clj-kondo")
      (require 'pod.borkdude.clj-kondo)
      (let [results (eval '(let [src (-> (slurp "project.edn")
                                         edn/read-string
                                         (:src-dirs ["src"]))]
                             (-> (pod.borkdude.clj-kondo/run! {:lint src})
                                 (doto pod.borkdude.clj-kondo/print!))))]
        (when (-> results :findings seq)
          (System/exit 1))))}
 outdated
 {:doc  "Find outdated dependencies"
  :task (apply clojure "-Sdeps"
               (pr-str '{:deps {com.github.liquidz/antq {:mvn/version "1.3.0"}}})
               "-M" "-m" "antq.core"
               *command-line-args*)}}

It's probably clear why I'd like to factor all this out into an external dependency!

I also evaluated the "tools" functionality recently added to the Clojure CLI (i.e. clj -T...), but I found that the startup time was a significant disadvantage over Babashka.

[borkdude]

Yes, those all seem reasonable limitations to get started with. Instead of :classpath we could also name it :resource similar to how (io/resource ...) gets things from the classpath.

[weavejester]

My thought is that :file and :resource would mirror io/file and io/resource for names. Do you want me to put together an experimental PR for this?

[borkdude]

Yep :)

[borkdude]

Just a thought but as a convention, it might work if a tool that wants to offer a set of predefined tasks, used a prefix for its tasks, like:

bb tool:lint

This way, when tool's tasks would depend on each other, you would never risk overriding one if its internal dependencies?

Additionally we can think of some way of "importing" tasks from a set of predefined tasks, being explicit about which ones you want to have and what they are called?

{:tasks
  {:task-imports [{:resource ... :refer :all :as new-alias #_(or :unqualified) :rename {tool:lint lint}]{{

[weavejester]

What about the other way around. So the tasks in tasks.edn are generally unprefixed, but you can specify a particular prefix when you import them:

{:tasks
 {:task-imports [{:resource "...", :prefix "tool:"}]}}

[borkdude]

That could also work. I was thinking, if your tool became some kind of standard across projects (similar to what, say, lein is) then it would be nice if everyone was using the same defaults?

[weavejester]

Maybe - I hadn't quite thought that far ahead! It could be that the prefix used is just by convention; whatever's in the README's installation instructions. Similar to how str is usually the prefix for clojure.string.

[borkdude]

Makes sense.

[borkdude]

Some more ideas/notes:

1. An alternative to the above is to use plain functions.
   When you include a library on the classpath, you can already invoke functions using:

bb -m library/function foo bar baz

So if you expose tasks as normal functions, there isn't currently any additional coding needed, but you do need to call it using -m.

Mapping functions to tasks is done using {:tasks {task-name library/function}}

2. Use real namespaces for imported tasks. If you want to expose tasks for others to use, perhaps we can namespace them. This should already work when you include (ns foo) in init:

{:tasks {:init (ns reusable)
         my-task (prn :hello)}}

But the merge logic should be made aware of the namespace stuff, so having a more declarative way of saying which namespace you're in is necessary. We can then perhaps also use :require etc to import other tasks.

[borkdude]

@weavejester Hmm yeah, option 1 from my previous comment seems way simpler. If we would have a way to map all public functions from a namespace to tasks, we would already be there almost right?

[borkdude]

@weavejester I made a proof of concept of 1 here:

https://github.com/borkdude/bb-issue-1103

Let me know what you think.

[borkdude]

I'm inclined to say this should be already sufficient:

{:deps {common/tasks
        {:git/url "https://github.com/borkdude/bb-issue-1103"
         :git/sha "4102881c929f31a7959180487001ddf68199de18"}}
 :tasks {lint     common.tasks/clj-kondo
         outdated common.tasks/antq}}

Just copy pasting the mappings into bb.edn isn't a lot of work and gives users control over the naming, without introducing additional complexity.

[weavejester]

Hey, that looks good! I didn't realize you could add symbols from a dependency like that to bb.edn. That should be perfectly fine for the use-case I had in mind.

[borkdude]

OK, I'm converting this to a discussion rather than something we should work on then, so we can continue there if more questions/ideas arise.