Dynamically set dependency classifiers depending on platform

[sbrunk]

I have a project where I depend on native artifacts which have a different classifier per platform. So in a bleep.yaml, I have the following dependency:

    - module: org.bytedeco:pytorch:1.13.0-1.5.9-SNAPSHOT
      publication:
        classifier: linux-x86_64 # could also be macosx-x86_64 or another platform or even a variant like linux-x86_64-gpu
        ext: jar
        name: pytorch
        type: jar

I can set the classifier manually and it totally works fine, but now I'd like to automatically detect the current platform and set the classifier accordingly (JavaCPP already has a function that can be used for that). To make things more complicated, there's also a gpu suffix in the mix which I'd like to be able to append to the classifier as an additional manual configuration.

For sbt there's a plugin called sbt-javacpp which does exactly that (among a few other things).

I'm fairly new to bleep, so I'd like to ask whether this is doable and what would be the best way to do it. I.e. using build rewrites?

[oyvindberg]

Just a preface: I'm not sure I ever added a dependency with a classifier in a project I worked on, so I don't know all that much about it.

For bleep I've tried to reduce the surface of dependencies somewhat, and drop a few ivyisms in particular. I begrudgingly kept the publication structure from coursier (which includes classifier) because I saw it was used, and my tests didn't pass without it.

So after adding that, I saw that typically all classifiers were added unconditionally, for instance in zio-http.

So my hypothesis was that it would always be fine to just spell all of them out and add all to the build, and then let your code choose to load whatever it needed from the jars, and that would be that.

Any insights into what kind of pain that approach can bring? :D

[sbrunk]

Yeah I guess I could use pytorch-platform as dependency which does essentially that, depending on all native artifacts. Picking the right native lib at runtime already works fine. The only downside is increased artifact size but I can live with that. I guess I could still offer a variant with a provided scope where downstream users could choose, or they can exclude the pytorch-platform dep and override or s.th. like that.

[sbrunk]

I'd still like to be able to easily switch to the GPU variant in the build without having to edit bleep.yaml manually. I figured I can create a build-rewrite like this:

package scripts

import bleep.{BleepScript, Commands, Started}

import bleep.rewrites.BuildRewrite
import bleep.model
import bleep.model.Dep.JavaDependency

object AddGPUClassifier extends BuildRewrite {
  override val name = model.BuildRewriteName("gpu")

  override protected def newExplodedProjects(oldBuild: model.Build): Map[model.CrossProjectName, model.Project] =
    oldBuild.explodedProjects.map { case (crossName, p) =>
      (
        crossName,
        p.copy(dependencies = p.dependencies.map {
          case d: JavaDependency if d.moduleName.value == "pytorch-platform" => d.copy(moduleName = d.moduleName.map(_ + "-gpu"))
          case d: JavaDependency if d.baseModuleName.value == "pytorch" && d.publication.classifier.nonEmpty =>
            d.copy(publication = d.publication.withClassifier(d.publication.classifier.map(_ + "-gpu")))
          case other => other
        })
      )
    }
}

object GPU extends BleepScript("GPU") {
  override val rewrites: List[BuildRewrite] = List(AddGPUClassifier)

  def run(started: Started, commands: Commands, args: List[String]): Unit = {
    started.logger.error("Adding gpu classifier for PyTorch")
  }
}

So I this seems to work fine creating a modified bloop config under builds/gpu. Is there a way to switch to that build somehow for the IDE or for tests?

[sbrunk]

The script writing experience is amazing BTW!

[oyvindberg]

So I this seems to work fine creating a modified bloop config under builds/gpu. Is there a way to switch to that build somehow for the IDE or for tests?

No, not possible. You could extend your script to call Commands#test within the builds/gpu build for tests, but there is no direct path to mounting it in the IDE.

My recommendation for bleep as it is today would be to create your build exactly as you want it in your IDE, with dependencies with all the classifiers you want. Then if that build differs from what you want to publish, you perform a build rewrite in your publish script.

Notes about publication

• Currently setting up publication requires you to copy/paste this script . support for codegen-ing common scripts #224 will make that much more smooth.
• There is additional flexibility if you implement a custom CoordinatesFor. You can do things such as having a build rewrite fork a project into multiple projects, which you again map to maven coordinates with classifiers in CoordinatesFor - for instance.
• If you want to publish with classifiers I think we'll need to patch GenLayout.{maven, ivy} to grab a bit more information from bleep.Dep (such as classifier). PR welcome for that.

Static builds

It is my belief that builds are simple, and simple things should be static and declarative. I'm trying to create a simple build tool (pardon the pun) for such simple projects, which I believe are ~all projects out there.

One meaning of "simple" is "easy to understand", but it can imply other qualities as well.
For instance consistency - The build will always yield the exact same list of dependencies.
Does that matter? Maybe! It's one less detail to worry about, and one thing less which can break.

Take for instance recent Apple machines: ARM64 machines with near-perfect x86_64 emulation. It's easy to flip-flop between architectures without noticing. Thousands of engineer hours have gone into that compatibility, from custom silicon all the way up to user-land code. And then it would reach a JVM build tool which breaks it, if you generated your build for one arch and ended up running it with another.

It is my hope that we can express enough builds without this kind of dynamism in the core of bleep. One of the luxuries of having a solid competitor like sbt is that it's ok for bleep to only support 99% of builds out there.

Dynamic builds in a static build world

There are builds which really, really are dynamic - they exist. Let's take the Scala 3 build as an example:

• (AFAIU) It first builds ("bootstraps") a compiler
• it then uses that bootstrapped compiler to publish, run tests and so on.

What would that look like in a bleep world?

Note sure yet. One idea is to write a publish-local script which takes the bootstrapped compiler, and (re-)generates a temporary, derived build file using that compiler. I think it makes perfect sense for it to be a derived build. It would also lift that fact up to be a first-class semantic and physical thing. (You're already doing this semantically with the gpu build rewrite, but this would take it a bit further and write it back to disk, so you can work on that build file instead. you could actually do that today, though I'm not sure how practical it would be)

Dynamic builds

So yeah, that's where I'm coming from. Then there is reality. Reality is that I don't know if my assumptions are good yet, so let's keep the discussion open going forwards.

You have come up with a somewhat enticing use-case, and I'm sure there will be others.

If we reach a decision to give users the power to do what you describe here, it'll likely be as a thing in the build file format. For instance the format already supports a limited form of string replacements, so you can say things like build.bleep::bleep-core:${BLEEP_VERSION}. We could in theory add something to support ${ARCH} as well.

Another option we have is to have a tool of your choice generate the build file, see #185 . Currently on the fence on that as well, for pretty much the reasons given above.

[oyvindberg]

The script writing experience is amazing BTW!

Awesome! It really feels so liberating after all this years of trying to express the same things inside sbt and similar :)

[sbrunk]

Thanks for the explanation. I completely agree with you regarding simple, static and declarative builds.

I'm realizing that what makes my use-case somewhat difficult is not so much the need to depend on native libs, which already works by adding all classifiers or perhaps in the future even better with ${ARCH} support as you've suggested.

The challenge is that there's also an additional GPU axis, and unfortunately, I can't just add the -gpu suffixed classifiers as well because they are exclusive. If both native libraries are on the classpath, the CPU version always wins so I can't use the CUDA version. I don't think I can control the loading order although I'd need to investigate how the mechanism works exactly.

Even if I could though, I probably want to control the behavior while hacking on the project, i.e. I want to disable GPU support for tests or debugging reasons. And I still might want to publish a variants with and without GPU support, that is, different dependencies.

The GPU config axis is of course quite project specific. Other than that, I expect the build to be quite static though, so I'm not giving up hope yet. :D

For hacking on the project it might be OK to switch between the variants manually, but then you'd have to be careful not to accidentally commit that change. Generating the build with #185 based on some local (not in git) config switch could be a cleaner way to do it. I.e. a bleep.scala script that outputs the default bleep.yaml but in case of a gpu_enabled config it outputs the rewritten, GPU variant.

But then I wonder, what if I could still have a default bleep.yaml (CPU) in git, but generate, as you've said above, a temporary derived build (GPU) via rewrite, and somehow select that build, for run/test/publish etc. (I realized for the IDE it wouldn't actually make any difference in my case, as difference is only native libs).

I'm not sure, would that bring us too far into dynamic/sbt land again? ;)

[oyvindberg]

Thanks for working with me through long explanations here @sbrunk :)

The challenge is that there's also an additional GPU axis, and unfortunately, I can't just add the -gpu suffixed classifiers as well because they are exclusive. If both native libraries are on the classpath, the CPU version always wins so I can't use the CUDA version. I don't think I can control the loading order although I'd need to investigate how the mechanism works exactly.

What if we pretend the dependency with classifier was just a dependency with a different maven coordinate - what would you have done then?

Maybe you can model this in your build as two separate projects, one which depend on the -gpu classifiers, and one which depend on the normal classifiers. Then you put all your source in either a) an upstream project (if there are interfaces you can code against outside these classifier dependencies), or b) a directory outside both projects which you add as a source folder to both projects:

projects:
  foo-gpu:
    sources: ../shared
  foo:
    sources: ../shared

That way you can work on the one you want, run tests against both (or only one, you'll still have all the normal source directories for each project), mount the one you want (or both!) in your IDE?

But then I wonder, what if I could still have a default bleep.yaml (CPU) in git, but generate, as you've said above, a temporary derived build (GPU) via rewrite, and somehow select that build, for run/test/publish etc. (I realized for the IDE it wouldn't actually make any difference in my case, as difference is only native libs).

If you don't need to mount the gpu build variant in your IDE, we are very close to a solution in this direction with current bleep as well. You have a build rewrite, and you can write scripts with that build rewrite for each of run, test, publish and whatever else. No need to persist the gpu build variant to file in this case.

It'll be a bit slower performing since you need to compile scripts and start a JVM before handing it off to bloop, but that's exactly what I wanted with bleep. If you do simple things it'll be fast, and if you need more complex things that's when you pay the performance cost of compiling and starting JVMs.

I'm not sure, would that bring us too far into dynamic/sbt land again? ;)

I think this direction with build variants produced by build rewrites is fairly promising, in that the builds are immutable (deriving a new build doesn't change the original build), and you have an original build which makes sense without the rewrites.

If we end up needing all this power, we could in theory lift the build variants into the build file as a first class thing:

build-variants:
  gpu: scripts/com.foo.ApplyGpuBuildRewrite 
projects: ...

ApplyGpuBuildRewrite would produce a new build variant output as yaml to stdout for instance.
Then you could tell bleep something like bleep --variant gpu compile, and it would pick that build variant.
There would just be some bookkeeping with running the script, caching the output, invalidating on changes to scripts sources and so on.

Thoughts? :)

[sbrunk]

What if we pretend the dependency with classifier was just a dependency with a different maven coordinate - what would you have done then?

This is something we actually have already available here with pytorch-platform and pytorch-platform-gpu, which in turn depend on the correct dependencies with classifier.

Maybe you can model this in your build as two separate projects, one which depend on the -gpu classifiers, and one which depend on the normal classifiers. Then you put all your source in either a) an upstream project (if there are interfaces you can code against outside these classifier dependencies), or b) a directory outside both projects which you add as a source folder to both projects:

projects:
  foo-gpu:
    sources: ../shared
  foo:
    sources: ../shared

I really like this approach. It's simple, static, easy to understand. Fits well into what you envision for bleep builds I guess. :)
And it would still work in a similar way if we might have ${ARCH} support at some point.

In fact, I just tried the shared sources variant (although an upstream project should work just as well here) with a template for common deps and pytorch-platform and pytorch-platform-gpu as dependencies respectively, and it seems to work fine. The only potential downside I'm seeing is that for a more modularized build, each sub-project needs to be split up into cpu and gpu variants as well.
But right now, with one or two sub-projects, I'm very much inclined to use this approach.

If you don't need to mount the gpu build variant in your IDE, we are very close to a solution in this direction with current bleep as well. You have a build rewrite, and you can write scripts with that build rewrite for each of run, test, publish and whatever else. No need to persist the gpu build variant to file in this case.

Even though I think I'll try the other approach for now, I'm curious how i.e. a run-gpu script could look like. After applying the rewrite, is it possible to re-use some of the bleep logic used in regular run for that?

If we end up needing all this power, we could in theory lift the build variants into the build file as a first class thing:

build-variants:
  gpu: scripts/com.foo.ApplyGpuBuildRewrite 
projects: ...

ApplyGpuBuildRewrite would produce a new build variant output as yaml to stdout for instance. Then you could tell bleep something like bleep --variant gpu compile, and it would pick that build variant. There would just be some bookkeeping with running the script, caching the output, invalidating on changes to scripts sources and so on.

Yes, this is pretty much what I had imagined how rewrites could be used when I first reading the docs. IMHO It's a powerful and quite general approach that could cover many use-cases where more flexibility is required.