Proposal for Java Support (orchestrated by skaffold)

[loosebazooka]

This is a proposal for java builds orchestrated by skaffold (this issue is separate from a maven/gradle skaffold plugin which delegate orchestration to the build tool -- instead of skaffold)

It seems like in general skaffold and the build tools have overlapping responsibilities and it's a little hard to decide which tools owns which part of the process.

I'm just opening this issue so we can start a discussion. What follows is one (of many) possible way to connect java build tools with skaffold.

apiVersion: skaffold/v1alpha2
kind: Config
build:
  artifacts:
  - imageName: gcr.io/k8s-java-skaffold/maven-app
    workspace: ./maven-app
    maven: {}
  - imageName: gcr.io/k8s-java-skaffold/gradle-app
    workspace: ./gradle-app
    gradle: {}
deploy:
  kubectl:
    manifests:
    - ./maven-app/src/kubernetes/*
    - ./gradle-app/src/kubernetes/*

Java Build Tools (gradle/maven) will define a convention for running a build to docker environments. (this is what skaffold calls when it decides a build needs to be done)

./mvnw install -Pskaffold

./gradlew skaffold

The user must use this convention to define the profile (maven) or task (gradle) that triggers the correct configuration and goals/task where the end result is an image on the docker daemon.

Triggering skaffold: There are two options

1. Watch source (kind of matches the skaffold convention better)
   • maven: ./pom.xml, ./src, ...
   • gradle: ./build.gradle, ./settings.gradle, ./gradle.properties, ./src, ...
2. Watch build output (would require the user to do a build first)
   • maven: ./target/*
   • gradle: ./build/*

Any ideas, potential problems?
@dgageot @glaforge @coollog @patflynn @r2d4

[loosebazooka]

Had a chat with @balopat about this. Exploring punting this whole build tool specific plan for a multistage docker build (via documentation) or a generic builder that can be configured to trigger maven/gradle.

[balopat]

hey, I'm super-new to the project, but hopefully I can help in adding perspective. From what I gather, currently skaffold supports the multistage Dockerfile setup very nicely. I played around to see what kind of issues I'd run into with that approach:

https://github.com/GoogleContainerTools/skaffold/pull/527/files#diff-b77cc69b9137a6d07207bffdfe75066b

FROM maven as builder
COPY pom.xml .
RUN mvn dependency:resolve dependency:resolve-plugins
ADD src ./src
RUN mvn package

FROM gcr.io/google-appengine/openjdk
COPY --from=builder target/spring-boot-sample-simple-2.0.1.RELEASE.jar app.jar
CMD ["java", "-jar", "app.jar"]

This works for a single simple project for now and maybe even can be a good enough initial story?

I can see some benefits to a bespoke maven builder:

• with multistage Dockerfile the user would have to manually setup some of the things that could be inferred from the maven project (I think), e.g. final jar file, source folders, etc. although maven/gradel inference capabilities need to be double-checked
• with maven running in a build step, the download of the dependencies tends to get hairy and kill or the productivity benefits - that's why I separated the resolution of the dependencies and plugin dependencies to a separate step, which works, but it's far from perfect - with a skaffold maven build step, we could build locally, relying on the local caches of maven to build the artifact before building the docker image

[dgageot]

@loosebazooka Thanks for starting this thread!

For me, a good maven/gradle integration would have those properties:

• It supports a maven/gradle project as is it. The same way we support docker builds without changing them, we should support java builds without changing them. That could imply adding more configuration to the skaffold.yaml.
• It builds faster that a multi-stage docker build, mostly by leveraging local cache.
• Builds are triggered on code change, the same way it does for docker/bazel builds.

[patflynn]

+1 to @dgageot comments.

I have some questions about the last bullet though. e.g. 'Builds are triggered on code change..'

I think we do want to support that workflow, but I'm not sure that it will be the most productive for Java users.

A few things I'd like to better understand are:

1. What would be the performance impact on a typical dev machine of triggering maven build on file change while doing a series of edits on springboot petclinic using IntelliJ?
2. How should skaffold interact with the typical Java workflow of doing incremental IDE builds in IntelliJ and Eclipse?
3. What about static resources and hotswap? Developers usually expect a certain class of changes to live-reload, can we enable that with Skaffold?

I think 3. can probably wait, but @loosebazooka do you have ideas for 1 & 2?

[coollog]

In AFAIU, for Skaffold builds with Bazel, Skaffold:

1. Watches the files returned by bazel query.
2. Runs the bazel build on the target specified in skaffold.yaml.
3. Loads the target (image tar) into the Docker daemon.

Perhaps another way we could implement build tool specific integration is to have the user specify the files to watch (or some command output to parse for the files) and the command to run to do the build. The configuration could look something like:

build:
  artifacts:
  - imageName: gcr.io/k8s-java-skaffold/maven-app
    workspace: ./maven-app
    dependencies:
    - ./maven-app/src
    - ./maven-app/pom.xml
    command: ./mvnw install -Dimage.name={{.imageName}}

This way, it could support any arbitrary command-based build as is.

[jstrachan]

I can see both sides of the multi-stage docker file running the maven/gradle versus doing the maven/gradle first before Docker (so you can more easily customise proxies and caches). For doing releases the latter is better; for skaffold dev though, the more skaffold knows about the source code used in the Dockerfile the better it can watch for changes etc. For simplicity on Jenkins X we've stuck with single-stage Dockerfiles and do the maven/gradle outside of skaffold.

[balopat]

This thread is going to grow large and a lot of questions need to be still answered/decided: e.g. what if the user already has a "source to image" strategy? How are we going to plug that? (I like @coollog's approach of a generic script). What is the default "source to image" strategy, that we recommend? Do we detect that? There are differences building executable slim JARs, fat JARs, WARs deployed to different application servers. Hot-swap, static resources, incremental IDE builds mentioned by @patflynn are an interesting area for optimization as well. And finally @jstrachan's comment about the CI/CD usecase is a very interesting one too and might shift the implementation priorities within that user story.

I think I have enough material to compile a proposal soon (tomorrow) in form of a markdown doc - we can continue commenting on that, if that format is okay with everyone.

[balopat]

Update: I underestimated the write-up - haven't got around to finish this today. I am actively working on it.

[balopat]

Update: had lots of conversations with people, didn't come to a final conclusion yet and I ran out of time - going to go on a leave today. However I believe @loosebazooka had a catchup with the team and came to a conclusion?

[loosebazooka]

As discussed, we are all okay with a generic build option, but it's not certain how to configure it.

I think there's a conflict between how skaffold is organized and how a contained build that doesn't use Dockerfiles would interface with it. The separation or build-config and builder makes things a little strange (for example: if someone configured a java-jib build and had kaniko set as their builder config, the behavior would not be defined)

Anyway, the target for the custom build is to configure everything at the artifact level. And just interface with local: {} for now.

There are a few things we need to deal with:

1. We need to understand the different artifact options:

   1. an image on the docker daemon <-- like a docker-maven-plugin
   2. an image on a container registry <-- like jib
   3. a tarbar <-- like bazel (but maybe we shouldn't cover this case)

2. How to ensure the build system is creating an artifact that is named correctly

   1. can skaffold.yaml template in values from itself?
   2. does skaffold config persist across build/deploy boundaries - we need the customBuilder to be aware of the tag that skaffold is using during deploy when we are configuring the build phase.

build:
  artifacts:
  - imageName: gcr.io/k8s-java-skaffold/maven-app
    workspace: ./maven-app   
    # here's where the generic build stuff comes in (sits with docker, bazel).
    custom:
      #skaffoldImageName (or whatever) is imageName:skaffoldInjectedTag
      command: ./mvnw install -Dimage.name={{.skaffoldImageName}}
      watch:      
      - ./maven-app/src
      - ./maven-app/pom.xml
  # the builder here might just be a push only (maven-docker-build/bazel) or no-op (jib)
  local: {}

• workspace is the root of the custom build, we do not look for Dockerfile
• adding in the custom option disables the docker build (like bazel)
• local now is just a registry pusher.
• the final artifact may either be on the docker daemon or on a registry, and that will affect how local is configured
• we might want some way to detect minikube config to configure docker env when running custom.command

[saturnism]

+1 on the custom builder. I do recommend supporting multiple commands:

build:
  artifacts:
  - imageName: gcr.io/k8s-java-skaffold/maven-app
    workspace: ./maven-app   
    # here's where the generic build stuff comes in (sits with docker, bazel).
    custom:
      #skaffoldImageName (or whatever) is imageName:skaffoldInjectedTag
      command:
      - ./mvnw install -Dimage.name={{.skaffoldImageName}}
      watch:      
      - src/**
      - pom.xml

Or, just delegate this through a plugin model. I'm not sure why Bazel got its own build block, but if we were to retrhink how to externalize bazel build, it will help w/ all the other build tools.

build:
  artifacts:
  - imageName: saturnism/demo
    workspace: .
    plugin:
      name: maven               # infers a second binary that conforms to an CLI API, e.g., skaffold-maven
                                # skaffold-maven query-files-to-watch
                                # skaffold-maven build-this-artifact ...

[balopat]

As the JIB builder is the recommended approach now, I'll close this issue. Please reopen/comment if you have other ideas.

[lazydevleo]

I have an microservice app with docker image being built by fabric8 maven plugin and I deploy to K8. Will this be supported by Skaffold currently or in the planned roadmap ? This would make the skaffold just a breeze