Implement matrix for bake targets

[jedevc]

🔨 Fixes #1150 and #1312

This PR implements the proposal from #1150 (comment), setting a matrix property on targets which can be used to easily parameterize builds such as #1312 (comment):

group "validate" {
  targets = ["validate-lint", "validate-vendor", "validate-docs", "validate-authors"]
}

target "validate-lint" {
  dockerfile = "./hack/dockerfiles/lint.Dockerfile"
  output = ["type=cacheonly"]
}

target "validate-vendor" {
  dockerfile = "./hack/dockerfiles/vendor.Dockerfile"
  target = "validate"
  output = ["type=cacheonly"]
}


target "validate-docs" {
  dockerfile = "./hack/dockerfiles/docs.Dockerfile"
  target = "validate"
  output = ["type=cacheonly"]
}

target "validate-authors" {
  dockerfile = "./hack/dockerfiles/authors.Dockerfile"
  target = "validate"
  output = ["type=cacheonly"]
}

With this PR, the above example can be simplified to:

group "validate" {
  targets = ["validate-lint", "validate-vendor", "validate-docs", "validate-authors"]
}

target "validate" {
  matrix = {
    tgt = ["lint", "vendor", "docs", "authors"]
  }
  name = "validate-${tgt}"
  dockerfile = "./hack/dockerfiles/${tgt}.Dockerfile"
  target = "validate"
  output = ["type=cacheonly"]
}

See the added tests for more information on the functionality - I'll add some docs once it's clear there's some consensus on the design + implementation.

There's still some work to be done around the matrix feature, though I'd be happy to do them as follow-ups, I think there's enough functionality to review in this PR. Specifically, we should consider:

• ✔️ Can we provide a shortcut to easily refer to all targets from a matrix without needing to manually specify them (see in the above example, where the ["lint", "vendor", "docs", "authors"] is effectively duplicated, once in the matrix definition and once in the group definition)? We could implicitly turn the old label from the target (validate in this case) into a group that contains all the targets (so the validate group would be automatically created). This feels like the simplest option, but I'm not sure about the semantics of automatically creating groups 🤔
• Should we allow for a GitHub-matrix like syntax, to include or exclude specific matrix combinations? I think this would be useful, though I'm not sure of the exact syntax we'd want for this.
• Should we support matrixes for groups as well? There's no reason why not, and adds consistency, but I struggle to see a concrete use case.

[crazy-max]

• Can we provide a shortcut to easily refer to all targets from a matrix without needing to manually specify them (see in the above example, where the ["lint", "vendor", "docs", "authors"] is effectively duplicated, once in the matrix definition and once in the group definition)? We could implicitly turn the old label from the target (validate in this case) into a group that contains all the targets (so the validate group would be automatically created). This feels like the simplest option, but I'm not sure about the semantics of automatically creating groups 🤔

I don't think it's related to the matrix proposal but #1312 to use target pattern rules with stem:

group "validate" {
  targets = ["validate-lint", "validate-vendor", "validate-docs", "validate-authors"]
}

target "validate-%" {
  dockerfile = "./hack/dockerfiles/${*}.Dockerfile"
  target = "${*}" != "lint" ? "validate" : ""
  output = ["type=cacheonly"]
}

[crazy-max]

With this PR, the above example can be simplified to:

group "validate" {
  targets = ["validate-lint", "validate-vendor", "validate-docs", "validate-authors"]
}

target "validate" {
  matrix = {
    tgt = ["lint", "vendor", "docs", "authors"]
  }
  name = "validate-${tgt}"
  dockerfile = "./hack/dockerfiles/${tgt}.Dockerfile"
  target = "validate"
  output = ["type=cacheonly"]
}

Should be:

group "validate" {
  targets = ["validate-lint", "validate-vendor", "validate-docs", "validate-authors"]
}

target "validate" {
  matrix = {
    tgt = ["lint", "vendor", "docs", "authors"]
  }
  name = "validate-${tgt}"
  dockerfile = "./hack/dockerfiles/${tgt}.Dockerfile"
  target = "${tgt}" != "lint" ? "validate" : ""
  output = ["type=cacheonly"]
}

[tonistiigi]

@ciaranmcnulty @cpuguy83

[tonistiigi]

Can we provide a shortcut to easily refer to all targets from a matrix without needing to manually specify them

Yes, please. There should not be a need to define an extra group with the same matrix values repeated as targets. When calling a target that defines matrix, all matrix combinations are built.

[jedevc]

Can we provide a shortcut to easily refer to all targets from a matrix without needing to manually specify them

Yes, please. There should not be a need to define an extra group with the same matrix values repeated as targets. When calling a target that defines matrix, all matrix combinations are built.

Done in 0806870.

For the input:

# no separate group necessary!

target "validate" {
  matrix = {
    tgt = ["lint", "vendor", "docs", "authors"]
  }
  name = "validate-${tgt}"
  dockerfile = "./hack/dockerfiles/${tgt}.Dockerfile"
  target = "${tgt}" != "lint" ? "validate" : ""
  output = ["type=cacheonly"]
}

$ buildx bake --print validate
[+] Building 0.0s (0/0)                                                                                                                                           
{
  "group": {
    "default": {
      "targets": [
        "validate"
      ]
    },
    "validate": {
      "targets": [
        "validate-lint",
        "validate-vendor",
        "validate-docs",
        "validate-authors"
      ]
    }
  },
  "target": {
    "validate-authors": {
      "context": ".",
      "dockerfile": "./hack/dockerfiles/authors.Dockerfile",
      "target": "validate",
      "output": [
        "type=cacheonly"
      ]
    },
    "validate-docs": {
      "context": ".",
      "dockerfile": "./hack/dockerfiles/docs.Dockerfile",
      "target": "validate",
      "output": [
        "type=cacheonly"
      ]
    },
    "validate-lint": {
      "context": ".",
      "dockerfile": "./hack/dockerfiles/lint.Dockerfile",
      "target": "",
      "output": [
        "type=cacheonly"
      ]
    },
    "validate-vendor": {
      "context": ".",
      "dockerfile": "./hack/dockerfiles/vendor.Dockerfile",
      "target": "validate",
      "output": [
        "type=cacheonly"
      ]
    }
  }
}

If the group is defined separately, it will be correctly merged with the previous target - the underlying representation is a little, but I think it's what the user expects.

[ciaranmcnulty]

My main use case for this sort of thing is this sort of thing:

PHP_VERSIONS=81,82 FLAVOURS=debian,alpine docker buildx bake

It looks good for the basic cases 👍

[ciaranmcnulty]

As it's generating the product of all the lists in matrix, is there any way to remove an item from the resulting generated targets?

I'm thinking of cases where a particular combination isn't supported (e.g. latest version of a package isn't available on Debian)

[jedevc]

As it's generating the product of all the lists in matrix, is there any way to remove an item from the resulting generated targets?

Yeah, I think we probably will want that, see my second point:

Should we allow for a GitHub-matrix like syntax, to include or exclude specific matrix combinations? I think this would be useful, though I'm not sure of the exact syntax we'd want for this.

I'm not quite sure on syntax, but it's also definitely a thing for a follow-up PR. The easiest suggestion that springs to mind:

target "validate" {
  matrix = {
    tgt = ["lint", "vendor", "docs", "authors"]
  }
  matrix-include = [
    {tgt = "another-option"}  # this option is added to the list of matrix combinations 
  ]
  matrix-exclude = [
    {tgt = "authors"}  # this option is removed from the list of matrix combinations
  ]

  name = "validate-${tgt}"
  dockerfile = "./hack/dockerfiles/${tgt}.Dockerfile"
  target = "${tgt}" != "lint" ? "validate" : ""
  output = ["type=cacheonly"]
}

It feels not great though, maybe it would make sense to group all of them under one single property, so that matrix.args is the block that actually contains the parameters, and then we could have matrix.include and matrix.exclude?

I'm not a huge fan of how GitHub mixes the include and exclude fields into the top-level, it feels unintuitive to me, but I could be convinced otherwise.

[ciaranmcnulty]

It seems to me it's trivial to add to the matrix (which is slightly different to GHA's include) by simply defining some explicit additional targets that don't use this new mechanism

[jedevc]

As another option instead of having a way of excludeing, we could potentially at some point in the future add another target option: skip. If skip is true, then we don't include it in the matrix, if it's false (the default), it is included.

For example:

target "test" {
  matrix = {
    driver = ["docker", "docker-container", "kubernetes", "remote"]
    test = ["a", "b", "c"]
  }
  name = "test-${driver}-${test}"
  skip = driver == "kubernetes" && test == "c"
}

I'd prefer something like this, instead of another matrix property of needing to nest them if possible - but let's hold off on adding anything to resolve this in the current PR.

[tonistiigi]

This should be public if we expect caller to implement it. And type checked with var _ WithEvalContexts = &Group{}

[jedevc]

Done. I've type checked for Group + Target for both the interfaces.

[tonistiigi]

LGTM but could you update docs as well

[jedevc]

We've moved the file definition into https://github.com/docker/docs, so I'll follow up there, cc @dvdksn.

[jedevc]

🎉 Started the docs work in docker/docs#17041.