Adding support for Functions in crank validate (fixes #5491)

[jtyr]

Description of your changes

This PR is fixing issue described in the issue #5491. Without this fix, the crossplane beta validate execution fails because there is no image detected. There is still an issue with version constraints (e.g. >=v1.2.3) as the current implementation takes the version string as is and tries to fetch the image with that exact tag.

Fixes #5491

I have:

• Read and followed Crossplane's contribution process.
• Run make reviewable to ensure this PR is ready for review.
• Added or updated unit tests.
• Added or updated e2e tests.
• Linked a PR or a docs tracking issue to document this change.
• Added backport release-x.y labels to auto-backport this PR.

The PR was tested like this:

Clone the repo, apply patch and build a new crossplane version:

gh repo clone crossplane/crossplane
cd crossplane
curl -s https://patch-diff.githubusercontent.com/raw/crossplane/crossplane/pull/5570.patch | git apply
make submodules
make go.build
ln -s ./_output/bin/linux_amd64/crank ~/bin/crossplane

Create directory structure for the test:

mkdir -p {apis/foo.custom-api.example.org,test/{functions,validation}}

Create files:

# Configuration for the "foo" package
cat <<END > apis/configuration.yaml
apiVersion: meta.pkg.crossplane.io/v1alpha1
kind: Configuration
metadata:
  name: foo
spec:
  dependsOn:
    - function: xpkg.upbound.io/crossplane-contrib/function-patch-and-transform
      version: v0.4.0
    - provider: xpkg.upbound.io/upbound/provider-aws-s3
      version: v1.3.1
  crossplane:
    version: ">=v1.15.2"
END

# Foo composition
cat <<END > apis/foo.custom-api.example.org/composition.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: xfoos.custom-api.example.org
spec:
  compositeTypeRef:
    apiVersion: custom-api.example.org/v1alpha1
    kind: XFoo
  mode: Pipeline
  pipeline:
    - step: patch-and-transform
      functionRef:
        name: crossplane-contrib-function-patch-and-transform
      input:
        apiVersion: pt.fn.crossplane.io/v1beta1
        kind: Resources
        resources:
          - name: storage-bucket
            base:
              apiVersion: s3.aws.upbound.io/v1beta1
              kind: Bucket
              spec:
                forProvider:
                  region:  # Patched
            patches:
              - type: FromCompositeFieldPath
                fromFieldPath: spec.location
                toFieldPath: spec.forProvider.region
END

# XRD for the Foo composition
cat <<END > apis/foo.custom-api.example.org/definition.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xfoos.custom-api.example.org
spec:
  group: custom-api.example.org
  names:
    kind: XFoo
    plural: xfoos
  claimNames:
    kind: Foo
    plural: Foos
  versions:
    - name: v1alpha1
      schema:
        openAPIV3Schema:
          type: object
          required:
            - spec
          properties:
            spec:
              type: object
              required:
                - location
              properties:
                location:
                  type: string
END

cat <<END > test/claim.yaml
apiVersion: custom-api.example.org/v1alpha1
kind: Bar
metadata:
  name: mybar
spec:
  location: us-east-2
END

# Bar composition
cat <<END > test/composition.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: bar
spec:
  compositeTypeRef:
    apiVersion: custom-api.example.org/v1alpha1
    kind: Bar
  mode: Pipeline
  pipeline:
    - step: patch-and-transform
      functionRef:
        name: crossplane-contrib-function-patch-and-transform
      input:
        apiVersion: pt.fn.crossplane.io/v1beta1
        kind: Resources
        resources:
          - name: foo
            base:
              apiVersion: custom-api.example.org/v1alpha1
              kind: Foo
              spec:
                location:  # Patched
            patches:
              - type: FromCompositeFieldPath
                fromFieldPath: spec.location
                toFieldPath: spec.location
END

# Function used bu the Bar composition
cat <<END > test/functions/patch-and-transform.yaml
apiVersion: pkg.crossplane.io/v1beta1
kind: Function
metadata:
  name: crossplane-contrib-function-patch-and-transform
spec:
  package: xpkg.upbound.io/crossplane-contrib/function-patch-and-transform:v0.4.0
END

# Configuration for the "foo" package
cat <<END > test/validation/configuration.yaml
apiVersion: pkg.crossplane.io/v1beta1
kind: Configuration
metadata:
  name: foo
spec:
  package: localhost:5000/foo:v0.1.0
END

# XRD for the Bar composition
cat <<END > test/validation/definition.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xbar.custom-api.example.org
spec:
  group: custom-api.example.org
  names:
    kind: XBar
    plural: xbars
  claimNames:
    kind: Bar
    plural: Bars
  versions:
    - name: v1alpha1
      schema:
        openAPIV3Schema:
          type: object
          required:
            - spec
          properties:
            spec:
              type: object
              required:
                - location
              properties:
                location:
                  type: string
END

Run local Docker registry:

docker run -d -p 5000:5000 --restart always --name registry registry:2

Package and push the foo package:

crossplane xpkg build --package-root apis --package-file foo.xpkg
crossplane xpkg push --package-files foo.xpkg localhost:5000/foo:v0.1.0

Validate the Bar composition:

crossplane beta render test/claim.yaml test/composition.yaml test/functions --include-full-xr | crossplane beta validate test/validation -

Output without the patch (missing image):

package schemas does not exist, downloading:  xpkg.upbound.io/upbound/provider-aws-s3:v1.3.1
package schemas does not exist, downloading:  localhost:5000/foo:v0.1.0
package schemas does not exist, downloading:  :v0.4.0
crossplane: error: cannot download and load cache: cannot cache package dependencies: cannot download package :v0.4.0: cannot get config: parsing reference ":v0.4.0": could not parse reference: :v0.4.0

Output with the patch (image is found):

package schemas does not exist, downloading:  localhost:5000/foo:v0.1.0
package schemas does not exist, downloading:  xpkg.upbound.io/crossplane-contrib/function-patch-and-transform:v0.4.0
package schemas does not exist, downloading:  xpkg.upbound.io/upbound/provider-aws-s3:v1.3.1
[✓] custom-api.example.org/v1alpha1, Kind=Bar, mybar validated successfully
[✓] custom-api.example.org/v1alpha1, Kind=Foo, foo validated successfully
Total 2 resources: 0 missing schemas, 2 success cases, 0 failure cases

When I change the apis/configuration.yaml to use version constraints:

apiVersion: meta.pkg.crossplane.io/v1alpha1
kind: Configuration
metadata:
  name: foo
spec:
  dependsOn:
    - function: xpkg.upbound.io/crossplane-contrib/function-patch-and-transform
      version: ">=v0.4.0"
    - provider: xpkg.upbound.io/upbound/provider-aws-s3
      version: ">=v1.3.1"
  crossplane:
    version: ">=v1.15.2"

The output even with the patch is this (image tag is the same as the version constraint and the validator fails to find such image):

package schemas does not exist, downloading:  localhost:5000/foo:v0.1.0
package schemas does not exist, downloading:  xpkg.upbound.io/crossplane-contrib/function-patch-and-transform:>=v0.4.0
crossplane: error: cannot download and load cache: cannot cache package dependencies: cannot download package xpkg.upbound.io/crossplane-contrib/function-patch-and-transform:>=v0.4.0: cannot get config: parsing reference "xpkg.upbound.io/crossplane-contrib/function-patch-and-transform:>=v0.4.0": could not parse reference: xpkg.upbound.io/crossplane-contrib/function-patch-and-transform:>=v0.4.0

[jbw976]

Awesome @jtyr, thanks for taking the initiative to fix this issue too! 🙇‍♂️

Some high level questions/feedback:

• Is it not necessary to also add Functions to the block in https://github.com/crossplane/crossplane/blob/master/cmd/crank/beta/validate/manager.go#L118-L134? when i did a quick look earlier, that's where I thought needed to be updated - is that for a different scenario maybe?
• Do you have some details you can add to the PR body for how you tested this change? e.g. what manifests/commands did you run and what was their output to show that the functionality is now correct?

[jtyr]

Thanks for your feedback, @jbw976.

I have updated the PR description with the way how this was tested.

Not sure if the portion of the code you are referring to is relevant to this issue. I was just following the places where code is failing and fixed it so it doesn't fail anymore. But if you wish, I can add the missing portion of the code to this PR as it looks it's missing there anyway. I have tested it already and it doesn't seem to influence the functionality of this fix in any way.

[phisco]

I don't think we need to update that portion of the code, we don't have XRDs in functions, and the only CRD should be the one for the input, which we don't need.

[jtyr]

@jbw976 If we don't have to add any other code then let's merge it so it at least works well for Configurations without version constraints.

[github-actions]

Successfully created backport PR for release-1.15:

• [Backport release-1.15] Adding support for Functions in crank validate (fixes #5491) #5585