Example of how to create a CRD #96

davesmith00sky · 2023-04-26T11:22:18Z

Between being fairly new to k8s and scala-k8s, I'm struggling to figure out how to create a CRD from a manifest. I've tried a few things, but essentially what I'm trying to do is simply the equivalent of:

kubectl apply -f my-crd-manifest.yaml

Where my-crd-manifest.yaml contains the definition of a CRD.

Could you point me in the right direction please?

Once I've got it set up, I've got some custom look ups defined that are based on the example in the readme. Can I just check if that is the right way to go about that?

Thanks in advance.

The text was updated successfully, but these errors were encountered:

github-actions · 2023-04-26T11:22:50Z

Thank you for submitting this issue and welcome!
We are glad that you are here and we appreciate your contributions.

hnaderi · 2023-04-26T12:05:26Z

Hi,
You can create a CustomResourceDefinition like any other kubernetes objects, also if you have a manifest in yaml, you can also use manifests module to parse it directly from the file.

import dev.hnaderi.k8s.manifest
val obj = manifest.parse("file content")

Having a custom resource object at hand, you can then create any API requests you want.
Note that kubectl apply is not a single API call in most cases, it consists of a get call to read the state and decide what to do next, for example update using put or patch based on different object behaviors.

Also, for creating your own custom requests, you need to implement dev.hnaderi.k8s.client.CreateRequest as custom resource APIs are not included in the current client, however adding it is trivial; let me know if you want them so I can work on adding them.

davesmith00sky · 2023-04-26T12:49:27Z

Right, I think I have a clue where I've gone wrong, thanks. I'll report back if unsuccessful or I'll close this ticket.

I couldn't tell you how popular CustomResourceDefinitions are or how wide spread their usage is, but they are used extensively at the company I'm with, so might be worth adding.

hnaderi · 2023-04-26T12:57:35Z

I couldn't tell you how popular CustomResourceDefinitions are or how wide spread their usage is, but they are used extensively at the company I'm with, so might be worth adding.

I will add the APIs ASAP!

hnaderi · 2023-04-26T19:40:28Z

ThisBuild / resolvers ++= Resolver.sonatypeOssRepos("snapshots")
libraryDependencies ++= Seq(
  "dev.hnaderi" %% "scala-k8s-client" % "0.11-19e12b0-SNAPSHOT"
)

This version contains CRD APIs and more cluster wide APIs as well, Your feedback is highly appreciated!
Example:

val define = APIs.crds.create(???)
val replace = APIs.crds.replace("name", ???)
// and a lot of more methods

davesmith00sky · 2023-04-27T10:38:08Z

Hi @hnaderi, many thanks for the new APIs!

We've done some testing (still going), getting a CRD works perfectly, but we're having a bit of trouble doing the creation call. We followed your suggestion of loading the manifest using manifest.parse and that works fine, but produces a KObject. Is there a nice way to convert a KObject into a CustomResourceDefinition without doing some sort of casting? (Which we tried incidentally, and it didn't work 😄)

We're going to try using a different method to load the data and convert it to a CustomResourceDefinition and see if that works in the meantime.

I wonder if manifest.parse could be something like manifest.parse[CustomResourceDefinition](data) // Either[String, CustomResourceDefinition], similar to a circe's decode function?

Separate comment: I noticed that all the error types are encoded to String, might be worth wrapping that up into an ADT of some kind to avoid accidental usage?

Thanks again!

hnaderi · 2023-04-27T10:47:22Z

I wonder if manifest.parse could be something like manifest.parseCustomResourceDefinition // Either[String, CustomResourceDefinition], similar to a circe's decode function?

Actually that's also possible, the manifest module has changed recently to use yaml4s, which you can use to decode/encode any of the objects in scala k8s, as all the codecs are available.
parse method is the general case which should be KObject.

hnaderi · 2023-04-27T11:01:26Z

I wonder if manifest.parse could be something like manifest.parseCustomResourceDefinition // Either[String, CustomResourceDefinition], similar to a circe's decode function?

That's a good idea and can be really helpful, I wonder why I didn't do it like that in the first place?! 😅
Just added the generic methods too.

davesmith00sky · 2023-04-27T13:33:18Z

Hi @hnaderi - thanks for the quick turn around. Wasn't sure if you'd published that but I've published it locally and it looks good!

However, we're now getting this error when we try and use the loaded CRD with APIs.crds.create(...):

[error] java.lang.Exception: 422 Unprocessable Entity
[error] 	at dev.hnaderi.k8s.client.Http4sKubernetesClient.send$$anonfun$1(Http4sKubernetesClient.scala:72)
[error] 	at org.http4s.client.DefaultClient.expectOr$$anonfun$1(DefaultClient.scala:103)

Any ideas? Is there a more meaningful message we could get back somehow?

hnaderi · 2023-04-27T13:59:23Z

@davesmith00sky Kubernetes responds with 422 when it can't process the your request.
What is the kubectl output with manifest now?
It might be a bad request due to your prior requests, for example k8s cannot create an existing object.
To test this scenario use kubectl create -f manifest.yml instead of apply.

davesmith00sky · 2023-04-27T14:40:58Z

Hi @hnaderi, so there was a bunch of things going on. The issue isn't resolved but I have more information.

The proxy logs don't have much to say of any interest (to my eyes anyway), this is the relevant bit:

I0427 15:33:28.311672   10302 upgradeaware.go:311] Request was not an upgrade
I0427 15:33:28.311695   10302 round_trippers.go:466] curl -v -XPOST  -H "X-Forwarded-For: 127.0.0.1" -H "User-Agent: http4s-ember/0.23.18" -H "Content-Type: application/json" -H "Content-Length: 478" -H "Accept: application/json" -H "Date: Thu, 27 Apr 2023 14:33:28 GMT" 'https://127.0.0.1:50510/apis/apiextensions.k8s.io/v1/customresourcedefinitions'
I0427 15:33:28.320772   10302 round_trippers.go:553] POST https://127.0.0.1:50510/apis/apiextensions.k8s.io/v1/customresourcedefinitions 422 Unprocessable Entity in 9 milliseconds

Is there anywhere else I can check?

I have two examples that I'm playing with and as it turns out, the 422 was masking two different errors, which I found by running kubectl create as you suggested.

The resource already exists
The name must use one of the plural forms (oops)

However, if I fix those to issues (1. remove, 2. correct the name) and run again I get another 422 status code.

Maybe I could run something like Charles proxy in front of my kubectl proxy and see what the request looks like (as-in the serialised data)? Any other suggestions?

hnaderi · 2023-04-27T14:53:27Z

@davesmith00sky I think it's better to get it working on the kubectl first, as it looks like a manifest problem.

Also to see the serialized data you can simply re-encode your object, to yaml or json.

obj.encodeTo[Json]

davesmith00sky · 2023-04-27T14:59:18Z

Oh sorry, it does work with kubectl, I should have said that. I'll take a look at the encoded version. Thanks.

hnaderi · 2023-04-27T15:37:39Z

Also to check if there is a problem with serialization, it would be helpful to use kubectl on the re-encoded manifest to get a fast feedback.

davesmith00sky · 2023-04-27T15:40:15Z

Charles proxy to the rescue! (my app <-> Charles proxy as reverse proxy <-> kubectl proxy <-> k8s api)

{
	"kind": "Status",
	"apiVersion": "v1",
	"metadata": {},
	"status": "Failure",
	"message": "CustomResourceDefinition.apiextensions.k8s.io \"xxx\" is invalid: [spec.names.kind: Required value, spec.names.listKind: Required value]",
	"reason": "Invalid",
	"details": {
		"name": "xxx",
		"group": "apiextensions.k8s.io",
		"kind": "CustomResourceDefinition",
		"causes": [{
			"reason": "FieldValueRequired",
			"message": "Required value",
			"field": "spec.names.kind"
		}, {
			"reason": "FieldValueRequired",
			"message": "Required value",
			"field": "spec.names.listKind"
		}]
	},
	"code": 422
}

...and sure enough when I looked at the obj.encodeTo[Json] encoded version, the spec.names.kind field was missing / had been dropped.

I hadn't added spec.names.listKind when I ran that, but a) if I add it, it does actually work as expected and appears in the encoded version, so that's ok, but b) it seems to be optional / has a default of kind + "List"? If I'm right about that then I'm not sure why that should be an error - but it's the kubernetes API returning the error so maybe it's right, or maybe kubectl massages the shape before it sends it - not sure.

hnaderi · 2023-04-27T15:49:55Z

I think the kubectl updating the manifest on the fly is more likely, can you check that by the Charles proxy as well?

davesmith00sky · 2023-04-27T16:18:00Z

I think the kubectl updating the manifest on the fly is more likely, can you check that by the Charles proxy as well?

I.... don't think so. 😅 I've had a bit of a look around and I'm not sure how I'd set that up.

davesmith00sky · 2023-05-02T14:15:26Z

Hi @hnaderi - I'm back from a long weekend, hope you're well. 😄

So I've done some more digging and the culprit is here!

final case class CustomResourceDefinitionNames(
  plural : String,
  singular : Option[String] = None,
  listKind : Option[String] = None,
  categories : Option[Seq[String]] = None,
  shortNames : Option[Seq[String]] = None
)

Note that this is missing the kind field. What's interesting is that I followed your plugin code and that led me to the following understanding:

CustomResourceDefinitionNames is a generate class
The KubernetesSpecPlugin generates it by reading the k8s spec based on the version supplied in the build.sbt file.
Having downloaded the spec as json, it parses and writes the details into src_managed.

So far so good. (I like how you defined the plugin to do that, btw - nice!)

The trouble is that it isn't working correctly. If you go to the spec, you can clearly see the kind field defined, and also declared as required in the JSON schema definition - but for some reason, this is being ignored.

Here's the spec, search for CustomResourceDefinitionNames.
https://raw.githubusercontent.com/kubernetes/kubernetes/v1.27.1/api/openapi-spec/swagger.json

...and you get this:

    "io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.CustomResourceDefinitionNames": {
      "description": "CustomResourceDefinitionNames indicates the names to serve this CustomResourceDefinition",
      "properties": {
        "categories": {
          "description": "categories is a list of grouped resources this custom resource belongs to (e.g. 'all'). This is published in API discovery documents, and used by clients to support invocations like `kubectl get all`.",
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        "kind": {
          "description": "kind is the serialized kind of the resource. It is normally CamelCase and singular. Custom resource instances will use this value as the `kind` attribute in API calls.",
          "type": "string"
        },
        "listKind": {
          "description": "listKind is the serialized kind of the list for this resource. Defaults to \"`kind`List\".",
          "type": "string"
        },
        "plural": {
          "description": "plural is the plural name of the resource to serve. The custom resources are served under `/apis/<group>/<version>/.../<plural>`. Must match the name of the CustomResourceDefinition (in the form `<names.plural>.<group>`). Must be all lowercase.",
          "type": "string"
        },
        "shortNames": {
          "description": "shortNames are short names for the resource, exposed in API discovery documents, and used by clients to support invocations like `kubectl get <shortname>`. It must be all lowercase.",
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        "singular": {
          "description": "singular is the singular name of the resource. It must be all lowercase. Defaults to lowercased `kind`.",
          "type": "string"
        }
      },
      "required": [
        "plural",
        "kind"
      ],
      "type": "object"
    }

Note:

        "kind": {
          "description": "kind is the serialized kind of the resource. It is normally CamelCase and singular. Custom resource instances will use this value as the `kind` attribute in API calls.",
          "type": "string"
        },

and

      "required": [
        "plural",
        "kind"
      ],

What do you think is going on?

hnaderi · 2023-05-02T14:44:24Z

@davesmith00sky Welcome back! 😊
Wow! You really have dug deeply. Thank you for the effort 🙏
If I remember correctly, kind fields are handled by an exception to the main rule; which seems to be incorrect for this one.
I'll open a separate issue and will try to resolve it ASAP!

davesmith00sky · 2023-05-02T14:45:42Z

Thank you kindly!

hnaderi · 2023-05-02T19:07:23Z

#101 (comment)

hnaderi · 2023-05-03T08:43:13Z

I close this issue as it is resolved in other issues.
Feel free to open another issue or discussion, Your feedback is highly appreciated!

hnaderi mentioned this issue Apr 26, 2023

APIs for CustomResourceDefinition #97

Closed

This comment was marked as resolved.

Sign in to view

hnaderi mentioned this issue May 2, 2023

Code generation misses kind field in the CustomResourceDefinitionNames #101

Closed

hnaderi added bug Something isn't working enhancement New feature or request question Further information is requested labels May 2, 2023

hnaderi added this to the V0.11 milestone May 3, 2023

hnaderi closed this as completed May 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Example of how to create a CRD #96

Example of how to create a CRD #96

davesmith00sky commented Apr 26, 2023

github-actions bot commented Apr 26, 2023

hnaderi commented Apr 26, 2023

davesmith00sky commented Apr 26, 2023

hnaderi commented Apr 26, 2023

hnaderi commented Apr 26, 2023

davesmith00sky commented Apr 27, 2023

hnaderi commented Apr 27, 2023

hnaderi commented Apr 27, 2023

davesmith00sky commented Apr 27, 2023

This comment was marked as resolved.

hnaderi commented Apr 27, 2023 •

edited

Loading

davesmith00sky commented Apr 27, 2023 •

edited

Loading

hnaderi commented Apr 27, 2023

davesmith00sky commented Apr 27, 2023

hnaderi commented Apr 27, 2023

davesmith00sky commented Apr 27, 2023 •

edited

Loading

hnaderi commented Apr 27, 2023

davesmith00sky commented Apr 27, 2023

davesmith00sky commented May 2, 2023

hnaderi commented May 2, 2023

davesmith00sky commented May 2, 2023

hnaderi commented May 2, 2023

hnaderi commented May 3, 2023

Example of how to create a CRD #96

Example of how to create a CRD #96

Comments

davesmith00sky commented Apr 26, 2023

github-actions bot commented Apr 26, 2023

hnaderi commented Apr 26, 2023

davesmith00sky commented Apr 26, 2023

hnaderi commented Apr 26, 2023

hnaderi commented Apr 26, 2023

davesmith00sky commented Apr 27, 2023

hnaderi commented Apr 27, 2023

hnaderi commented Apr 27, 2023

davesmith00sky commented Apr 27, 2023

This comment was marked as resolved.

hnaderi commented Apr 27, 2023 • edited Loading

davesmith00sky commented Apr 27, 2023 • edited Loading

hnaderi commented Apr 27, 2023

davesmith00sky commented Apr 27, 2023

hnaderi commented Apr 27, 2023

davesmith00sky commented Apr 27, 2023 • edited Loading

hnaderi commented Apr 27, 2023

davesmith00sky commented Apr 27, 2023

davesmith00sky commented May 2, 2023

hnaderi commented May 2, 2023

davesmith00sky commented May 2, 2023

hnaderi commented May 2, 2023

hnaderi commented May 3, 2023

hnaderi commented Apr 27, 2023 •

edited

Loading

davesmith00sky commented Apr 27, 2023 •

edited

Loading

davesmith00sky commented Apr 27, 2023 •

edited

Loading