Creating CRDs with schema validation is broken #1486
Comments
|
@Jiri-Kremser : Hi, Could you please elaborate on your use case? What are you trying to do here? We're right now trying to simplify our Custom Resources support. We've introduced a new endpoint in our dsl which handles raw custom resources in form of hashmaps: |
|
Cool, this looks little bit easier than the
sure, I want to create CRDs that have the OpenAPI Schema inside and make the validation for CRs working (https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#validation). The issue is (or at least with the previous version, where ...
builder.withNewOpenAPIV3SchemaLike(item)
...
builder.build()The resulting CRD had the I am currently using this hot-fix crdToReturn.getSpec().getValidation().getOpenAPIV3Schema().setDependencies(null);this works, but it doesn't seem right. |
|
No no, this is an example of improved support for custom resources. For custom resource definitions it's still the same: |
|
here is the code if you need a bigger picture https://github.com/jvm-operators/abstract-operator/blob/master/src/main/java/io/radanalytics/operator/common/crd/CrdDeployer.java#L69 |
|
Hmm, will try to reproduce your problem .... |
|
here is the reproducer: public class Reproducer {
public static JSONSchemaProps readSchema() {
ObjectMapper mapper = new ObjectMapper();
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
URL in = Reproducer.class.getResource("/sparkCluster.json");
if (null == in) {
return null;
}
try {
return mapper.readValue(in, JSONSchemaProps.class);
} catch (IOException e) {
e.printStackTrace();
return null;
}
}
public static void main(String[] args) {
JSONSchemaProps schema = readSchema();
CustomResourceDefinitionBuilder builder = new CustomResourceDefinitionBuilder()
.withApiVersion("apiextensions.k8s.io/v1beta1")
.withNewMetadata().withName("sparkclusters.radanalytics.io")
.endMetadata()
.withNewSpec()
.withNewNames()
.withKind("SparkCluster")
.endNames()
.withGroup("radanalytics.io")
.withVersion("v1")
.withScope("Namespaced")
.withNewValidation()
.withNewOpenAPIV3SchemaLike(schema)
.endOpenAPIV3Schema()
.endValidation()
.endSpec();
new DefaultKubernetesClient().customResourceDefinitions().createOrReplace(builder.build());
}
}content of {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "A Spark cluster configuration",
"dependencies": null,
"type": "object",
"extends": {
"type": "object",
"existingJavaType": "io.radanalytics.operator.common.EntityInfo"
},
"properties": {
"master": {
"type": "object",
"properties": {
"instances": {
"type": "integer",
"default": "1",
"minimum": "1"
},
"memory": {
"type": "string"
},
"cpu": {
"type": "string"
},
"labels": {
"existingJavaType": "java.util.Map<String,String>",
"type": "string",
"pattern": "([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]"
},
"command": {
"type": "array",
"items": {
"type": "string"
}
},
"commandArgs": {
"type": "array",
"items": {
"type": "string"
}
}
}
},
"worker": {
"type": "object",
"properties": {
"instances": {
"type": "integer",
"default": "1",
"minimum": "0"
},
"memory": {
"type": "string"
},
"cpu": {
"type": "string"
},
"labels": {
"existingJavaType": "java.util.Map<String,String>",
"type": "string",
"pattern": "([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]"
},
"command": {
"type": "array",
"items": {
"type": "string"
}
},
"commandArgs": {
"type": "array",
"items": {
"type": "string"
}
}
}
},
"customImage": {
"type": "string"
},
"metrics": {
"type": "boolean",
"default": "false"
},
"sparkWebUI": {
"type": "boolean",
"default": "true"
},
"sparkConfigurationMap": {
"type": "string"
},
"env": {
"type": "array",
"items": {
"type": "object",
"javaType": "io.radanalytics.types.Env",
"properties": {
"name": { "type": "string" },
"value": { "type": "string" }
},
"required": ["name", "value"]
}
},
"sparkConfiguration": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": { "type": "string" },
"value": { "type": "string" }
},
"required": ["name", "value"]
}
},
"labels": {
"type": "object",
"existingJavaType": "java.util.Map<String,String>"
},
"historyServer": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"type": {
"type": "string",
"default": "sharedVolume",
"enum": [
"sharedVolume",
"remoteStorage"
],
"javaEnumNames": [
"sharedVolume",
"remoteStorage"
]
},
"sharedVolume": {
"type": "object",
"properties": {
"size": {
"type": "string",
"default": "0.3Gi"
},
"mountPath": {
"type": "string",
"default": "/history/spark-events"
},
"matchLabels": {
"type": "object",
"existingJavaType": "java.util.Map<String,String>"
}
}
},
"remoteURI": {
"type": "string",
"description": "s3 bucket or hdfs path"
}
}
},
"downloadData": {
"type": "array",
"items": {
"type": "object",
"properties": {
"url": {
"type": "string"
},
"to": {
"type": "string"
}
},
"required": [
"url",
"to"
]
}
}
},
"required": []
}..but it would fail with any schema, even smaller one. Thanks for looking into this! |
|
Hmm, I tried reproducing your issue. When I run this example, I get this error: Surprisingly, when I load it from a yaml containing validation schema, creation works 😕 : Need to check how can we change this behavior of object mapper |
|
yes, ^ that's that same error I was getting |
|
Any progress here?
btw. I hit another similar issue, where I had to change the behavior of the default object mapper, and this helped: Serialization.jsonMapper().configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false); |
|
i ran into the same issue :( |
|
Same issue here :( public static final CustomResourceDefinition CR_DEFINITION =
new CustomResourceDefinitionBuilder()
.withApiVersion("apiextensions.k8s.io/v1beta1")
.withNewMetadata()
.withName(NAME)
.endMetadata()
.withNewSpec()
.withGroup(GROUP)
.withVersion(VERSION)
.withScope("Namespaced")
.withNewNames()
.withKind(KIND)
.withListKind(KIND + "List")
.withSingular(SINGULAR)
.withPlural(PLURAL)
.endNames()
.withValidation(getSchemaValidation())
.endSpec()
.build();
private static CustomResourceValidation getSchemaValidation() {
Map<String, JSONSchemaProps> properties = new HashMap<>();
properties.putIfAbsent("version", new JSONSchemaPropsBuilder()
.withType("integer")
.withMinimum(11d)
.build());
properties.putIfAbsent("conf", new JSONSchemaPropsBuilder()
.withType("object")
.build());
Map<String, JSONSchemaProps> spec = new HashMap<>();
spec.putIfAbsent("spec", new JSONSchemaPropsBuilder()
.withRequired(properties.keySet().stream().collect(Collectors.toList()))
.withProperties(properties)
.build());
return new CustomResourceValidationBuilder()
.withOpenAPIV3Schema(new JSONSchemaPropsBuilder()
.withRequired("spec")
.withProperties(spec)
.build())
.build();
}This creates the following yaml: And creating the CRD in K8s gives: |
|
I remember I looked into this few months ago... Maybe there was some bug on model side itself or some problem after deserialization. Need to revisit this again after Devconf |
|
I am applying @jkremser 's workaround currently and i am seeing a very strange issue: the CRD I tried running my code with Docker's builtin Kubernetes (Version: according to okhttp logs the client queries for the existing resource twice and then proceeds to Is it possible that this might be the reason? Stepping through the code shows that at the time of sending the replace request to k8s api the updated CRD has a dependencies field again. edit: this also happens on 1.15+ clusters consistently |
|
@rohanKanojia are you still aware of this? can't really find a workaround for this issue on new k8s versions anymore |
|
ah, Yes. I forgot to take a look at this again. |
|
I don't think this is coming from jackson, I think
Whereas after CustomResourceDefinition is created from the builder, the resulting object becomes like this, But when I skipped setting schema from the builder, and explicitly set it, I was able to create CRD like this: Another thing that I observed is that when I try to load the custom resource from yaml and then apply; it gets applied properly. I took Jimmi's crd spec and converted it to yaml: sparkclusters-radanalytics-crd.yml: Code which I used for this: |
The error i described still happens with 4.6.1. It is pretty simple to reproduce, create a CRD using an openAPI validation schema, set dependencies to null like described in this thread and run |
|
Have you tried creating CRD with the workaround I specified? |
Yes, it shows the same behavior. Here's a quick and dirty reproducer: ObjectMapper mapper = new ObjectMapper();
final URL resource = CreateFailIT.class.getResource("/schema/test.json");
final JSONSchemaProps jsonSchemaProps = mapper.readValue(resource, JSONSchemaProps.class);
final CustomResourceDefinitionFluent.SpecNested<CustomResourceDefinitionBuilder> crdBuilder = new CustomResourceDefinitionBuilder()
.withApiVersion("apiextensions.k8s.io/v1beta1")
.withNewMetadata()
.withName("somethings.example.com")
.endMetadata()
.withNewSpec()
.withNewNames()
.withKind("Example")
.withPlural("somethings")
.withShortNames(Arrays.asList("ex")).endNames()
.withGroup("example.com")
.withVersion("v1")
.withScope("Namespaced");
final CustomResourceDefinition customResourceDefinition = crdBuilder
.withNewValidation()
.endValidation()
.endSpec()
.build();
// Original workaround, same behavior
//customResourceDefinition.getSpec().getValidation().getOpenAPIV3Schema().setDependencies(null);
// just to make sure dependencies field is really null
jsonSchemaProps.setDependencies(null);
customResourceDefinition.getSpec().getValidation().setOpenAPIV3Schema(jsonSchemaProps);
// To make this test idempotent
client.customResourceDefinitions().delete(customResourceDefinition);
do {
System.out.println("Waiting until CRD is gone from K8s");
Thread.sleep(500);
} while (client.customResourceDefinitions().list()
.getItems()
.stream()
.anyMatch(crd -> crd.getMetadata().getName().contains("somethings")));
client.customResourceDefinitions().createOrReplace(customResourceDefinition);
// This will always fail
client.customResourceDefinitions().createOrReplace(customResourceDefinition);test.json: {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Test configuration",
"dependencies": null,
"type": "object",
"properties": {
"spec": {
"type": "object",
"properties": {
"replicas": {
"type": "integer",
"minimum": 1
}
}
}
},
"required": ["spec"]
} |
|
@sbaier1 : You're right. I'm able to reproduce this during Looks like I need to investigate why builder is converting null values to empty LinkedHashMaps. Maybe we need to look into sundrio for this. |
+ Upgrade to sundrio v0.19.2 + Change sundrio configuration to disable lazy collection initialization
+ Upgrade to sundrio v0.19.2 + Change sundrio configuration to disable lazy collection initialization
+ Upgrade to sundrio v0.19.2 + Change sundrio configuration to disable lazy collection initialization
+ Upgrade to sundrio v0.19.2 + Change sundrio configuration to disable lazy collection initialization
+ Upgrade to sundrio v0.19.2 + Change sundrio configuration to disable lazy collection initialization
+ Upgrade to sundrio v0.19.2 + Change sundrio configuration to disable lazy collection initialization
the K8s api server makes a lot of strict checks, for instance it fails if the schema contains default values, definitions, $ref elements, etc. One of the checks is:
source: https://github.com/kubernetes/apiextensions-apiserver/blob/master/pkg/apis/apiextensions/validation/validation.go#L671
However, even if the
JSONSchemaPropsobject has thedependeciesfield set tonull, the object mapper from Jackson that's being used in here converts thenullto an emptyLinkedHashMapand this fails on the K8s api server. So there is no way currently to create the schema validation with the fabric8 k8s client.I am using the
.withNewOpenAPIV3SchemaLike(schema)method for theCustomResourceDefinitionBuilder.I believe there must be some configuration of the object mapper so that it shouldn't translate nulls to empty maps.
..looks promising
The text was updated successfully, but these errors were encountered: