Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Schema Generation for nested yml configurations #1027

Merged
merged 102 commits into from Jan 10, 2020
Merged
Show file tree
Hide file tree
Changes from 42 commits
Commits
Show all changes
102 commits
Select commit Hold shift + click to select a range
774506b
Added initial schema test
Aug 7, 2019
7051ca9
Added Schema Validation functions
Aug 8, 2019
f1b2d30
Added test jsonSchemGenerator
Aug 10, 2019
17079c7
Added test to generate an intermediate schema
Aug 16, 2019
886bc5a
Added schema generation for a single descriptor
Aug 17, 2019
0271cd3
Completed the string generation 2/4 of the schema
Aug 21, 2019
522f4e5
Added exclusion
Aug 22, 2019
f264c70
generic get configurators
jetersen Aug 22, 2019
076fe14
Finished Descriptor Configuration generation
Aug 22, 2019
2bd7f9d
Base configurator and HetroDescribable Configurator Schema Generation…
Aug 22, 2019
0da381b
Added Base configurators with no attributes
Aug 23, 2019
3d45f6b
Added Base configurator attribute enumeration schema
Aug 23, 2019
cd9a9dd
Added Non-enumerated attributes to schema
Aug 23, 2019
4bf306c
Added fully validated Configured schema
Aug 23, 2019
4657e0c
Added validation initial test
Aug 24, 2019
c734c0f
Using JSON objects to construct Schema
Aug 24, 2019
fe1ce60
Added valid schema integration tests
Aug 25, 2019
8108038
Converted class names to simple names
Aug 26, 2019
38dad38
Upgraded schema version
Aug 26, 2019
d89e24c
Changes to test and pom.xml
Aug 26, 2019
3c4510c
Added functions to split up the generateSchema() function
Aug 26, 2019
f079505
Added additional methods in generateSchema()
Aug 26, 2019
76c0932
Code Cleanup
Aug 26, 2019
623914f
Inital impl for describeStructure
Aug 27, 2019
13e893a
Incremental Impl changes
Aug 27, 2019
2ede6f6
Concept implementation
Aug 27, 2019
ec07960
Added annotation and test implementations
Aug 28, 2019
448c4a2
Added addition configurator wise tests
Aug 28, 2019
3165323
Added the utlisation of describeStructure
Aug 28, 2019
ad57f8d
Added default configurators describe structure
Aug 29, 2019
14197bc
Added describe Strucure function to default Config
Aug 29, 2019
a338fbc
Added JSON Schema mode and boolean
Aug 30, 2019
d0f9349
Added describeStructure to Attribute and Configurator
Aug 31, 2019
6c4cb3c
Merge branch 'master' into nested_yaml_schema
Sep 4, 2019
901ac01
Lookup addition for configurators
Sep 6, 2019
4df5e1e
Added tests and added baseConfigurator LookupFeature
Sep 6, 2019
81b9a68
Merge master into current branch
Sep 12, 2019
fd886c8
Added test for rootConfiguratorGeneration
Sep 14, 2019
6c42055
Sequence indication fix
Sep 20, 2019
b0b870b
Configurator Lookup Addition
Sep 20, 2019
fe7b0b8
Added javadoc and logger warning for failing tests
Sep 24, 2019
dcd5f6f
Added storeConfigurators function
Sep 25, 2019
122c8d9
Added root configurator nesting
Sep 25, 2019
e5bde57
Fixed root configurators
Sep 26, 2019
b97f380
Merge branch 'master' into nested_yaml_schema
Sep 26, 2019
1e0a6a2
Added support for the root configurators
Sep 27, 2019
d0083d8
Added mock test
Sep 27, 2019
a030035
Merge branch 'master' into nested_yaml_schema
Sep 27, 2019
2b105a4
Added nesting support for every base configurator and tests for the same
Sep 28, 2019
79892a2
Removed functions that are not used
Sep 28, 2019
51ae5fd
Added jenkins base configurator test
Sep 30, 2019
cee3cf6
Added validSelfConfig Test
Sep 30, 2019
1f72339
Added validSelfConfig and validJenkinsBaseConfig files
Oct 2, 2019
67c3f81
Don't fetch the jenkins.model.Jenkins
timja Oct 2, 2019
dff9199
Resolved merge conflicts and reverted tim's changes
Oct 4, 2019
5388f66
Resolved conflicts
Nov 15, 2019
e992aee
WIP
timja Dec 2, 2019
0e9e3bd
Merge branch 'master' into nested_yaml_schema
timja Dec 2, 2019
5bacdd2
Fix self configurator
timja Dec 2, 2019
b07c22b
Fix formatting
timja Dec 2, 2019
9bc089a
More formatting
timja Dec 2, 2019
94e14d4
Undo whitespace changes
timja Dec 2, 2019
a96a4f7
Change json schema test API to return list of errors
timja Dec 3, 2019
802c9c9
Merge branch 'change-json-schema-test-api' into nested_yaml_schema
timja Dec 3, 2019
d50ad6e
Adding failing test demonstrating all attributes getting flattenned to
timja Dec 3, 2019
26341c4
Fix flattening issues
timja Dec 3, 2019
3ce5e2b
Remove print lines
timja Dec 3, 2019
b92032d
Fix spot bugs
timja Dec 3, 2019
5e9ce85
Checkstyle
timja Dec 3, 2019
a71b71e
Fix test
timja Dec 3, 2019
ce565f2
Fix checkstyle
timja Dec 3, 2019
cd01391
Introduce test-harness module
timja Dec 7, 2019
e50c94f
shush codacy + docs
timja Dec 7, 2019
55e0216
Fix javadoc
timja Dec 7, 2019
80221ea
Fix import
timja Dec 7, 2019
2ef14a3
Merge branch 'change-json-schema-test-api' into nested_yaml_schema
timja Dec 8, 2019
53342bf
Add integration tests for some of the remaining issues
timja Dec 8, 2019
2403838
Fix enforcer
timja Dec 8, 2019
0395bea
Comment out debug
timja Dec 8, 2019
30fdbf8
Fix tests
timja Dec 8, 2019
912b46d
Add symbol support and fix tests
timja Dec 8, 2019
603a577
Add for symbol support
timja Dec 9, 2019
d271135
Checkstyle
timja Dec 9, 2019
896d2cb
Add long to case
timja Dec 9, 2019
ee1c946
Add one level of array support
timja Dec 9, 2019
6ab2a31
remove unused imports
jetersen Dec 10, 2019
3e32f25
Merge branch 'master' into nested_yaml_schema
timja Dec 10, 2019
3be3926
Revert unneeded change
timja Dec 10, 2019
e7b2a8f
Fix earlier merge, restore help file support
timja Dec 10, 2019
314f1a7
Fix checkstyle
timja Dec 10, 2019
b646a04
Added system log and removed exception
Dec 14, 2019
423ee6c
Improved warning string
Dec 14, 2019
d5d10f4
Added describeForSchema Function
Jan 5, 2020
1cdd881
Fixed Checkstyle
Jan 5, 2020
1040a6a
Removed boolean
Jan 7, 2020
a242048
Fixed ClassCast Exception
Jan 7, 2020
0cfde4a
Combined declaration and assignment
Jan 8, 2020
1aa84b1
Added javadoc and beta class
Jan 9, 2020
1ff2aaf
Update plugin/src/main/java/io/jenkins/plugins/casc/Configurator.java
sladyn98 Jan 10, 2020
15a7500
Update plugin/src/main/java/io/jenkins/plugins/casc/Attribute.java
sladyn98 Jan 10, 2020
b468ee9
Update plugin/src/main/java/io/jenkins/plugins/casc/Configurator.java
sladyn98 Jan 10, 2020
772445a
Update plugin/src/main/java/io/jenkins/plugins/casc/Attribute.java
sladyn98 Jan 10, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
82 changes: 76 additions & 6 deletions plugin/src/main/java/io/jenkins/plugins/casc/Attribute.java
Expand Up @@ -61,6 +61,7 @@ public class Attribute<Owner, Type> {
private Setter<Owner, Type> setter;
private Getter<Owner, Type> getter;
private boolean secret;
private boolean isJsonSchema;

private boolean deprecated;

Expand Down Expand Up @@ -111,6 +112,14 @@ public Class<? extends AccessRestriction>[] getRestrictions() {
return restrictions != null ? restrictions : EMPTY;
}

/**
* Set jsonSchema is used to tell the describe function to call the describe structure
* so that it supports and returns a nested structure
*/
public void setJsonSchema(boolean jsonSchema) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add Javadoc?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added +1

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this still required?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for closing now it is the separate thread

isJsonSchema = jsonSchema;
}

public boolean isRestricted() {
return restrictions != null && restrictions.length > 0;
}
Expand Down Expand Up @@ -225,18 +234,29 @@ public CNode describe(Owner instance, ConfigurationContext context) throws Confi
": No configurator found for type " + type);
}
try {
Object o = getValue(instance);
if (o == null) {
return null;
Object o;
if(isJsonSchema) {
o = getType();
sladyn98 marked this conversation as resolved.
Show resolved Hide resolved
if (o == null) {
return null;
}
} else {
o = getValue(instance);
if (o == null) {
return null;
}
}

// In Export we sensitive only those values which do not get rendered as secrets
boolean shouldBeMasked = isSecret(instance);
if (multiple) {
Sequence seq = new Sequence();
if (o.getClass().isArray()) o = Arrays.asList((Object[]) o);
for (Object value : (Iterable) o) {
seq.add(_describe(c, context, value, shouldBeMasked));
if(o instanceof Iterable) {
System.out.println("This is iterable");
for (Object value : (Iterable) o) {
seq.add(_describe(c, context, value, shouldBeMasked));
}
}
return seq;
}
Expand All @@ -249,6 +269,51 @@ public CNode describe(Owner instance, ConfigurationContext context) throws Confi
}
}


/**
* This function is for the JSONSchemaGeneration
* @param instance
* @param context
* @return
*/
timja marked this conversation as resolved.
Show resolved Hide resolved
public CNode describeStructure(Owner instance, ConfigurationContext context) {

sladyn98 marked this conversation as resolved.
Show resolved Hide resolved
final Configurator c = context.lookup(type);
if (c == null) {
return new Scalar("FAILED TO EXPORT\n" + instance.getClass().getName()+"#"+name +
": No configurator found for type " + type);
}
try {
Object o = new Object();
if(isJsonSchema) {
sladyn98 marked this conversation as resolved.
Show resolved Hide resolved
o = getType();
if (o == null) {
return null;
}
}

// In Export we sensitive only those values which do not get rendered as secrets
boolean shouldBeMasked = isSecret(instance);
if (multiple) {
Sequence seq = new Sequence();
if (o.getClass().isArray()) o = Arrays.asList((Object[]) o);
if(o instanceof Iterable) {
sladyn98 marked this conversation as resolved.
Show resolved Hide resolved
sladyn98 marked this conversation as resolved.
Show resolved Hide resolved
for (Object value : (Iterable) o) {
seq.add(_describe(c, context, value, shouldBeMasked));
}
}
return seq;
}
return _describe(c, context, o, shouldBeMasked);
} catch (Exception | /* Jenkins.getDescriptorOrDie */AssertionError e) {
sladyn98 marked this conversation as resolved.
Show resolved Hide resolved
// Don't fail the whole export, prefer logging this error
LOGGER.log(Level.WARNING, "Failed to export", e);
return new Scalar("FAILED TO EXPORT\n" + instance.getClass().getName() + "#" + name + ": "
+ printThrowable(e));
}
}


/**
* Describes a node.
* @param c Configurator
Expand All @@ -261,7 +326,12 @@ public CNode describe(Owner instance, ConfigurationContext context) throws Confi
*/
private CNode _describe(Configurator c, ConfigurationContext context, Object value, boolean shouldBeMasked)
throws Exception {
CNode node = c.describe(value, context);
CNode node;
if(isJsonSchema) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Casz Added this so that it would not throw any casting errors, since in the descirbe structure we were initally calling with value whereas now we needed the type. This reduces the castClassException

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this still required?

sladyn98 marked this conversation as resolved.
Show resolved Hide resolved
node = c.describeStructure(value, context);
} else {
node = c.describe(value, context);
}
if (shouldBeMasked && node instanceof Scalar) {
((Scalar)node).sensitive(true);
}
Expand Down
Expand Up @@ -30,6 +30,8 @@ public class ConfigurationContext implements ConfiguratorRegistry {

private transient final ConfiguratorRegistry registry;

private String mode;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's this mode value for? needs docs and probably should be an enum if required

Copy link
Contributor Author

@sladyn98 sladyn98 Sep 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its just a mode to tell if the context is jsonSchema and if it is then run the appropriate function

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be an enum or a boolean really

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah will incorporate it in the next set of changes.


public ConfigurationContext(ConfiguratorRegistry registry) {
this.registry = registry;
}
Expand Down Expand Up @@ -62,6 +64,15 @@ public void setUnknown(Unknown unknown) {
this.unknown = unknown;
}

public String getMode() {
return mode;
}

public void setMode(String mode) {
this.mode = mode;
}



// --- delegate methods for ConfigurationContext

Expand Down
17 changes: 17 additions & 0 deletions plugin/src/main/java/io/jenkins/plugins/casc/Configurator.java
Expand Up @@ -34,6 +34,7 @@
import org.apache.commons.lang.StringUtils;
import org.jenkinsci.Symbol;


/**
* Define a {@link Configurator} which handles a configuration element, identified by name.
* @author <a href="mailto:nicolas.deloof@gmail.com">Nicolas De Loof</a>
Expand Down Expand Up @@ -167,4 +168,20 @@ default CNode describe(T instance, ConfigurationContext context) throws Exceptio
return mapping;
}

@CheckForNull
Copy link
Member

@oleg-nenashev oleg-nenashev Jan 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs Javadoc since it will be consumed by plugins. @since TODO also

Copy link
Contributor Author

@sladyn98 sladyn98 Jan 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah added javadoc. Confused about the @since TODO ?

default CNode describeStructure(T instance, ConfigurationContext context)
timja marked this conversation as resolved.
Show resolved Hide resolved
throws Exception {
Mapping mapping = new Mapping();
for (Attribute attribute : getAttributes()) {
if(context.getMode().equals("JSONSchema")) {
sladyn98 marked this conversation as resolved.
Show resolved Hide resolved
attribute.setJsonSchema(true);
}
CNode value = attribute.describeStructure(instance, context);
if (value != null) {
mapping.put(attribute.getName(), attribute.getType().getSimpleName());
}
}
return mapping;
}

}
Expand Up @@ -4,7 +4,11 @@
import com.google.gson.GsonBuilder;
import com.google.gson.JsonElement;
import com.google.gson.JsonParser;
import io.jenkins.plugins.casc.impl.DefaultConfiguratorRegistry;
import io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator;
import io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator;
import io.jenkins.plugins.casc.model.CNode;
import io.jenkins.plugins.casc.model.Mapping;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.LinkedHashSet;
Expand Down Expand Up @@ -36,14 +40,15 @@ public static JSONObject generateSchema() {
* Iterates over the base configurators and adds them to the schema.
*/
JSONObject schemaConfiguratorObjects = new JSONObject();
ConfigurationAsCode configurationAsCode = ConfigurationAsCode.get();
for (Object configuratorObject : configurationAsCode.getConfigurators()) {
ConfigurationAsCode configurationAsCodeObject = ConfigurationAsCode.get();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bad merge I think, revert?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I will just get this to its original state.

for (Object configuratorObject : configurationAsCodeObject.getConfigurators()) {
if (configuratorObject instanceof BaseConfigurator) {
BaseConfigurator baseConfigurator = (BaseConfigurator) configuratorObject;
List<Attribute> baseConfigAttributeList = baseConfigurator.getAttributes();

if (baseConfigAttributeList.size() == 0) {
schemaConfiguratorObjects
.put(((BaseConfigurator) configuratorObject).getTarget().getSimpleName().toLowerCase(),
.put(baseConfigurator.getName().toLowerCase(),
new JSONObject()
.put("type", "object")
.put("properties", new JSONObject()));
Expand Down Expand Up @@ -132,6 +137,10 @@ private static JSONObject generateNonEnumAttributeObject(Attribute attribute) {
attributeType.put("type", "integer");
break;

case "hudson.Secret":
attributeType.put("type", "string");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice to add a comment to the schema to indicate that this field is sensitive, but it is rather a follow-up enhancement

break;

case "java.lang.Long":
timja marked this conversation as resolved.
Show resolved Hide resolved
attributeType.put("type", "integer");
break;
Expand All @@ -154,6 +163,7 @@ private static JSONObject generateRootConfiguratorObject() {
RootElementConfigurator rootElementConfigurator = i.next();
rootConfiguratorObject
.put(rootElementConfigurator.getName(), new JSONObject().put("type", "object"));
System.out.println("root Name" + rootElementConfigurator.getName());
}
return rootConfiguratorObject;
}
Expand Down Expand Up @@ -187,5 +197,50 @@ private static void generateEnumAttributeSchema(JSONObject attributeSchemaTempla
.put("enum", new JSONArray(attributeList)));
}
}

public static void rootConfigGeneration() throws Exception {
DefaultConfiguratorRegistry registry = new DefaultConfiguratorRegistry();
final ConfigurationContext context = new ConfigurationContext(registry);
context.setMode("JSONSchema");
for (RootElementConfigurator root : RootElementConfigurator.all()) {
final CNode config = root.describeStructure(root.getTargetComponent(context), context);
final Mapping mapping = config.asMapping();
final List<Map.Entry<String, CNode>> entries = new ArrayList<>(mapping.entrySet());
Copy link
Contributor Author

@sladyn98 sladyn98 Aug 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Casz globalNodeProperties NodeProperty The mapping is as follows, so we are not given back the entire structure as such, whereas what we need is an iterable NodeProperty which at the moment is not possible.

      globalNodeProperties:

    - envVars:

        env:

          - key: FOO

            value: BAR

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@timja Any ideas on How We can an Iterable Node Property ? Because we are returned a CNode which we cannot iterate on.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a failing test or point to one I should run please?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this what you're after?

                CNode value = entry.getValue();
                if (value.getType() == Type.SEQUENCE) {
                    Sequence cNodes = value.asSequence();
                    cNodes.forEach(node -> { 
                        
                    });
                }

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really because we wont enter this if statement at all.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You’re going to need to indicate that it’s a sequence some how, where I pointed is where the information is lost

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean that we should be iterating over the attribute type and keep adding it to the mapping something like this
if (attribute.isMultiple == true) {
Iterate over the sequence and add it to the mapping
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no but you need to indicate it's a sequence?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indicate what as a sequence ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we indicate the mapping as a sequence would it make it easier to iterate further?

for (Map.Entry<String, CNode> entry : entries) {
System.out.println(entry.getKey());
}
System.out.println("End of an iteration of configurators");
}
}

public static void storeConfiguratorNames() {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@timja I am not sure this is necessary anymore

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove if not needed

ConfigurationAsCode configurationAsCodeObject = ConfigurationAsCode.get();
for (Object configuratorObject : configurationAsCodeObject.getConfigurators()) {
if (configuratorObject instanceof BaseConfigurator) {
BaseConfigurator baseConfigurator = (BaseConfigurator) configuratorObject;
List<Attribute> baseConfigAttributeList = baseConfigurator.getAttributes();
for (Attribute attribute : baseConfigAttributeList) {
if (attribute.multiple) {
System.out.println(
"This is a multiple attribute " + attribute.getType() + " " + attribute
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@timja We need a way to get the most out of the multiple attributes.
This is a multiple attribute class hudson.slaves.NodeProperty globalNodeProperties
If we could expand globalNodeProperties that would be great.

.getName());
} else {
if (attribute.type.isEnum()) {
System.out.println("This is an enumeration attribute: ");
if (attribute.type.getEnumConstants().length != 0) {
System.out.println("Printing Enumeration constants for: " + attribute.getName());
for (Object obj : attribute.type.getEnumConstants()) {
System.out.println("EConstant : " + obj.toString());
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@timja The Enumeration constants are working fine:
Printing Enumeration constants for: mode
EConstant : NORMAL
EConstant : EXCLUSIVE

}
}
}
}
}
} else if (configuratorObject instanceof HeteroDescribableConfigurator) {
System.out.println("Instance of HeteroDescribable Configurator");
}
}
}

}

Expand Up @@ -117,6 +117,7 @@ public CNode describe(T instance, ConfigurationContext context) {
}).getOrNull();
}


Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert

@SuppressWarnings("unused")
public Map<String, Class<T>> getImplementors() {
return getDescriptors()
Expand Down
Expand Up @@ -67,6 +67,7 @@ public CNode describe(Object instance, ConfigurationContext context) {
return new Scalar(((Secret) instance).getEncryptedValue()).encrypted(true);
}
if (target.isEnum()) {
System.out.println(instance.getClass().getSimpleName());
return new Scalar((Enum) instance);
}

Expand Down