Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

馃毀 Schema Generation for nested yml configurations #1027

Open
wants to merge 56 commits into
base: master
from

Conversation

@sladyn98
Copy link
Contributor

sladyn98 commented Aug 27, 2019

This PR deals with the schema generation for nested yml files.
We would be focusing on
a) Implementation of a describeStructure function for all the configurators which would then be used in the schema generation.
b) Testing Framework for the Schema
Closes #1038
Closes #997

Your checklist for this pull request

馃毃 Please review the guidelines for contributing to this repository.

  • [x ] Make sure you are requesting to pull a topic/feature/bugfix branch (right side) and not your master branch!
  • [ x] Ensure that the pull request title represents the desired changelog entry
  • [x ] Please describe what you did
  • [x ] Link to relevant issues in GitHub or in Jenkins JIRA
  • [x ] Link to relevant pull requests, esp. upstream and downstream changes
  • Did you provide a test-case? That demonstrates feature works or fixes the issue.
@@ -117,6 +117,18 @@ public CNode describe(T instance, ConfigurationContext context) {
}).getOrNull();
}

@CheckForNull
public void describeStructure(T instance, ConfigurationContext context) {
lookupConfigurator(context, instance.getClass())

This comment has been minimized.

Copy link
@sladyn98

sladyn98 Aug 27, 2019

Author Contributor

@casz I am trying to understand the inital implementation of the describe function.
Would this return to me all of the configurators given a context.

This comment has been minimized.

Copy link
@casz

casz Aug 27, 2019

Member

it provides it looks up the configurator and then you could loop over that and ask for it's confiurators

return new Scalar(preferredSymbol(instance.getDescriptor()));
}).getOrNull();

System.out.println(instance.getDescriptor());

This comment has been minimized.

Copy link
@sladyn98

sladyn98 Aug 27, 2019

Author Contributor

Should this be used in a different context. I could not quite use the getDescriptorList.

@sladyn98 sladyn98 force-pushed the sladyn98:nested_yaml_schema branch from 521d972 to 13e893a Aug 27, 2019
lookupConfigurator(context, instance.getClass())
.map(configurator -> convertToNode(context, configurator, instance))
.filter(Objects::nonNull)
.map(node -> {
return new Scalar(preferredSymbol(instance.getDescriptor()));
if(node.getClass().isEnum()){
return instance.getDescriptor().getDisplayName();

This comment has been minimized.

Copy link
@sladyn98

sladyn98 Aug 27, 2019

Author Contributor

@casz So if the node is an enumeration probably I could get the descriptors for the node. But as in the jelly we need all of the implementors from the heterodescribable configurators which if we use the instance we cannot seem to obtain.
This would just return all of the descriptors for the heterodescribable configurators right?

@sladyn98 sladyn98 added the dev-tools label Aug 27, 2019
@sladyn98 sladyn98 self-assigned this Aug 27, 2019
Community Bridge: Jenkins Configuration-as-Code developer tools automation moved this from Done to In progress Sep 27, 2019
@timja

This comment has been minimized.

Copy link
Member

timja commented Sep 27, 2019

Could you add a test for the configuration-as-code root element for the schema validation @sladyn98

configuration-as-code:
version: 1
deprecated: warn
restricted: warn
unknown: warn

@@ -117,6 +117,7 @@ public CNode describe(T instance, ConfigurationContext context) {
}).getOrNull();
}


This comment has been minimized.

Copy link
@timja

timja Sep 28, 2019

Member

revert

@timja

This comment has been minimized.

Copy link
Member

timja commented Sep 28, 2019

@sladyn98 the new schema is worse than earlier, after the pair coding we had the right base categories showing up,
Now everything is showing up as allowed at the base,

image

I'm using the schema from SchemaGenerationTest.writeSchema

we need i.e. a test that puts an invalid base category in that proves it gets rejected

i.e.
something like:

 jenkins:
  systemMessage: "Hi"
invalidCategory:
  iDoNothing: "Hi"
@sladyn98

This comment has been minimized.

Copy link
Contributor Author

sladyn98 commented Sep 28, 2019

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "description": "Jenkins Configuration as Code",
  "type": "object",
  "properties": {
    "configuration-as-code": {
      "additionalProperties": false,
      "id": "#/definitions/configurationcontext",
      "type": "object",
      "title": "Configuration base for the configuration-as-code classifier"
    },  
      }

Thats because all the root configurators come under the base and not outside, I had to fix it because the previous change was breaking a lot of the other fields.
I am looking at fixing it now.

@timja

This comment has been minimized.

Copy link
Member

timja commented Sep 28, 2019

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "description": "Jenkins Configuration as Code",
  "type": "object",
  "properties": {
    "configuration-as-code": {
      "additionalProperties": false,
      "id": "#/definitions/configurationcontext",
      "type": "object",
      "title": "Configuration base for the configuration-as-code classifier"
    },  
      }

Thats because all the root configurators come under the base and not outside, I had to fix it because the previous change was breaking a lot of the other fields.
I am looking at fixing it now.

could you write a failing test first please, I think we could benefit from defining what we expect in tests, so less manual testing will be required especially on changes

@sladyn98

This comment has been minimized.

Copy link
Contributor Author

sladyn98 commented Sep 28, 2019

could you write a failing test first please, I think we could benefit from defining what we expect in tests, so less manual testing will be required especially on changes

Yeah sure I have already added it, adding it in the next commit

@sladyn98 sladyn98 requested a review from timja Sep 28, 2019
@sladyn98

This comment has been minimized.

Copy link
Contributor Author

sladyn98 commented Sep 30, 2019

jenkins:
  jenkins:
    systemMessage: "hello"
    numExecutors: 2

Looks like the root configurator jenkins has its own attribute jenkins so the schema ends up like this
cc @timja

@timja

This comment has been minimized.

Copy link
Member

timja commented Sep 30, 2019

jenkins:
  jenkins:
    systemMessage: "hello"
    numExecutors: 2

Looks like the root configurator jenkins has its own attribute jenkins so the schema ends up like this
cc @timja

Have you debugged the JenkinsConfigurator to see why it's adding that?

@timja

This comment has been minimized.

Copy link
Member

timja commented Sep 30, 2019

Test failures for:

Couldn't find file: validSelfConfig.yml

Couldn't find file: validJenkinsBaseConfig.yml

https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fconfiguration-as-code-plugin/detail/PR-1027/37/tests

@sladyn98

This comment has been minimized.

Copy link
Contributor Author

sladyn98 commented Oct 1, 2019

Test failures for:

Couldn't find file: validSelfConfig.yml

Couldn't find file: validJenkinsBaseConfig.yml

https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fconfiguration-as-code-plugin/detail/PR-1027/37/tests

This is strange the tests were passing locally. @timja

@sladyn98

This comment has been minimized.

Copy link
Contributor Author

sladyn98 commented Oct 1, 2019

jenkins:
  jenkins:
    systemMessage: "hello"
    numExecutors: 2

Looks like the root configurator jenkins has its own attribute jenkins so the schema ends up like this
cc @timja

Have you debugged the JenkinsConfigurator to see why it's adding that?

@timja Yeah the jenkins base configurator seeems to have an attribute jenkins

@timja

This comment has been minimized.

Copy link
Member

timja commented Oct 1, 2019

Try run the tests on the command line with mvn test -Dtest=Testname likely it will fail there

sladyn98 and others added 2 commits Oct 2, 2019
Caused by node linking back to jenkins
Doesn't fully work, all base attributes to the jenkins field seem to be
missing, i.e. systemMessage, numExecutors
@sladyn98

This comment has been minimized.

Copy link
Contributor Author

sladyn98 commented Oct 3, 2019

@timja
Screenshot from 2019-10-03 20-47-57

Your last change seems to be breaking the writeSchemaTest.

@casz

This comment has been minimized.

Copy link
Member

casz commented Oct 4, 2019

If it errors on

return Jenkins.getInstance().getExtensionList(ConfigurationAsCode.class).get(0);
it means that the plugin was not loaded correctly

@sladyn98

This comment has been minimized.

Copy link
Contributor Author

sladyn98 commented Oct 4, 2019

java.lang.NullPointerException
	at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:705)

Looks like its a NPE here @timja I guess it is not due to your changes.Strange that the tests are failing out of the blue.

cc @oleg-nenashev

@timja

This comment has been minimized.

Copy link
Member

timja commented Nov 13, 2019

can you resolve conflicts please?

@oleg-nenashev oleg-nenashev self-requested a review Nov 13, 2019
@sladyn98

This comment has been minimized.

Copy link
Contributor Author

sladyn98 commented Nov 15, 2019

@timja resolved

}
}

public static void storeConfiguratorNames() {

This comment has been minimized.

Copy link
@sladyn98

sladyn98 Nov 15, 2019

Author Contributor

@timja I am not sure this is necessary anymore

This comment has been minimized.

Copy link
@timja

timja Nov 15, 2019

Member

remove if not needed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
4 participants
You can鈥檛 perform that action at this time.