Support seed job creation in Configuration as Code, updated #1173

jetersen · 2019-04-25T04:17:46Z

It is using a released version and the changes that have happened to the configurator in the meantime.
SeedJob configurator now has access to JCasC secrets.

…-as-code managed jenkins master Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>

daspilker

LGTM. Please remove any noise from the code as mentioned in my comments. Can we close #1142?

daspilker · 2019-04-25T07:23:29Z

job-dsl-plugin/src/main/groovy/javaposse/jobdsl/plugin/casc/SeedJobConfigurator.java

+
+    @Override
+    public GeneratedItems[] check(CNode config, ConfigurationContext context) throws ConfiguratorException {
+        // Any way to dry-run a job-dsl script ?


No, not yet.

daspilker · 2019-04-25T07:24:26Z

job-dsl-plugin/src/main/groovy/javaposse/jobdsl/plugin/casc/ConfigurableScriptSource.java

+import io.jenkins.plugins.casc.model.CNode;
+
+/**
+ * @author <a href="mailto:nicolas.deloof@gmail.com">Nicolas De Loof</a>


Please remove all author tags.

jetersen · 2019-04-25T09:07:39Z

@daspilker Done would be great to get this into job dsl 🤞

as demonstrated by Matrix auth plugin the configurators can co-exist as long as the ordinal is higher than what is than what configuration-as-code-support-plugin ordinal is 😅

jenkinsci/matrix-auth-plugin#48

jetersen · 2019-04-25T09:55:28Z

We just merged jenkinsci/configuration-as-code-plugin#845
I will see if I can create a similar PR for removing job dsl from jcasc-support plugin

timja · 2019-04-25T09:56:21Z

There's no incremental build here @Casz

jetersen · 2019-04-25T10:11:22Z

I know, will have to deploy a snapshot 😆

daspilker · 2019-04-25T11:10:14Z

@Casz Can I release this change anytime or do you need to remove Job DSL from jcasc-support plugin first?

jetersen · 2019-04-25T11:16:12Z

you can release it anytime @daspilker 🙌 it was just if you wanted proof that it works since we have some integrations tests.

The PR would be very similar to jenkinsci/configuration-as-code-plugin#845

jetersen · 2019-04-25T11:22:26Z

@daspilker let me get the integration tests working... 😓 Seems we might have trouble

daspilker · 2019-04-25T11:23:31Z

@Casz I did a manual test. I never used CasC before, I wanted to see how it works. Quite nice actually 👍 . This Job DSL section does not appear in the reference. Is that expected?

jetersen · 2019-04-25T11:25:34Z

Oh I just forgot to deploy all the snapshots... I only deployed the plugin not core...

jetersen · 2019-04-25T11:26:57Z

That is expected, we could add some sort of support for exporting like storing the sources. Shouldn't be too difficult 😓

daspilker · 2019-04-25T12:29:38Z

What do you mean by storing the sources? Must that be done on the CasC side? Will that be available soon?

I would also be fine with adding a wiki page with some examples that we can point to. https://github.com/jenkinsci/job-dsl-plugin/tree/master/docs will be pushed to the wiki on every Job DSL release. So you could add a short page to this PR. Maybe you have some existing docs from the support plugin?

timja · 2019-04-25T12:31:11Z

Existing docs are here:
https://github.com/jenkinsci/configuration-as-code-plugin/tree/master/demos/jobs

Seems retrieving from a url or file is undocumented there, it only has inline

jetersen · 2019-04-25T12:41:31Z

What do you mean by storing the sources? Must that be done on the CasC side? Will that be available soon?

The configurator just has to describe the sources so it can be exported.
Something we can fix in this PR.

jetersen · 2019-04-25T16:01:42Z

@timja, I can't for the life of me figure out how to access the ScriptSource instances and describe them.

Unless you have some ideas let's skip the export.

I'll add some docs

jetersen · 2019-04-26T06:51:31Z

🎉

linux-samurai · 2019-04-27T23:57:21Z

The release yesterday broke all of our jobs that are using some custom code we wrote in a shared-pipeline-library. I can provide more details about the custom code if needed, but the error message from the jenkins master is here. Jobs were working up until jenkins master ran auto-updates and pulled this new version down (from 1.72 to 1.73). Please advise.

WARNING: Failed to instantiate Key[type=javaposse.jobdsl.plugin.casc.FromFileScriptSource$DescriptorImpl, annotation=[none]]; skipping this component
com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Error injecting constructor, java.lang.NoClassDefFoundError: javaposse/jobdsl/plugin/casc/FromFileScriptSource
  at javaposse.jobdsl.plugin.casc.FromFileScriptSource$DescriptorImpl.<init>(FromFileScriptSource.java:28)

Caused by: java.lang.NoClassDefFoundError: javaposse/jobdsl/plugin/casc/FromFileScriptSource

com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        ... 54 more

jetersen · 2019-04-28T00:29:25Z

Not sure whether you're actually using JCasC or not.

if you're using JCasC you should define your shared libraries inside JCasC.
So JobDSL can access your shared library code I guess.

You seem to have left out the context of where this error message appeared @linux-samurai
Inside Jenkins startup? When builds are run? Somewhere else.

Do you have JCasC installed or not?
https://github.com/jenkinsci/configuration-as-code-plugin (the plugin should be optional, you do not need to install it)

linux-samurai · 2019-04-28T01:44:01Z

@Casz I did not write our shared library code, and still getting up to speed on Groovy coding, so please bear with me. We are not using the 'configuration as code' plugin, but do a lot of the similar functionality within our shared library that the configuration as code plugin looks to be doing.

An example of one of our Jenkinsfiles is as follows, which loads the shared library:

@Library('company-pipeline-lib') _
archetype('GitOrganizationNameRedacted/jenkins-repository-name-redacted') {
config_select = [ applications: params.deploy_application ]
 }

We then have an 'archetype.groovy' in our shared library that takes the input and pulls that declarative pipeline code and injects the 'config_select'. We have a lot of custom DSL methods we wrote that allow us to extend the declarative pipeline a lot, for example we have a 'define' custom method that will take just about any input (string, boolean, key/value pair, array, etc). So this release of job-dsl plugin, and from the error, this specific PR, broke every job we have where this job-dsl-plugin is installed and where we are using our custom DSL methods in our shared library. We have downgraded to 1.72 to get unblocked, but I would like to help flesh out what the root cause is (in our code and how should we improve it or improvements or plugin code improvements) and help improve the code in general, because I wouldn't want us to be locked at this version forever.

If you could let me know if there is something specific I can look for in our shared library code I can circle back.

here is another example of a custom method that was written by the original author, in case it helps:```

@NonCPS
def getBuild(jobName, buildId) {
Jenkins.get().getItemByFullName(jobName)?.getBuild(buildId)
}

linux-samurai · 2019-04-28T02:19:42Z

@Casz Sorry, I want to explicitly answer your questions, in case they were not laid out fully in my response earlier.

We are not using the config-as-code plugin.
This only affects systems where we have job-dsl plugin installed.
For now we have downgraded job-dsl plugin to 1.72.
The error was being thrown in a part of the code where we are calling our 'gcloud' custom DSL method.

(job code)

 gcloud 'app deploy', app_yaml, version: appVersion, transform: { it.versions?.first() }

In the shared library, the gcloud.groovy file is a bit lengthy, but here is what we are importing. I'd need to review the code a bit more, and get approval to share it. Initial review doesn't have any glaring sensitivity concerns.

import groovy.json.*
import util.*

timja · 2019-04-28T07:04:17Z

Sounds like an optional plugin dependency issue?

jetersen · 2019-04-28T11:58:38Z

Nah, sounds we have duplicate symbols:

job-dsl-plugin/job-dsl-plugin/src/main/groovy/javaposse/jobdsl/plugin/casc/FromFileScriptSource.java

Line 27 in ff202a7

@Symbol("file")

daspilker · 2019-04-30T08:21:19Z

This has been reported as JENKINS-57218.

Shouldn't symbols have their descriptor type as namespace? Why would the Pipeline DSL pick the FromFileScriptSource in this case? Isn't this a Pipeline problem?

timja · 2019-04-30T08:30:38Z

Would the easiest fix be to rename fromFileScriptSource symbol to be something like?:

filePath

or

fileLocation
``

jetersen · 2019-04-30T09:21:27Z

or jobdslFile to avoid conflicts all together

jetersen · 2019-04-30T09:56:26Z

I agree @daspilker it is a problem in the pipeline. It has been reported several times that Symbol lacks more control.
CC @abayer

michelzanini · 2019-04-30T13:00:44Z

This has broke our builds also. It fails in startup but also on pipeline executions.
I do not have JCasC installed.

jetersen · 2019-05-05T11:47:06Z

It seems the fix was: #1175 and jenkinsci/structs-plugin#45
Sorry for not marking it optional in this PR completely forgot 😆
Lucky we caught a null pointer in structs plugin 🙌
@linux-samurai @michelzanini hopefully v1.74 should fix it.

jetersen · 2020-05-24T23:56:01Z

FYI jenkinsci/configuration-as-code-plugin#1394 which is released in v1.41 should fix any ordering to the root elements so the job dsl seed job should load last.

ndeloof and others added 2 commits April 25, 2019 05:49

(optional) extension to support defining seed jobs on a configuration…

0313cfa

…-as-code managed jenkins master Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>

update jcasc to released version

fa8dd82

daspilker requested changes Apr 25, 2019

View reviewed changes

remove comments as requested

b3cf97f

daspilker mentioned this pull request Apr 25, 2019

Support seed job creation in Configuration as Code #1142

Closed

daspilker approved these changes Apr 25, 2019

View reviewed changes

jetersen mentioned this pull request Apr 25, 2019

Remove job-dsl from support plugin jenkinsci/configuration-as-code-plugin#861

Merged

5 tasks

jetersen added 2 commits April 25, 2019 19:56

attempt at add docs for jcasc jobs

a259cc6

add jcasc as a test dependency

e28f683

daspilker merged commit f52cda2 into jenkinsci:master Apr 26, 2019

jetersen deleted the casc branch April 30, 2019 09:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support seed job creation in Configuration as Code, updated #1173

Support seed job creation in Configuration as Code, updated #1173

jetersen commented Apr 25, 2019

daspilker left a comment

daspilker Apr 25, 2019

daspilker Apr 25, 2019

jetersen commented Apr 25, 2019 •

edited

Loading

jetersen commented Apr 25, 2019

timja commented Apr 25, 2019

jetersen commented Apr 25, 2019

daspilker commented Apr 25, 2019

jetersen commented Apr 25, 2019

jetersen commented Apr 25, 2019

daspilker commented Apr 25, 2019

jetersen commented Apr 25, 2019 •

edited

Loading

jetersen commented Apr 25, 2019

daspilker commented Apr 25, 2019

timja commented Apr 25, 2019

jetersen commented Apr 25, 2019 •

edited

Loading

jetersen commented Apr 25, 2019 •

edited

Loading

jetersen commented Apr 26, 2019

linux-samurai commented Apr 27, 2019

jetersen commented Apr 28, 2019 •

edited

Loading

linux-samurai commented Apr 28, 2019 •

edited

Loading

linux-samurai commented Apr 28, 2019

timja commented Apr 28, 2019

jetersen commented Apr 28, 2019

daspilker commented Apr 30, 2019

timja commented Apr 30, 2019

jetersen commented Apr 30, 2019

jetersen commented Apr 30, 2019

michelzanini commented Apr 30, 2019

jetersen commented May 5, 2019 •

edited

Loading

jetersen commented May 24, 2020 •

edited

Loading

Support seed job creation in Configuration as Code, updated #1173

Support seed job creation in Configuration as Code, updated #1173

Conversation

jetersen commented Apr 25, 2019

daspilker left a comment

Choose a reason for hiding this comment

daspilker Apr 25, 2019

Choose a reason for hiding this comment

daspilker Apr 25, 2019

Choose a reason for hiding this comment

jetersen commented Apr 25, 2019 • edited Loading

jetersen commented Apr 25, 2019

timja commented Apr 25, 2019

jetersen commented Apr 25, 2019

daspilker commented Apr 25, 2019

jetersen commented Apr 25, 2019

jetersen commented Apr 25, 2019

daspilker commented Apr 25, 2019

jetersen commented Apr 25, 2019 • edited Loading

jetersen commented Apr 25, 2019

daspilker commented Apr 25, 2019

timja commented Apr 25, 2019

jetersen commented Apr 25, 2019 • edited Loading

jetersen commented Apr 25, 2019 • edited Loading

jetersen commented Apr 26, 2019

linux-samurai commented Apr 27, 2019

jetersen commented Apr 28, 2019 • edited Loading

linux-samurai commented Apr 28, 2019 • edited Loading

linux-samurai commented Apr 28, 2019

timja commented Apr 28, 2019

jetersen commented Apr 28, 2019

daspilker commented Apr 30, 2019

timja commented Apr 30, 2019

jetersen commented Apr 30, 2019

jetersen commented Apr 30, 2019

michelzanini commented Apr 30, 2019

jetersen commented May 5, 2019 • edited Loading

jetersen commented May 24, 2020 • edited Loading

jetersen commented Apr 25, 2019 •

edited

Loading

jetersen commented Apr 25, 2019 •

edited

Loading

jetersen commented Apr 25, 2019 •

edited

Loading

jetersen commented Apr 25, 2019 •

edited

Loading

jetersen commented Apr 28, 2019 •

edited

Loading

linux-samurai commented Apr 28, 2019 •

edited

Loading

jetersen commented May 5, 2019 •

edited

Loading

jetersen commented May 24, 2020 •

edited

Loading