Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot launch GCE agents when plugin configured with JCasC #53

Closed
robinbanbury opened this issue Mar 22, 2019 · 23 comments
Closed

Cannot launch GCE agents when plugin configured with JCasC #53

robinbanbury opened this issue Mar 22, 2019 · 23 comments

Comments

@robinbanbury
Copy link

Hi all,

When configuring this plugin using JCasC, GCE agent VMs will not launch. The relevant fields seems to be populated in the Jenkins 'Configure System' UI, but the VMs are not able to launch until Jenkins' configuration is saved using the UI.

Here is my JCasC configuration relating to this plugin. In my case, I'm creating a brand new Jenkins instance from scratch, as you might do when running the Jenkins master in a Docker container.

jenkins:
  clouds:
  - computeEngine:
      cloudName: gce-jenkins-build
      projectId: gce-jenkins
      instanceCapStr: 1
      credentialsId: gce-jenkins
      configurations:
      - namePrefix:         jenkins-agent-image
        description:        Jenkins agent
        launchTimeoutSecondsStr: 6
        retentionTimeMinutesStr: 300
        mode:               EXCLUSIVE
        labelString:        jenkins-agent
        numExecutorsStr:    1
        runAsUser:          jenkins
        remoteFs:           '' # tried not setting this, field added when 'save' clicked in UI
        windows:            false
        windowsPasswordCredentialsId: ''    # tried not setting, added when saved in UI
        windowsPrivateKeyCredentialsId: ''  # tried not setting, added when saved in UI
        oneShot:            true
        createSnapshot:     false
        region:             "https://www.googleapis.com/compute/v1/projects/gce-jenkins/regions/europe-west1"
        zone:               "https://www.googleapis.com/compute/v1/projects/gce-jenkins/zones/europe-west1-a"
        template:           '' # tried not setting, added when 'saved' in UI
        machineType:        "https://www.googleapis.com/compute/v1/projects/gce-jenkins/zones/europe-west1-a/machineTypes/n1-standard-2"
        preemptible:        false
        minCpuPlatform:     '' # tried not setting, added when 'saved' in UI
        startupScript:      '' # tried not setting, added when 'saved' in UI
        networkConfiguration:
          sharedVpc:
            projectId:      gce-jenkins-cloud-123456
            region:         europe-west1
            subnetworkShortName: gce-jenkins-cloud
        networkTags:        jenkins-agent
        externalAddress:    true
        useInternalAddress: false
        bootDiskSourceImageProject: gce-jenkins
        bootDiskSourceImageName: "https://www.googleapis.com/compute/v1/projects/gce-jenkins/global/images/gce-jenkins-build-image"
        bootDiskType:       "https://www.googleapis.com/compute/v1/projects/gce-jenkins/zones/europe-west1-a/diskTypes/pd-standard"
        bootDiskSizeGbStr:  50
        bootDiskAutoDelete: true
        serviceAccountEmail: 'jenkins-test@gce-jenkins-ops.iam.gserviceaccount.com'

I did a comparison of config.xml before and after hitting save in the UI, and there's no difference in the GCE plugin configuration section, but Jenkins is suddenly able to launch GCE VMs.

(For the fields marked with '#' above, I initially tried leaving out that configuration entirely from the JCasC configuration, but it got added to Jenkins' config.xml when I hit 'save' in the Jenkins 'Configure System' UI (and still had the same problem of no VMs launching). In order to minimise the diff between the state of config.xml before an after hitting save in the UI, I added it to the JCasC configuration.)

As noted by @devqore in this JCasC issue Google Compute nodes are also being disconnected during a JCasC configuration reload - if Jenkins has more than one permanent node configured in addition to the GCE nodes.

@robinbanbury
Copy link
Author

Here are the logs of JCasC configuring my Jenkins instance:

Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.ComputeEngineCloud.cloudName = gce-jenkins-build
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.ComputeEngineCloud.projectId = gce-jenkins
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.ComputeEngineCloud.credentialsId = gce-jenkins
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.ComputeEngineCloud.instanceCapStr = 1
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.namePrefix = jenkins-agent-image
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.region = https://www.googleapis.com/compute/v1/projects/gce-jenkins/regions/europe-west1
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.zone = https://www.googleapis.com/compute/v1/projects/gce-jenkins/zones/europe-west1-a
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.machineType = https://www.googleapis.com/compute/v1/projects/gce-jenkins/zones/europe-west1-a/machineTypes/n1-standard-2
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.numExecutorsStr = 1
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.startupScript =
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.preemptible = false
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.minCpuPlatform =
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.labelString = jenkins-agent
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.description = Jenkins agent
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.bootDiskType = https://www.googleapis.com/compute/v1/projects/gce-jenkins-ops/zones/europe-west1-a/diskTypes/pd-standard
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.bootDiskAutoDelete = true
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.bootDiskSourceImageName = https://www.googleapis.com/compute/v1/projects/gce-jenkins/global/images/gce-jenkins-build-image
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.bootDiskSourceImageProject = gce-jenkins
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.bootDiskSizeGbStr = 100
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.windows = false
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.windowsPasswordCredentialsId =
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.windowsPrivateKeyCredentialsId =
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.createSnapshot = false
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.remoteFs =
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.SharedVpcNetworkConfiguration.projectId = gce-jenkins-cloud-123456
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.SharedVpcNetworkConfiguration.region = europe-west1
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.SharedVpcNetworkConfiguration.subnetworkShortName = gce-jenkins-cloud
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.networkConfiguration = {sharedVpc={}}
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.externalAddress = true
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.useInternalAddress = false
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.networkTags = jenkins-agent
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.serviceAccountEmail = jenkins-test@gce-jenkins-ops.iam.gserviceaccount.com
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.retentionTimeMinutesStr = 300
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.launchTimeoutSecondsStr = 6
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.mode = EXCLUSIVE
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.runAsUser = jenkins
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.oneShot = true
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.template =
Mar 22, 2019 11:52:25 AM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
INFO: Setting class com.google.jenkins.plugins.computeengine.ComputeEngineCloud.configurations = [{}]

And the logs when I try to perform a build:

Mar 22, 2019 11:58:20 AM com.google.jenkins.plugins.computeengine.ComputeEngineCloud provision
INFO: Provisioning node from config com.google.jenkins.plugins.computeengine.InstanceConfiguration@37d78ac8 for excess workload of 1 units of label 'jenkins-agent'
Mar 22, 2019 11:58:20 AM hudson.triggers.SafeTimerTask run
SEVERE: Timer task hudson.slaves.NodeProvisioner$NodeProvisionerInvoker@b9e23c failed
java.lang.NullPointerException
    at com.google.jenkins.plugins.computeengine.ComputeEngineCloud.availableNodeCapacity(ComputeEngineCloud.java:243)
    at com.google.jenkins.plugins.computeengine.ComputeEngineCloud.provision(ComputeEngineCloud.java:197)
    at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715)
    at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320)
    at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61)
    at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809)
    at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
    at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

// several more messages like this, until I click 'save' in the Jenkins Configuration UI

Mar 22, 2019 12:00:00 PM com.google.jenkins.plugins.computeengine.ComputeEngineCloud provision
INFO: Provisioning node from config com.google.jenkins.plugins.computeengine.InstanceConfiguration@3e1245f6 for excess workload of 1 units of label 'jenkins-agent'
Mar 22, 2019 12:00:00 PM com.google.jenkins.plugins.computeengine.ComputeEngineCloud availableNodeCapacity
INFO: Found capacity for 1 nodes in cloud gce-gce-jenkins
Sent insert request
Mar 22, 2019 12:00:02 PM com.google.jenkins.plugins.computeengine.ComputeEngineComputerLauncher launch
INFO: Launch will wait 6000 for operation operation-1553256000737-584ad982b50d6-ac3f0754-acaf0393 to complete...
Mar 22, 2019 12:00:02 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning jenkins-agent-image-ipuyo8 from gce-gce-jenkins with 1 executors. Remaining excess workload: 0
Mar 22, 2019 12:00:02 PM com.google.jenkins.plugins.computeengine.ComputeEngineCloud lambda$provision$0

// node connects, build proceeds

@ingwarsw
Copy link
Contributor

I would say that readResolve(); is never called..
Probably would be nice to add it to provision method..

@robinbanbury
Copy link
Author

@ingwarsw I'm not too familiar with this plugin, but happy to be a guinea pig for an experimental fix, as I'm working on an ephemeral Jenkins master configured with JCasC at the moment.

I see ComputeEngineCloud and InstanceConfiguration both have readResolve and provision methods. I can try building the plugin and testing on my instance, if you have a suggestion of what to change?

@ingwarsw
Copy link
Contributor

@robincsmith Can you try using code from #54

@robinbanbury
Copy link
Author

Yes - I can try building and installing that today

@robinbanbury
Copy link
Author

@ingwarsw looks like the changes in #54 fix the problem :-D

Jenkins can spin up ephemeral VMs in GCE without needing to hit 'save' in the configuration UI

@robinbanbury
Copy link
Author

This doesn't fix the other issue I mentioned (GCE slaves disconnected after a JCasC config reload), but that might be a more general issue with JCasC - it seems to remove all ephemeral nodes (coming from plugins like this one, or the Docker / Yet Another Docker plugins) when JCasC config is reloaded

@ingwarsw
Copy link
Contributor

@robincsmith In your config you have nodes configured?

@robinbanbury
Copy link
Author

Yes - I think that's what causes the disconnection issue

@ingwarsw
Copy link
Contributor

@robincsmith I think I fixed it, see jenkinsci/configuration-as-code-plugin#804

@robinbanbury
Copy link
Author

Thanks @ingwarsw!

I built and deployed the plugin into my local environment. With your changes, the nodes do indeed persist when I click 'Reload existing configuration' or 'Apply new configuration'. However, clicking either those buttons also results in Demon Jenkins, with the following stacktrace:

java.lang.AssertionError: class io.jenkins.docker.DockerTransientNode is missing its descriptor
	at jenkins.model.Jenkins.getDescriptorOrDie(Jenkins.java:1531)
	at hudson.model.Slave.getDescriptor(Slave.java:567)
	at hudson.model.Slave.getDescriptor(Slave.java:100)
	at io.jenkins.plugins.casc.core.JenkinsConfigurator.isCloudNode(JenkinsConfigurator.java:90)
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
	at io.jenkins.plugins.casc.core.JenkinsConfigurator.lambda$null$4(JenkinsConfigurator.java:70)
	at io.jenkins.plugins.casc.Attribute.setValue(Attribute.java:170)
	at io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:354)
	at io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:266)
	at io.jenkins.plugins.casc.ConfigurationAsCode.lambda$configureWith$6(ConfigurationAsCode.java:654)
	at io.jenkins.plugins.casc.ConfigurationAsCode.invokeWith(ConfigurationAsCode.java:624)
	at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:654)
	at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:544)
	at io.jenkins.plugins.casc.ConfigurationAsCode.configure(ConfigurationAsCode.java:274)
	at io.jenkins.plugins.casc.ConfigurationAsCode.doReload(ConfigurationAsCode.java:151)
	at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
	at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:77)
	at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)
	at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:537)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
Caused: javax.servlet.ServletException
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:789)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
	at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:458)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:99)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:503)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
	at java.lang.Thread.run(Thread.java:748)

That seems to apply to ephemeral nodes added by the 'Docker' plugin. If I only have ephemeral GCE nodes active when I reload the config, no error is thrown. I guess that could be an issue with the underlying plugin? Or might it be something that can be remedied here?

@jetersen
Copy link
Member

jetersen commented Mar 29, 2019

Should be an easy fix for in the docker plugin. Though depends on how long PRs sit.
If you like @robincsmith, I can ship a PR that you could install and see if it fixes it.

@ingwarsw
Copy link
Contributor

@Casz @robincsmith Docker plugin could fix it (and probably should) but I can fix It also in PR (at least I think so)..

@ingwarsw
Copy link
Contributor

And personally I think that checking by isInstantiable() from Descriptor if node is cloud node is actually bad idea..
But seems like E2C plugin have only this as indicator..

@jetersen
Copy link
Member

jetersen commented Mar 29, 2019

Perhaps check isInstantiable last if the other two are true?

@jetersen
Copy link
Member

Perhaps fix E2C while we are at it 😕

@ingwarsw
Copy link
Contributor

They should implement their nodes based on AbstractCloudSlave As I see it should be rather easy..

@jetersen
Copy link
Member

@ingwarsw feel free to join gitter 😄
Are you fixing up E2C or Docker, I'll jump on the other one then.

@robinbanbury
Copy link
Author

robinbanbury commented Apr 1, 2019

I'm not sure exactly where to report this, given the plethora of PRs that have been raised (thanks for that!), but my testing has confirmed the following:

  • @ingwarsw with GCE Plugin Small refactoring fo JCasC #54, the GCE agents can start without needing 'save' to be hit in the UI.
  • Adding CasC Plugin PR #804 (already merged), ephemeral GCE nodes remain connected after a config reload (and no Devil Jenkins!)
  • @Casz adding Docker Plugin PR #726, ephemeral Docker nodes remain connected after a config reload too! Perhaps you can include Nicolas De Loof on this review, as I gather he is the maintainer of the Docker plugin and involved with Jenkins CasC too?

@jetersen
Copy link
Member

jetersen commented Apr 1, 2019

@robincsmith see https://groups.google.com/forum/#!searchin/jenkinsci-dev/maintainer$20role$20clarification%7Csort:date/jenkinsci-dev/BLIfRisUyag/NRfv7QzdBAAJ regarding Nicolas current maintainer status.

Hopefully, he is willing to review it 😅

@robinbanbury
Copy link
Author

🤞🤞🤞

@robinbanbury
Copy link
Author

👍 nice work, @Casz :-)

@ingwarsw
Copy link
Contributor

@stephenashank I think it can be closed.. AFAIK all issues with JCasC was resolved..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants