SCM Import/Export with high job count #1982

Closed
penekk opened this Issue Aug 4, 2016 · 6 comments

Projects

None yet

5 participants

@penekk
penekk commented Aug 4, 2016 edited

With versions 2.6.6 and 2.6.8 configuring SCM export for a substantially sized project ( ~ 750 jobs ) results in awfully long processing time (from my understanding its generating files in local repository) - nginx as proxy times out and the process continues in the background. Moreover, during that time accessing Job list of the project resulted in "Cannot serialize job" errors, until all of the files were generated. Scheduled jobs however work fine.
Once I discovered variacode95@0d80538
and variacode95@52674c9) I decided to upgrade to 2.6.9 a moment ago. No serialize error has occured yet , although the process still takes ages.

Configuring the Import however, results in:

Request: /project/[project]/jobs?max=20&offset=0
Message: Cannot get property 'tree' on null object 
Caused by: Cannot get property 'tree' on null object 
Class: MenuController 
At Line: [231] 
Code Snippet:
Stack Trace

java.lang.NullPointerException: Cannot get property 'tree' on null object
    at org.rundeck.plugin.scm.git.GitImportPlugin.importSynchStateForStatus(GitImportPlugin.groovy:302)
    at org.rundeck.plugin.scm.git.GitImportPlugin.refreshJobStatus(GitImportPlugin.groovy:242)
    at org.rundeck.plugin.scm.git.GitImportPlugin.getJobStatus(GitImportPlugin.groovy:349)
    at org.rundeck.plugin.scm.git.GitImportPlugin.getJobStatus(GitImportPlugin.groovy:337)
    at rundeck.services.ScmService$_importStatusForJobs_closure15.doCall(ScmService.groovy:961)
    at rundeck.services.ScmService.importStatusForJobs(ScmService.groovy:956)
    at rundeck.controllers.MenuController$_closure10.doCall(MenuController.groovy:323)
    at rundeck.controllers.MenuController$_closure9.doCall(MenuController.groovy:231)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1360)
    at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
    at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:539)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
    at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:288)
    at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:115)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
    at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
    at com.dtolabs.rundeck.server.filters.AuthFilter.doFilter(AuthFilter.java:58)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:486)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
    at org.eclipse.jetty.server.Server.handle(Server.java:349)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:910)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:634)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
    at java.lang.Thread.run(Thread.java:745)

Repeating configuration for a much much smaller project (31 jobs) seems to be working okay, at least during few brief tests, although for the jobs to start being tracked (import) I have to restart rundeck entirely - after that it both detects changes made from rundeck ui and in remote repo, which then can be pushed/pulled respectively.

Has anyone tested/investigated this? Current state makes the plugin unsusable for a project of this size which at the moment is a crucial feature I am trying to get to work. I am also wondering, whether having tried configuring the project multiple times is the reason the 'null tree' error persists. Mariadb datasource and rundeck instance(s) hosts are located in the same dc.

Edit: perhaps this may be of relevance, there are 2 rundeck instances connected to the same database, one is running in passive mode as slave, with cluster mode disabled (takeover is via vrrp).

@rophy
Contributor
rophy commented Aug 30, 2016

+1 on the exception: java.lang.NullPointerException: Cannot get property 'tree' on null object

I have 108 jobs.

@penekk penekk changed the title from SCM Import/Export with high job amount to SCM Import/Export with high job count Aug 30, 2016
@penekk
penekk commented Aug 30, 2016

During that time new errors haven't occured. The loading times are still extremely long, which renders rundeck webui close to excruciating sometimes.

@gschueler gschueler added the bug label Sep 1, 2016
@johnpaularthur
johnpaularthur commented Sep 2, 2016 edited

+1, i get the error on only < 30 jobs

Strangely enough, i downgraded from 2.6.9-1 to 2.6.7-1 (previous install). Had the same error as above. I've had it working before, but not sure which version it was.

@ARentz07

I am seeing the same thing with only 23 jobs in my project. I'm on 2.6.7-1.

screen shot 2016-09-15 at 2 59 32 pm

@gschueler
Contributor

I believe this can occur if you originally imported jobs from one repo to a project, then later changed to another repo

@gschueler gschueler added this to the 2.6.10 milestone Sep 16, 2016
@gschueler gschueler added a commit to gschueler/rundeck that referenced this issue Sep 20, 2016
@gschueler gschueler fix #1982 only compare existing commit id if import url matches and c…
…ommit exists
09f9225
@gschueler gschueler self-assigned this Sep 20, 2016
@rophy
Contributor
rophy commented Sep 20, 2016

@gschueler In case it helps: in my case, I wanted to sync jobs between 2 different rundeck servers. (identical project name, but different server).

@gschueler gschueler closed this in #2077 Sep 20, 2016
@gschueler gschueler removed the in progress label Sep 20, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment