Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error while testing GitLab's Jenkins CI Integration with Jenkins' GitLab Plugin #272

Closed
radium226 opened this issue Apr 26, 2016 · 37 comments

Comments

@radium226
Copy link

Hello,

I'm trying to make GitLab 8.7 (using directly https://gitlab.com) and Jenkins 2.0 work together, but I got this error from Jenkins when I test (using the Test button of GitLab's Jenkins CI Service interface):

java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:320)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:163)
    at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:378)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
    at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:380)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
    at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
    at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:49)
    at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:44)
    at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:106)
    at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:44)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.BasicHeaderProcessor.success(BasicHeaderProcessor.java:140)
    at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:82)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.Server.handle(Server.java:499)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
    at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
    at com.dabsquared.gitlabjenkins.webhook.ActionResolver.onPost(ActionResolver.java:93)
    at com.dabsquared.gitlabjenkins.webhook.ActionResolver.resolveAction(ActionResolver.java:53)
    at com.dabsquared.gitlabjenkins.webhook.ActionResolver.resolve(ActionResolver.java:47)
    at com.dabsquared.gitlabjenkins.webhook.GitLabWebHook.getDynamic(GitLabWebHook.java:44)
    ... 73 more

I looked at the source code and it seems that the X-GitLab-Event is missing from the HTTP request.

Do you guys have an idea what going on?

@crussell52
Copy link

I am seeing the same behavior with a local jenkins 2.0 install and Gitlab 8.5.5-ee.

Same stacktrace.

Plugin version 1.2.0

@crussell52
Copy link

Downgraded the plugin to 1.1.32 in jenkins. Successfully works around this issue.

@4kochi
Copy link

4kochi commented Apr 28, 2016

Have same the problem. Downgraded to v1.1.32 and now it works fine.

@coder-hugo
Copy link
Contributor

I tested the plugin in Jenkins 2 running in the bundled Jetty and a Tomcat 8 and for both I hadn't trouble that the X-GitLab-Event header is missing. So I assume this is somehow related to your Jenkins/GitLab setup. Can you provide some details of your setup. Are there any systems in between Jenkins and GitLab, e.g. a proxy. Maybe you could try to use a packet sniffer (tcpdump) to see if the HTTP request from GitLab contains this header.

@pyther
Copy link

pyther commented Apr 30, 2016

I'm running Gitlab 8.6.4 and I'm not seeing any X-GitLab headers in the request sent from Gitlab -> Jenkins.

On my gitlab server 127.0.0.1:8090 maps -> jenkin2.example.com:8080 which is the raw Jetty server that is bundled with Jenkins.

Request

.POST /project/website/olcf HTTP/1.1
Content-Type: application/json
Authorization: Basic amVua2luczpkOTYzNjk0ODBhNTUxZWRiYTlhYjMwNDZmOTUzYmFhYg==
Connection: close
Host: 127.0.0.1:8090
Content-Length: 2858

{"object_kind":"push", <REDACTED>}

Response

.HTTP/1.1 500 Server Error
Date: Sat, 30 Apr 2016 21:05:12 GMT
X-Content-Type-Options: nosniff
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: no-cache,no-store,must-revalidate
X-Hudson-Theme: default
Content-Type: text/html;charset=UTF-8
Set-Cookie: JSESSIONID.e4f376e5=13uk8h13avml418djq3tme7xmz;Path=/;HttpOnly
X-Hudson: 1.395
X-Jenkins: 2.0
X-Jenkins-Session: 95b0fe99
X-Hudson-CLI-Port: 59437
X-Jenkins-CLI-Port: 59437
X-Jenkins-CLI2-Port: 59437
X-Frame-Options: sameorigin
X-Instance-Identity: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAntZ0W95stGDFeeuLAATV2ekTLn0/0/x1es0VGCM0J72r4U6DIfY5OHCA1WRobR7g8TgCK7xZ8k3QpI6Z5jTc6S5A2oTZP2bkzZs0G+wi+qvOyXfxGGWLJXnmhb5lYTAembwn66NDOFBejWkVAe2KFySLHUlzyeDdQnZy+gi4fj+9w
fvdBAw73Y8bqHk8RRwCugWjAMonD25GwDrg3XtZWURJNs0bUPRQ41sFYyuFiXt2RcDbs88+cQA7Nwlx4F8ypk7NuIq4dr0sKUFE3HEJ9rQVru2aAT4jZh+nxj8r4aUnH8qvEx19QLa8U2wz2eIaMzWNE53zPBm0eeM2V3AC0wIDAQAB
X-SSH-Endpoint: jenkins2.example.com:39643
Connection: close
Server: Jetty(9.2.z-SNAPSHOT)

@crussell52
Copy link

I am actually using apache mod_proxy to deliver over port 80.

If the root cause is indeed a missing header, maybe it is being stripped by apache. I won't be able to examine it until Monday.

@coder-hugo
Copy link
Contributor

I just tested this at work with GitLab EE and the Jenkins CI service. In this case the X-GitLab-Event header is missing for me, too. As a workaround you can configure a webhook instead of the Jenkins CI service in your GitLab.
@dblessing IIRC you were involved in the implementation of the Jenkins CI service in GitLab. Is there any reason why this doesn't send the X-GitLab-Event header?

@alexgrist
Copy link

Also experiencing this issue, had to downgrade to v1.1.32. Not using apache or nginx as a reverse proxy, just direct request to jenkins.

@crussell52
Copy link

As stated earlier, I am using apache as a reverse proxy to allow access to jenkins on port 80 (for user convenience).

To eliminate this as a variable, I modified my configuration in GitLab to point directly to the jenkins port. This bypasses apache entirely, but did not correct the issue.

Latest test with plugin: v1.2.1

@crussell52
Copy link

crussell52 commented May 3, 2016

Recreated once again, this time with:

Jenkins 2.1
Gitlab Jenkins CI Plugin: 1.2.1
Gitlab EE: 8.7.1

I also updated my gitlab configuration to access Jenkins by IP (internal network) to take the internal DNS out of the equation. Like my earlier test, I used my Jenkins port instead of port 80 to take the apache reverse-proxy out of the equation as well.

I got the same results. So I ran tcpdump on my gitlab server as suggested by @coder-hugo.

sudo tcpdump -nnvvASs 1514 dst 1.2.3.4

The relevant portion of the output from the GitLab service's "Test Settings" button:

(The IP has been changed to protect the innocent.)

...i.|..POST /project/<redacted> HTTP/1.1
Content-Type: application/json
Authorization: Basic <redacted>
Connection: close
Host: 1.2.3.4:8081
Content-Length: 5272

This does seem to confirm the absence of a X-GitLab-Event header at the point of origin.

@coder-hugo
Copy link
Contributor

@crussell52 as mentioned here #272 (comment) this seems to be a bug in the GitLab EE Jenkins CI Service. If you add a webhook instead of the Jenkins CI Service the request should contain the X-GitLab-Event header.

@coder-hugo
Copy link
Contributor

Add link to GitLab EE issue

@crussell52
Copy link

@coder-hugo Thank you for the followup and for raising the issue with GitLab. :)

@demaniak
Copy link
Contributor

demaniak commented May 4, 2016

Hi.

Same issue on:

  • Jenkins 2.1 (and 2.0)
  • GitLab Plugin 1.2.1
  • GitLab CE 7.10.4

I have been using webhooks all along (have not used GitLab CI feature once), everything was fine until somewhere after 28 April 2016 (date of the last successful build on our Jenkins instance)

I have builds triggered through a reverse proxy, and build triggers by-passing same said proxy (legacy...).

@coder-hugo
Copy link
Contributor

@demaniak the oldest tested version of GitLab for this plugin is 7.14 and this definitely sends the X-GitLab-Event header. I suggest you to upgrade your GitLab as 7.10.4 is almost one year old.

@apazga
Copy link

apazga commented May 4, 2016

@demaniak @coder-hugo I confirm the issue too using Gitlab CE 8.7.0.
I updated yesterday to Jenkins 2.1 and also updated every plugin except Gitlab one due to the Jenkins warning:
gitlabplugin

Also tried updating later the plugin and had to roll back due to all builds in Gitlab were in Skipped state.

Edit: they answered in your request @coder-hugo ;)

@coder-hugo
Copy link
Contributor

@apazga I openen an issue for the GitLab EE Jenkins CI service which doesn't send this header the normal webhook should send this header. I tested this with GitLab CE 8.7.0 and GitLab EE 8.7.0. If the header is missing for any GitLab CE installation this has to be related to your setup.

@demaniak
Copy link
Contributor

demaniak commented May 4, 2016

@coder-hugo @apazga personally I am completely willing to explore and assist with any explanation/resolution for this issue.
For the sake of clarity (and hopefully to help along resolution), the events for us were as follows:

  • Shortly before 28 April 2016, I upgrade to Jenkins 2.0 via normal apt-get channels. Had to re-tweak some things to get Jenkins working completely (max heap memory etc).
  • 28 April 2016: builds are all working as normal, with GitLab CE 7.10, Jenkins 2.0, webhooks. Not 100% sure about gitlab plugin version, but I suspect it was 1.2.0
  • Leave and long weekend for me, not much activity build wise, definitely no config changes in jenkins or gitlab
  • Tuesday (3 May 2016) morning arrives, MR created...and we are dead in the water.
  • Ugrade Jenkins to 2.1 and gitlab plugin to 1.2.1 in the hope that it might resolve the problem, but still broken

As far as I can tell things broke while standing still - so it feels like there is a piece missing in this puzzle somewhere. I would believe the problem is completely on our internal setup...except that this issue exists (so other persons are also experiencing the problem).

We are currently in the process of updating our Gitlab CE instance to see if that maybe solves anything. Will report on results.

@crussell52
Copy link

crussell52 commented May 4, 2016

tldr; It looks like GitLab has put up a MR which will make sure the missing header is delivered (https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/374). Compatibility with GitLab versions before the patch is a question.

The current version of this plugin (1.2.x) has a clear expectation of the X-Gitlab-Event header. I haven't done the research to see when it was introduced, but I do know that @coder-hugo has done some major cleanup of this plugin, so it is possible it was introduced during that process. (BTW, I've been poking around the code. IMHO, it is put together in a clean way which will make it easier to enhance down the road).

In order to work around this, @coder-hugo would need to (at best) parse the message body, extract one element, then hand it off to be parsed again. It seems possible, but ugly and contrary to the clean architecture he has applied to it. Instead (respectably), he has engaged the GitLab devs and they appear to be patching it.

The downside is that the new 1.2.x version of this plugin does not appear as though it will be compatible with builds of the GitLab Jenkins CI service proceeding the GitLab patch.

@coder-hugo, I hope I'm interpreting this correctly... if not please correct me. :) Will you just be raising the minimum required version of GitLab for v1.2+ of this plugin?

@coder-hugo
Copy link
Contributor

Looks like there is a lot of confusion about this issue so here are some insights:
A few months ago I became the new lead maintainer of this plugin and one of my first major tasks was to cleanup the code base which previously was a real mess. During this cleanup I've rewritten about 90% of the whole plugin. These changes came to you with the version 1.2.0 of the plugin which was release on the 25th of April.
One thing that was changed by the cleanup is the "routing" of the incoming requests from GitLab. The code doing this now looks for the X-GitLab-Event header of the webhooks to distinguish between the different web hook types. This was previously done by parsing the incoming JSON twice. Once for getting the value of object_kind and the second time to map it to the corresponding POJO (PushHook, MergeRequestHook). The X-GitLab-Event header is a documented feature of the webhooks since at least version 7.14 of GitLab (the oldest version I checked the documentation for). For the Jenkins CI service of GitLab EE there is currently a bug so that this header isn't sent by GitLab. If this header is missing for somebody who uses a normal webhook with GitLab CE the issue is related to his setup. There may be some component within the whole chain that strips the header which can't be fixed by this plugin

@crussell52
Copy link

@coder-hugo Thank you for working on this and thank you for taking the time to clarify. :)

It doesn't look like the GitLab Jenkins CI service (introduced in GitLab 8.3) has ever sent that header. So, if I understand correctly, this means admins will have to choose one of three options:

  1. Upgrade their GitLab to latest version (after the GitLab fix goes in)
  2. Use an older version of this plugin (e.g. 1.1.32)
  3. Integrate using GitLab webhooks instead of the GitLab Jenkins CI service

Is that accurate?

@coder-hugo
Copy link
Contributor

@crussell52 yes this is correct

@apazga
Copy link

apazga commented May 4, 2016

Thanks @coder-hugo for the detailed explanation.

I'm using now Gitlab CE 8.7.2 (just updated it) and captured the header from a Webhook, and it's ok like you said. So I'll re-try updating Gitlab Plugin to test it again, because using 1.1.32 worked and using 1.2.0 didn't (I also had to re-enable Gitlab Plugin in all jobs that use the plugin after updating from 1.1.32 to 1.2.0 due to the warning Jenkins shows like the screenshot I posted before).

Content-Type: application/json
X-Gitlab-Event: Push Hook
Connection: close
Host: myhost.blablaba.com
Content-Length: 2590

@demaniak
Copy link
Contributor

demaniak commented May 5, 2016

Just some feedback here:

  • Upgrading Gitlab is turning out to be problematic for us (internal problems).Still working on that.
  • Downgrading gitlab plugin to 1.1.32 resolved the issue (as others have actually mentioned I believe).

So the only logical conclusion I have for our particular case is that I must have been mistaken about gitlab plugin version up to 28 April 2016, and that somewhere after that, I must have allowed the plugin to upgrade.
Apologies for any confusion introduced, and thank you to @coder-hugo and others for assistance and patience!

@apazga
Copy link

apazga commented May 5, 2016

More feedback: After updating to plugin 1.2.1, Gitlab 8.7.2, it's working now.
The only problem is that (like the Jenkins warning said) I had to re-enable and reconfigure the Gitlab plugin check for each job.
It would be nice to support enabling this check from Configuration slicing plugin.
Thanks!

@omehegan
Copy link
Member

@coder-hugo should we update the documentation to explain this issue? It sounds like it will be a permanent problem for people using GitLab EE versions before the bug was fixed there.

@coder-hugo
Copy link
Contributor

@omehegan yes, we should update the README.

@timmolter
Copy link

Still doesn't work for me.

Jenkins 2.5
GitLab Enterprise Edition 8.8.0-rc1-ee
GitLab Plugin 1.2.2

Even if I re-enable and reconfigure like @apazga noted. Also since there have been GUI updates in both Gitlab and Jenkins, the README documentation is out of date, so it's tough to follow the instructions.

@omehegan
Copy link
Member

omehegan commented May 23, 2016

@timmolter this will be broken in GitLab EE until https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/374 is merged on their side.

Can you specify which parts of the README need updating? Especially for things that have changed on the plugin configuration side. I'm trying to keep on top of that.

@omehegan
Copy link
Member

I have referenced this issue in the README so that users will be aware of it. I don't see much benefit in keeping it open, given that the issue is on GitLab's side and we don't know when they will fix it.

@ZJvandeWeg
Copy link

This seems to be merged on our side @omehegan, should it be removed from the README?

@omehegan
Copy link
Member

I think the note is still relevant in case someone using an older GitLab EE runs into the problem, but I clarified in the README that the bug has been fixed.

@mahdit83
Copy link

@coder-hugo where i can get old version of Gitlab Plugin?

@mahdit83
Copy link

@omehegan what is best solution now?

@ghost
Copy link

ghost commented Jan 23, 2018

@mahdit83 old plugin versions are available here: https://updates.jenkins.io/download/plugins/gitlab-plugin/

But GitLab fixed this problem. You should probably upgrade if you are still on a version of GitLab that is almost two years old.

@mahdit83
Copy link

@omehegan I using online gitlab that Enterprise Edition 10.4.0-ee. and Jnkeins 2.103
Jnkins Gitlab plugin 1.5.2. finally I did it. I had to remove Gitlab plugin and Use Gitlab Hook plugin.

@rokx
Copy link

rokx commented Mar 29, 2018

Gitlab plugin for jenkins is fine. You have to enable "Allow requests to the local network from hooks and services" in Setting page in Gitlab with new updates.

capture

exceed-alae pushed a commit to exceed-alae/gitlab-plugin that referenced this issue May 20, 2022
exceed-alae pushed a commit to exceed-alae/gitlab-plugin that referenced this issue May 20, 2022
When the build is scheduled, add an extension that will go back and cancel queued builds, and abort running builds.
Fixes jenkinsci#272 and fixes jenkinsci#241
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests