Support for webhook integration with Pluggable SCM materials

[chadlwilson]

Issue Type

• Feature enhancement

Summary

Currently, GoCD supports incoming webhooks from a number of sources (GitHub, GitLab, Bitbucket Server + Cloud) all relating to Git materials, however my understanding is that it uses some heuristics to match the material and basically hard-codes the material search to filter for the inbuilt Git material.

There are a few SCM plugins in the ecosystem that also support Git, however I don't think there is a way for the webhooks to select these Git materials (and I'm not convinced there would be a way without an enhancement to the scm API that would allow

• the plugin to inform the server upfront that it handles git materials
• the server to inform the plugin of the webhook invocation metadata
• and, the plugin to inform the server whether it was a possible match

In particular, the Git Path Material plugin could benefit from this; as polling dozens of materials is expensive, and as noted here, can end up breaching API request limits.

Any other info

Even if such support existed, I'm not familiar enough with how material triggers work to know whether this would work with the git-path-material-plugin, since the plugin basically filters the git log when answering the latest-revision request.

Opening this for discussion/ideas about whether there might be a way to make this work.

[chadlwilson]

Seems to relate to #5328

[arvindsv]

@chadlwilson The webhook notification API endpoints are analogous to the notification API endpoints, with the only difference being the matching you mentioned.

The webhook notification code (around here and here for instance) is responsible for parsing the webhook payload, getting the parts related to the URL and then make guesses about the possible ways that the same URL could have been configured in GoCD, so that the material subsystem in GoCD can try and match all materials against those possible URLs and trigger pipelines related to those materials.

As you said, it does look like this will need a change to the SCM plugin API. The problem is the payload. If you look at the SCM plugin notification API endpoint, the body of the request needs to have the scm_name so that it can be used to match the material that will need to be polled (and so the pipelines that will need to be triggered).

With this case that you bring up, the SCM plugin will probably need to be given all the configurations (across the whole server) which match its type and it will be asked to make a determination about which materials match.

In an ideal world, if there was something in between (say a small server) which could translate the webhook payload to a valid SCM notification call, that would work. That can be a workaround for now, but the change you're mentioning looks like it will need a plugin API change, as you said.

[chadlwilson]

Thanks for the reply @arvindsv !

I spent some time looking at the codebase at the weekend and trying to figure out how I might be able to introduce such support, but couldn't quite find the right place because

• you need access to the ScmExtension to be able to ask a plugin a question via request/response, e.g "are you a possible match for this git url + branch?"
• the main points that own these interactions appear to be the MaterialService and PluggableScmService, neither of which have obvious interactions from the MaterialUpdateService which manages handling/matching these webhook interactions.

That's as far as I got before going down the thought process of "hmm, maybe as soon as the PluggableScmMaterialConfig is created/updated via a plugin interaction, the server should be able to ask it the relevant questions via the Scm API/config requests and store this as config metadata somehow, enough to build a Predicate on the material directly which can be invoked without a plugin interaction during the MaterialUpdateService#updateGitMaterial.

On the assumption that supporting this would be desirable, would you be able to help me understand what I might be getting into in submitting a draft PR for this by letting me know would I have to introduce a new Scm API version

• if I add a request/response type that is backward compatible from a server perspective (i.e if the plugin doesn't respond to the request type, life continues with the assumption that it's not matching)
• alternatively, how about if I introduced a backward compatible way to communicate this using the ScmProperty for the plugin to indicate with its config metadata that
  • property foo is a gitUrl
  • property bar is a gitBranch

[arvindsv]

@chadlwilson Of course! Happy to help. I'll think about this a bit more. I might not be the person with the most up-to-date information about the codebase. :) So, we might need to get some help from the others.

I'll reply once I've given this more thought.

[arvindsv]

@chadlwilson I thought about it. I'm going to write down some things here, which might help us have a discussion.

Assumption: We need to do something which is generic, which cannot be specific to one kind of plugin.

Thoughts:

1. Yes, you're right about ScmExtension. It's the entry point to all plugin communication. PluggableScmService is the service which wraps ScmExtension and can provide higher level services, while ScmExtension mirrors the documented plugin API.

2. From the perspective of GoCD services in the codebase, it is allowed for services to use one another. So, if we do need to introduce PluggableScmService as a dependency for MaterialUpdateService, that should be fine. It's not possible / allowed for services to access controllers, for example.

3. About the predicate idea: The idea of a predicate makes sense, but the creation of PluggableScmMaterialConfig might not be the right place. The reason is that PluggableScmMaterialConfig gets created with no interaction with the plugin. It's a representation of the configuration (the <scm> tag in the config XML). So, it can be created and recreated with no way to preserve the predicate across instances.

4. Probably the most appropriate place to store the predicate will be the SCMMetadataStore, which is available to the PluggableScmService. The SCMMetadataStore is used almost as a cache for information regarding different SCM plugins. That's why I think the predicate can belong there, because the predicate is not specific to the "SCM material", but is specific to the "SCM plugin". See ScmMetadataLoader.

5. As an aside: Plugin communication is not generally considered to be a performance bottleneck, since the plugin is loaded into the server process itself. So, what seems like an elaborate mechanism of conversation using serialization and deserialization of JSON messages starting from SCMExtension eventually leads to quick method calls across different Java classloaders, through the OSGi mechanism. However, getting a predicate will definitely be more performant.

6. One problem with the predicate idea: The predicate is a Java object, whereas the communication between the plugin and GoCD server happens using a serializable JSON message (as can be seen in the documented plugin API).

7. That brings up to one more decision we need to take: Our current assumption seems to be that GoCD server continues to be the one that parses the request body (from the webhook event) and extracts the URL, and then we need to somehow provide that URL to the plugin to match. However, to be truly generic, GoCD server should not be the one that parses the webhook event body. For instance, GoCD server would assume that the body is JSON. Then, it would assume that it knows which parts of that body are relevant to the plugin.

8. However, I am willing to put that aside, since GoCD's webhook notification endpoints assume git material, since they finally use MaterialUpdateService.updateGitMaterial. So, we can ignore that and reduce this problem to: "How do we ask the plugin whether a material of its type matches a given set of URLs".

9. Given point 6: Maybe, all we need from the plugin is for it to tell us which part of its configuration maps to branch and which part maps to URL. Then, GoCD server itself, say in PluggableScmService, would be able to do the matching. The equivalent of this piece of code.

10. As an example of point 9: Maybe this code in the git-path-material-plugin would need to change to send extra information somehow?

11. About backwards compatibility: The message handler mechanism allows us to easily introduce version 2.0 of the plugin API, and we can say that any plugin which doesn't implement version 2.0 does not support this behaviour and can return false when asked whether it matches. The ScmExtension will abstract this away.

cc: @maheshp @kritika-singh3 -- since my knowledge of the codebase is a little dated.

[maheshp]

My thoughts,

1. GoCD Corechanges

   • Extend webhook API to add support for pull requests. Currently the webhook API does not support pull request events, the API needs to be extended to add support for PR's.

2. SCM Plugin changes

   • Introduce Plugin Capabilities: This will help determine if the plugin can handle webhook events and event of a specific type. E.g GitHub PR plugin can say it handles GitHub pull request events.
   • Introduce a new plugin endpoint which given a webhook event and payload would match the materials which needs to be updated. Currently GoCD parses the webhook payload, based on the requirement we will have to decide on the information that needs to be sent to plugin. This JSON contract can definitely change later if the plugins require more information.

[arvindsv]

That seems neither here not there, right? Now the plugin API is tightly tied to something like "GitHub" and I can see that having to change all the time. If that's the case, we might as well send the whole JSON to the plugin and ask it to parse.

Maybe the notification endpoint can be something like: http://go-server:8153/go/api/webhooks/scm/notify?type=PLUGIN_ID and then, we gather up all the material configurations of that type and send it to the plugin, along with the full webhook payload and let it figure out which materials match.

At least that way plugin capabilities remains generic.

Problem with this: If a GoCD server has both a normal git material and a pluggable SCM material, then you will need to choose which URL to use to notify, and that decides which pipelines are triggered, which is a significant downside.

[maheshp]

After my conversation with @arvindsv we considered the below approach to implement this feature,

1. Bump up the SCM plugins version to introduce a new endpoint to fetch plugin capabilities, we have used capabilities for other plugins e.g Analytics Plugins capabilities.
   The capability exposed by the plugin can be something like,

{
    "supported_webhooks": [
        {
            "provider": "GitHub",
            "events": ["pull"]
        }
    ]
}

2. Introduce a new SCM plugin endpoint, which given a webhook event type, payload and a list of PluggableSCM materials(only materials which are configured for the plugin) matches and returns a list of materials which needs to be updated. (Essentially for a given webhook event, the plugin now decides which materials needs to be updated)

3. Changes in GoCD core: Since most of the SCM plugins support pull requests, the webhook endpoint needs to handle PR events. On receiving a webhook notification, based on the plugin capabilities GoCD will decide which plugin can handle the webhook event and interact with the same. GoCD should not parse the webhook payload and pass the entire data to the plugin.

4. The SCMExtension should handle the new version of the SCM plugin endpoint and expose appropriate methods. The ElasticAgentExtension can be used as reference to handle multiple versions of the plugin.

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 90 days.
If you can still reproduce this error on the master branch using local development environment or on the latest GoCD Release, please reply with all of the information you have about it in order to keep the issue open.
Thank you for all your contributions.

[chadlwilson]

I think would be good to keep this open, as a possible feature at some point. Unfortunately the complexity of implementing it is/was a bit beyond my available spare time to digest & invest/digest. Is there a way we can label issues as features so they don't get assumed to be stale/non-reproducible bugs by the bot?

[arvindsv]

This has been marked to be ignored by the stalebot, but I don't see much point in keeping something open forever. However, this one will remain open for now. :)

[kritika-singh3]

(long post) Carried out a quick spike for this with the following changes:

On GoCD server side

• Added a SCM extension v2. Added following 2 calls which will be made to the plugin.

  1. get-capabilities: On receiving this call, the plugin will return a list of webhooks it supports. Will be made post-successful plugin load. No request body.
     E.g. of the response body

     {
        "supported_webhooks": [
          {
            "provider": "GitHub",
            "events": ["push"]
          }
        ]
     }

  2. should-update: This will be made when GoCD receives a webhook event at the required endpoint, if there are materials configured for the plugin (with auto_update set to false).
     The plugin will receive this call if it supports the provider and the event for which the webhook notification was received.

     The plugin is expected to return the list of materials which needs to be updated. In case of no match, it should return empty list.

     E.g. request body

     {
       "provider": "Github",
       "event_type": "push",
       "event_payload": "{\"data\":\"complete-event-payload-data-without-parsing\"}",
       "materials": [
         {
            "id": "scm-id",
            "scm-configuration": {
              "url": {
                "value": "http://github.com/gocd/build_map"
              },
              "username": {
                "value": ""
              },
              "password": {
                "value": ""
              },
              "path": {
                "value": ""
              },
              "branch": {
                "value": "master"
              },
              "shallow_clone": {
                "value": ""
              }
            }
         }]
     }

     E.g. response body

     [
       {
         "id": "scm-id",
         "scm-configuration": {
           "url": "http://github.com/gocd/build_map",
           "username": "",
           "password": "",
           "path": "",
           "branch": "master",
           "shallow_clone": false
         }
       }
     ]

• Updated the webhook endpoint for Github to recognize PLUGIN_ID as query param.
  If present, the request payload was not parsed and the plugin for the given plugin id (if found), was notified. No validation for event is done.
  If absent, the current flow of parsing and matching with in-built materials is done. Event validation is done as only push and ping are supported.
  In case there are materials no update a 202 with OK message was send else 202 with No materials configured.
  In case of ping call, returned 202 without contacting the plugin.

Note:

• The webhook endpoint used was: /go/api/webhooks/github/notify?PLUGIN_ID=git-path
• For SCM extension v1: the capabilites was set to empty

On Plugin side

Tested the changes with gocd git path material plugin.

• Migrated plugin to support SCM extension v2.
• Implemented get-capabilities call to support Github provider with [push] event.
• Implemented should-update call:
  1. Parse the request body to identify event received.
  2. Based on the event, parsed the payload to get the repo name and branch. Used the logic of possible urls to match 'http/https/ssh/git@' based urls.
  3. Returned the matched materials to the server.

Questions to be discussed/decided upon

1. For the response body of should-update call: Should we expect a list of ids only as the scm-configuration is already present on the server side. For mapping it to the server objects, id is enough.

2. Currently, the endpoint will specify which plugin the webhook event should be passed to via the query param PLUGIN_ID.
   Take a scenario, wherein, the same repo is used

   • listen to main branch, for running security checks or such
   • to listen to particular folder via the gocd git path material plugin, for building modules separately
   • to monitor any pull requests via git pr plugin, for running unit, integrations and/or security tests
   • as config repo material, for providing pipeline configurations via pipeline as code.

   In this case,it the GoCD user wants to turn off polling, he/she would have to create 4 webhooks:

   • listen to push event with endpoint: /go/api/webhooks/github/notify
   • listen to push event for git path plugin with endpoint: /go/api/webhooks/github/notify?PLUGIN_ID=git-path
   • listen to pull_request event for git pr plugin with endpoint: /go/api/webhooks/github/notify?PLUGIN_ID=github.pr
   • list to [push,pull_request] for config repo material. WIP: Webhook triggers for configrepo branch support #8023

   Does this look ok or we should use a common endpoint?

Thoughts/Suggestions/Feedback welcomed!

[arvindsv]

1. I think ids should be enough. It should be simple enough to explain in the documentation.

2. Maybe using PLUGIN_IDS? I'm a little conflicted because it might not be a normal scenatio, but I can see it happening. Think about what it would take to change the behaviour later, if needed.

[maheshp]

@kritika-singh3 on second thoughts, if we can use the PLUGIN_ID why not use the ID of the PluggableSCM material in the query param. If that is an option, it would simplify lot of things for us. GoCD need not rely on the plugin to identify the material. It can trigger the material update for the material with the ID provided as part of the request.
Also, there would be no need for all the plugin changes.

Can we explore this option?

[kritika-singh3]

Can we think of a solution with the best of both options:
Provider: Github
Wehbook Endpoint: go/api/webhooks/github/notify

Flow on GoCD end:

1. Check for the presence of SCM_NAME. If present and event is a push event, trigger an MDU for the same. Also, we can supported multiple scms by allowing concatenating the param.
   E.g. go/api/webhooks/github/notify?SCM_NAME=abc&SCM_NAME=xyz

2. If absent, check for the PLUGIN_ID query param. If present, send the should-update call to the plugin based on it's capabilities and trigger an MDU for the results. Concatenation of params is allowed. No event validation is performed.
   E.g. go/api/webhooks/github/notify?PLUGIN_ID=p1&PLUGIN_ID=p2

3. If neither of the params are present, check if the event is a push event. If yes, then follow the current flow and trigger all the matched SCM materials. Additionally, send the should-update call to all the pluggable SCM plugins which supports the said event and trigger an MDU for all the results.

   P.S. this might be a time-consuming process. The provider might have timed-out.

4. For all the rest events, only the plugin flow will take place as GoCD does not understand them yet.

Thoughts/Reasoning involved:

@arvindsv's point that supporting multiple plugins might not be a normal scenario is right. Hence, instead of taking a list of ids in the same query param, preferred would be to allow the same param to be specified multiple times.

@maheshp 's point that we should be able to support pluggable SCMs without having to update the plugin is a valid point. Hence, supporting SCM_NAME. I would prefer using the name of the pluggable scm as it is the one which is configured by the user rather than using the id which is generated by the GoCD server.
This will allow supporting webhooks for a majority of cases for most of our users without having to update anything else other than GoCD server.

But there can be certain scenarios for which a plugin decision maybe required. For e.g. triggering when a tag with name release-* is created, or when a pull request is created/merged. In these cases, we can forward the request to a certain set of SCM plugins which will decide whether an MDU is required or not.
There might be a case where-in, the request needs to be forwarded to all the SCM plugins involved. Hence, point 4

Hope I was clear enought. LMK if you think this is an overkill solution.

[arvindsv]

My opinion: I think it would be better to keep it simple, and extend only if necessary later (meaning, multiple params).

[maheshp]

Maybe we should just start with adding support for material updates based on SCM_ID/SCM_NAME. Later based on the requirements we should add support for other things.

[kritika-singh3]

Ohk. Sounds good.
Will implement support for SCM_NAME first.

[maheshp]

Done as part of #8646

[chadlwilson]

Rather late to the party - but thank you for all the work on this @maheshp , @kritika-singh3, @arvindsv and @marques-work . Some appreciation for your work over at TWChennai/gocd-git-path-material-plugin#27 (comment) (although I haven't had the opportunity to try it out myself yet, sadly!)