External hooks support for notifying or validating changeset

[Yshayy]

Tweek publishing service is responsible for handling new git changes and publishing it to relevant services.

These mechanisms can be more generalized as:

1. Validation - check key payload is valid (format, compilation), check for circular dependencies between keys.
2. Publishing - bundle and upload keys and policies.

It can be useful to add support for running client code to support custom validation or integration with external keys.

For example, a team can create hook that allow changes to a specific key only in a specific time, or apply logic that enforce acknowledgement from external system when trying to open a specific feature flag.
Another example, is that when changing a key that has value of a url or external resource, an hook can check that this resource exist to prevent human errors and there are many other examples as well.

There are also the cases that a team want to be notified when there's a change to one of their keys, maybe in email, slack or similar tool.

We can investigate adding these types of hooks:

• ValidationHook -> run as part of validating a new change-set, can be invoked with the relevant changes, and can reject a change-set by returning an error code.
• NotificationHook -> run after validation, notifies a url with change-set details.
• MutationHook (?) -> probably should be done after the others or not at all since it can be dangerous, but the idea is the create a webhook that can mutate the changeset. (for example, adding meta data, formatting)

All hooks can be defined to only run on specific paths/changes. (we can use wild cards), for example:

| Path                         | Type         |  url 
|  *                           | notification |  http://some-external-service/hook1
|  manifests/path/to/key       | validation   |  http://some-external-service/hook2
|  implementations/*/module    | validation   |  http://some-external-service/hook3

[Yshayy]

Suggested workflow:

1. Adding a file of hooks, and rest api for adding, listing, and removing hooks.
2. Publishing read hook and see that changeset match relevant hook.
3. For each validation hook, the publishing service sends a http/s request in the validation process for each validation hook matching the changeset and wait for a 2xx response for a validation hook.
4. For each notification hook, the publishing trigger each notification hook after the commit is pushed to upstream successfully.

Some questions:

1. Would it be better to use a co-located file approach instead of one centralized file.
2. Do permission to write hooks are path level or global?
3. Do we allow invoking both http and https request
4. What's the policy for long validation hook, or unavailable validation hook, should it be customizable per hook registration.
5. Should we allow the format of hook protocol to be more flexible (not just webhook) to allow easier integration with other systems (such as slack/email).
6. Should we limit the hooks on several part of the repos, for example, hooks themselves or security policies?

[tomeresk]

@Yshayy A few questions/ideas:

1. Did you purposefully omit the rest api for updating a hook (and why?) or should we do that as well?

2. I'm not sure I understand Publishing read hook and see that changeset match relevant hook.

3. Do we allow any customization of the hook request? (http method, payload, headers, authentication/authorization etc.)

4. For notification hooks, WDYT about publishing the info to NATS and having a separate job for running the notification hooks? We don't really have to wait for the outcome of the notification hook since it won't change anything about the rest of the process either way

5. Regarding long/unavailable validation hooks, IMO an unavailable hook should count as a failure (why have the hook in the first place if its not important enough?) and timeout should be customizable per hook registration (perhaps with some upper limit?)

6. For hook protocol format we should probably have a hook type (e.g. web/slack/email) even if we only implement the web type initially. This should make adding the other types much easier later on.

Also, lets have a short chat about numbers 1,2,3,6 from your list of questions

[Yshayy]

1. I think there should be a REST api for writing and reading hooks, maybe something similar to policies api. (https://github.com/Soluto/tweek/blob/master/services/authoring/src/routes/policies.ts)
2. I meant 'publishing service' read the hooks, and see that the corresponding commit changeset has files updates that are matched by the hook path. (but maybe we should define matcher on 'keys' and not 'paths', to prevent creating a hook on policies/hooks files)
3. That's a good question, I think we need to find a good standard to match, I think the method is POST and the body should be content describing the changeset.
4. I think that all post-receive stuff happens asynchronously, so nats can be useful mainly if we want to delegate this responsibility to another service that handles these notifications, might be a bit premature at this point, but it could be a direction.
5. Sounds good. Maybe we can start with a fixed timeout and add the ability to change it if necessary
6. I agree. I would start with webhook as it's possible to create a hook that integrates with other systems and there various solutions for that. (or create serverless function)

[tomeresk]

Notification hooks were implemented:
Docs
PRs #1156 #1157 #1158 #1159 #1161