Web Hooks

[VygantasOBDeleven]

Describe the problem

The issue that I've encountered is due to the complexity of systems that integrates with the Weblate through the REST API. Right now we need to go through every single translation in the project to sync with the external system.

Describe the solution you'd like

The solution could be the implementation of WebHooks. Every hook could have a type which could be something like "Translation Added", "Translation Approved", "Suggestion Added", "Suggestions Accepted", etc. On every hook type, Weblate could call a configured endpoint with a specific payload of the change.

In addition to that, there could be a configurable retry mechanism and request batching.

[github-actions]

This issue has been added to the backlog. It is not scheduled on the Weblate roadmap, but it eventually might be implemented.

In case you need this feature soon, please consider helping or push it by funding the development.

[nijel]

Things to decide before we start implementing this:

• Should webhooks be configured at component, project, or both?
• What all should be configured for webhooks (GitHub has URL, payload type (form/JSON), secret, SSL verification, event selection)?
• Should we log deliveries?
• Do we want to use add-ons for this or implement webhooks standalone?

There is also https://github.com/WeblateOrg/fedora_messaging/ which is slightly related (it allows notification of Weblate events to external systems).

[ParthS007]

discussion dump

• Keep in mind:

  • It should be generic
  • Will be extended in future

• Implementation

  • Should webhooks be configured at component, project, or both?

    • Both
  • Hooks UI would be inspired from Github UI
    • check addons UI
  • There will be 1 task for scheduling the deliveries and there will be 1 task which will be doing delivery for the event.
  • Webhook Service: get the content and construct the json and trigger the webhook
    • check fedora messaging app
  • Final: Integrations app
    • Models: similar to addons
    • base.py
    • forms.py

• Step 1: Add the models.py of integrations app

[ParthS007]

Moving ahead on issue. 🤞

• Integration App

  • Integration (There will be json configuration to store the integration config.) <-> Integration Form
  • Integration web hook service
    • Gets the content of change, Schedule the delivery
    • Executing delivery of tasks to external service

• Flow to add integration:

  • Fill the Integration Form to watch for changes.

• Flow to update/delete integration:

  • Change in integration form - Triggers store_change in Integration - (Change/Remove) -> Integration config changed -> Integration service trigger -> get and process trigger data -> scheduling payload to external API which contains changed ACTION

[nijel]

#9937 made me think again about this. Maybe we don't really need a separate model for integrations, but use add-ons as base for implementing webhooks and others because it would be beneficial to have add-ons at project level as well.

In the end the major difference between add-ons and integrations would be that add-ons have hard-coded list of events, while it is possible to customize this with integrations. Also, add-ons are currently called synchronously (unless implemented in the add-on itself), while we want webhooks to be done asynchronously. But I don't see why integration could not be a BaseAddon subclass implementing the differences.

There is even an existing (simple) implementation for this at https://github.com/porscheinformatik/weblate-addon-webhook.

Can you please write down the pros and cons for both approaches?

[ParthS007]

Good thought, I worked on this and follows the overview of pros and cons

What is expected functionality?

• Integrations app: Notify when any event present in integration config is executed.

Path 1

• Implement Integration separately
  • Standalone app
  • Includes dynamic addition of events in config supported by a UI similar to Github UI
  • Async communication
• Pros
  • Dynamic Config
  • UI for config customization
  • Async
• Cons
  • Duplication of Code
  • existing code is not used

Path 2

• Implemented as a class inherited from BaseAddon

• AddOn is synchronous now and asynchronous part will be implemented as addition on top in inherited class.

• Hard Coded set of events

• Pros

  • Code reused and Duplication avoided.
  • Improvement of Addon module with async functionality.
  • Improvement of Addon module with config customization?

• Cons

  • Addon and Integration will be coupled tightly.
  • Do we need all functionalities of Addons?

• Is Integration a Addon to weblate? if yes, then it is a Pro else Con.

Let me know what you think and if I missed something. @nijel

[nijel]

Thanks for writing this down. I think using add-ons here is the way to go as that will make it more straightforward to configure it in a single place. Otherwise, there would be a lot of duplication between integrations and add-ons.

The good thing here is that it can be separated into independent steps:

• Project-wide installed add-ons #10051
• Site-wide add-ons #10052
• Add-on activity log #10053

For the actual webhook add-on I think we need the following:

• Add EVENT_CHANGE to add-ons and (empty) change_event method to the BaseAddon
• Add ChangeBaseAddonForm as a subclass of BaseAddonForm that allows selecting individual change actions to deliver
• Add task addon_change to addons/tasks.py that will do dispatching the task and filtering of the events
• Add @receiver(post_save, sender=Change) to addons/models.py that will just trigger the addon_change task in the background
• Add webhook add-on
  • Subscribing to EVENT_CHANGE
  • Using ChangeBaseAddonForm subclass for configuration
  • Hook delivery implemented as change_event method

Does that make sense?

[ParthS007]

Sounds good to me.

I am following the issues step wise and then will add the web hook part. Will ask more in the specific issues.

[nijel]

There is really no ordering in those issues, they can be implemented independently.

[git-pear]

Thank you for working on this (I can see your recent PR for #10051, #10052 and #10053).

Also with regard to #10054, there is a proposed activity in Fedora to develop a gateway from webhooks to Fedora Messaging (webhook2fedmsg):
This application will help us to lower our codebase. We wouldn’t need to maintain multiple applications service2fedmsg anymore, but everything that provides webhook could be managed by this application.

Do you think there is a room for some collaboration or coordination between you and Fedora, I guess this May (maybe sooner) to August?

I would like to see 'weblate fedora messaging' to be an 'external service pilot' of the webhook@fedmsg, considering this matter is open on both sides for several years.

[nijel]

Even if Fedora goes the webhook2fedmsg way, we will still aim to integrate Fedora messaging stuff as described in #10054 as there are users for that outside Fedora (AFAIR they use it as generic AMQP publisher).

Anyway, we're always open to collaborating.

[git-pear]

Michal Čihař píše v St 13. 03. 2024 v 00:53 -0700:

Even if Fedora goes the webhook2fedmsg way, we will still aim to integrate Fedora messaging stuff as described in #10054 as there are users for that outside Fedora

Ok, fine. I am looking forward to it.

 (AFAIR they use it as generic AMQP publisher).

Yes, as far as I understand it correctly.

Anyway, we're always open to collaborating.

We always did appreciate it. :) Thank you for your reply.

…

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>