Why Webhooks Matter?
It was 2016, and Paystack had just launched a few months ago, and they were all the rave among developers for their well-documented API and ease of use. I enjoyed reading their docs so much and became excited by the idea of easily adding payments to about just any idea I had. In fact, Paystack was foundational to many of the successes of startups in the Nigerian fintech space. From learning about their API, I wrote a series of blog posts explaining how to integrate them into applications. Doing this involved learning and writing about Webhooks. And I had this to say about Webhooks:
Little did I know that a few years down the line, I would be building Convoy, a webhooks gateway. Today when I speak to developers, what I hear oftentimes is quite shocking, especially from developers who haven’t had to build webhooks at scale. Some think it is simply an HTTP callback and doesn’t deserve the mysterious name: “webhook”. Some others do webhooks but pay little attention to making the developer experience great, so you end up with users who still find themselves polling providers with webhooks, and there’s a huge portion of developers who are hearing about webhooks for the first; to these sets of users, welcome! let me show you around the block.
I’ll start with the newcomers to the block. For a few years now, Postman has been championing the gospel of an API-First world. API-First Companies prioritise API development at the very beginning of their software development process. They see it as foundational building blocks for modern software. Even in the present market downturn, API development continues to receive consistent budgets across organisations of all sizes. The API-First era is akin to DevOps, where IT operations agility became critical to the survival of any business that wants to compete in this ever-changing technology-driven world. Similarly, the quicker organisations move from papers and folders to APIs, and automation would be critical to their survival. See what CEOs had to say about API Investments in their organisations:
The next generation of technology is centred around Productivity. AI is going to drive productivity across all industries to new heights formerly unfathomable. It should also not surprise you that ChatGPT was released not only as a product for the end-user but also as an API for endless possibilities, which we are only still in the early days of witnessing.
Put simply, an API is the synchronous mode of communication for two disparate systems to integrate. Webhooks are the asynchronous mode of communication for two disparate systems to integrate. Let’s take a quick example: You integrate Stripe for payments — you can make an API request to invoice a customer. The API responds with a 200, meaning the invoice was generated and sent successfully to the user. This doesn’t mean the user has made a payment, and it won’t be resourceful to leave an open connection waiting for the user to make the payment. Once the user makes payment hours or days later, Stripe sends a notification to your server for further processing. This notification is called a Webhook. You can easily see how this alleviates your integration process. I know, it’s so cool. That’s why I love it too. :)
Now let’s proceed to those who do webhooks but aren’t paying attention. If you’re going to succeed in the API-First Era, the developer experience is non-negotiable. If you’re going to put webhooks on your API, please do it well, or you might as well not do it at all. The first property of having a good experience is a webhooks system that is reliable, something that works, and we can rely on it. If it works today but doesn’t work tomorrow, there’s no point offering it in the first instance. Secondly, provide a webhook log; enable developers to debug webhooks delivery to their apps without reaching out to customer support. Of course, you should apply some form of retention policies to this data so you’re not saving lots of dummy data here for too long.
We released Nohooks last week, and a DigitalOcean user had this to say:
You can view the complete thread for context by following this link. Essentially, DigitalOcean provides a great API to provision resources but provides no webhooks. Even years after developers would still love to have webhooks to power certain workflows that would be much harder to create without great Webhooks. Polling is not a resourceful workflow for both the client and the server.
Lastly, to engineers that seem to think it’s only an HTTP post request. These engineers miss the point for several reasons. It is an HTTP post request with several bells and whistles. First, scalability, In my experience, I have spoken to engineers who have built webhooks at scale at companies like Datadog, Stripe and GitHub. And there’s one thing in common with all these engineers it’s simply not an HTTP post request. You can quickly run into several issues you never imagined would exist, such as DNS resolution limits, Head of Line blocking for zombie endpoints, endpoints response with massive payloads etc. You’ll then need an extensive design to manage client endpoints. Next, Security. This HN post captures very well the security problems you’re susceptible to when implementing a webhooks delivery system.
As your organisation gets bigger and bigger, you’d need to create a consistent abstraction for user teams across several microservices to rely on. This is where webhook gateways like Convoy come into the picture. With Convoy, you can abstract all your webhooks delivery needs into a service and serve the entire organisation.
Now that we’ve gone through the multiple criticisms of webhooks, I hope you’ll go forth and seek to learn more about webhooks and add great webhooks to your APIs.