fix: body payload of MEETING_STARTED & MEETING_ENDED webhook #15036

hussamkhatib · 2024-05-14T21:49:36Z

What does this PR do?

The PR now provides the proper body in the webhook payload for scheduled Triggers (MEETING_STARTED & MEETING_ENDED)

Mandatory Tasks (DO NOT REMOVE)

I have self-reviewed the code (A decent size PR without self-review might be rejected)
I have added a Docs issue here if this PR makes changes that would require a documentation change
I have added or modified automated tests that prove my fix is effective or that my feature works (PRs might be rejected if logical changes are not properly tested)

How should this be tested?

Follow steps to setup webhooks in your applciation from the docs
Adding logs to print (req.body.payload, req.body)
Trigger the MEETING_START or MEETING_ENDED event
You should find req.body.payload will be undefined and req.body having the contents of payload which isn't the format mentioned in the docs, neither do other events have.
What are the minimal test data to have?
Booking a event creates the necessary records in the database, for ease of testing set buffer time to book to 0 and booking interval to 1 min, so that MEETING_STARTED & MEETING_ENDED will be trigerred faster.

Checklist

I haven't checked if my changes generate no new warnings

vercel · 2024-05-14T21:49:41Z

@hussamkhatib is attempting to deploy a commit to the cal Team on Vercel.

A member of the Team first needs to authorize it.

github-actions · 2024-05-14T21:49:48Z

Thank you for following the naming conventions! 🙏 Feel free to join our discord and post your PR link.

graphite-app · 2024-05-14T21:50:11Z

Graphite Automations

"Add community label" took an action on this PR • (05/14/24)

1 label was added to this PR based on Keith Williams's automation.

"Add consumer team as reviewer" took an action on this PR • (05/14/24)

1 reviewer was added to this PR based on Keith Williams's automation.

github-actions · 2024-05-14T21:57:53Z

📦 Next.js Bundle Analysis for @calcom/web

This analysis was generated by the Next.js Bundle Analysis action. 🤖

This PR introduced no changes to the JavaScript bundle! 🙌

packages/features/webhooks/lib/handleWebhookScheduledTriggers.ts

Udit-takkar · 2024-05-15T07:06:15Z

The unit tests are failling

hussamkhatib · 2024-05-15T13:01:00Z

apps/web/test/utils/bookingScenario/expects.ts

@@ -295,8 +295,6 @@ export function expectWebhookToHaveBeenCalledWith(
      }
      if (data.payload.responses !== undefined)
        expect(parsedBody.payload.responses).toEqual(expect.objectContaining(data.payload.responses));
-      const { responses: _1, metadata: _2, ...remainingPayload } = data.payload;
-      expect(parsedBody.payload).toEqual(expect.objectContaining(remainingPayload));


Compare the 2 snippets at the end of the Issue for why this is not needed.

CarinaWolli

I’ve added some comments. I’m also concerned that these changes will break existing workflows for users, as fields won't be accessible anymore as they used to

CarinaWolli · 2024-05-15T14:21:51Z

apps/web/test/utils/bookingScenario/expects.ts

@@ -295,8 +295,6 @@ export function expectWebhookToHaveBeenCalledWith(
      }
      if (data.payload.responses !== undefined)
        expect(parsedBody.payload.responses).toEqual(expect.objectContaining(data.payload.responses));
-      const { responses: _1, metadata: _2, ...remainingPayload } = data.payload;
-      expect(parsedBody.payload).toEqual(expect.objectContaining(remainingPayload));


let's fix the test instead of deleting it, this function is also used for other tests were it passes

packages/features/webhooks/lib/handleWebhookScheduledTriggers.ts

CarinaWolli · 2024-05-16T20:09:40Z

I’m also concerned that these changes will break existing workflows for users, as fields won't be accessible anymore as they used to

To make sure we don't break existing workflows for users, we decided to keep all fields as they are right now but additionally add the missing fields payload and createdAt

hussamkhatib · 2024-05-16T20:26:45Z

I’m also concerned that these changes will break existing workflows for users, as fields won't be accessible anymore as they used to

To make sure we don't break existing workflows for users, we decided to keep all fields as they are right now but additionally add the missing fields payload and createdAt

Sounds good @CarinaWolli, So could I go ahead without using sendPayload as using that won't allow to handle existing workflow for users

CarinaWolli · 2024-05-17T13:20:14Z

Sounds good @CarinaWolli, So could I go ahead without using sendPayload as using that won't allow to handle existing workflow for users

Yes we can do that for now 👍

rereview please

PeerRich · 2024-07-31T12:36:59Z

/bonus 20

@hussamkhatib would you be down to look into the merge conflict?

algora-pbc · 2024-07-31T12:37:07Z

A bonus of $20 has been added by PeerRich.
@hussamkhatib: You will receive $20 once you implement the requested changes and your PR is merged.

zomars · 2024-07-31T16:30:07Z

Since both the payload varied significantly, I made a migration so that legacyPayload and payload is stored separately.

Alternatives i considered

Combine and store in the same payload: The payload usually has more data in it than what is mentioned in the types defined in the source code, moreover it would be very hard to pick the keys for each of the payload.

Store in the same Object but separate out with keys: ({legacy: ..., payload: ...}) This is similar to the approach I took, I din't proceed this as I thought this would be harder to differentiate old and new records and generally wouldn't be clean.

the migration payload is copied over to legacyPayload and payload is set to '' to make it a falsy value. All new records will support both actual and legacy payload. The drawback is that It might confuse users If they encounter a old and new webhook, the old webhook will not have payload in the payload key, whereas the new webhooks will have payload in the payload key as well as the old payload directly in req.body

Also, do you have idea how can I approach the unit tests failing ?

@hussamkhatib care to give more details on why a migration is need in the first place? What are the actual differences between payload and legacyPayload? Both system and DB wise.

CarinaWolli · 2024-08-01T07:11:37Z

packages/features/webhooks/lib/scheduleTrigger.ts

+        const { booking: bookingWithWebhookPayload, evt } = await getBooking(booking.id);
+        const webhookData = getWebhookPayloadForBooking({
+          booking: bookingWithWebhookPayload,
+          evt,
+        });


Can we move the fetching of the booking and creation of the webhookData outside addedEventTriggers.map?

Sure, thanks for catching that.

…ide addedEventTriggers.map

hussamkhatib · 2024-08-11T21:41:22Z

Since both the payload varied significantly, I made a migration so that legacyPayload and payload is stored separately.
Alternatives i considered

Combine and store in the same payload: The payload usually has more data in it than what is mentioned in the types defined in the source code, moreover it would be very hard to pick the keys for each of the payload.

Store in the same Object but separate out with keys: ({legacy: ..., payload: ...}) This is similar to the approach I took, I din't proceed this as I thought this would be harder to differentiate old and new records and generally wouldn't be clean.

the migration payload is copied over to legacyPayload and payload is set to '' to make it a falsy value. All new records will support both actual and legacy payload. The drawback is that It might confuse users If they encounter a old and new webhook, the old webhook will not have payload in the payload key, whereas the new webhooks will have payload in the payload key as well as the old payload directly in req.body
Also, do you have idea how can I approach the unit tests failing ?

@hussamkhatib care to give more details on why a migration is need in the first place? What are the actual differences between payload and legacyPayload? Both system and DB wise.

You can find example of legacy payload and payload below.
Reason for creating a migration was solely for storing the 2 payloads separately.

Compare the responses Object below, the legacy payload has a flatter Object compared to what's shown in the docs

After this change user should receive payload in the below format

createdAt: Date
triggerEvent: TriggerEvent
payload: { new payload } 
...legacyPayload

#15036 (comment)

Example of legacy payload

{
  "triggerEvent": "MEETING_ENDED",
  "id": 40,
  "uid": "cZQKhWVNzyX6EkuFPZwwRU",
  "idempotencyKey": "97d7cb03-5d7f-5713-ae8f-1ed776321bb2",
  "userId": 6,
  "userPrimaryEmail": "free@example.com",
  "eventTypeId": 1131,
  "title": "1min between Free Example and Hussam",
  "description": "",
  "customInputs": {},
  "responses": { "name": "Hussam", "email": "hussam@shippercrm.com", "guests": [] },
  "startTime": "2024-08-11T20:54:14.000Z",
  "endTime": "2024-08-11T20:55:14.000Z",
  "location": "integrations:daily",
  "createdAt": "2024-08-11T20:51:41.424Z",
  "updatedAt": null,
  "status": "ACCEPTED",
  "paid": false,
  "destinationCalendarId": null,
  "cancellationReason": null,
  "rejectionReason": null,
  "dynamicEventSlugRef": null,
  "dynamicGroupSlugRef": null,
  "rescheduled": null,
  "fromReschedule": null,
  "recurringEventId": null,
  "smsReminderNumber": null,
  "scheduledJobs": [],
  "metadata": {},
  "isRecorded": false,
  "iCalUID": "cZQKhWVNzyX6EkuFPZwwRU@Cal.com",
  "iCalSequence": 0,
  "rating": null,
  "ratingFeedback": null,
  "noShowHost": false,
  "user": {
    "email": "free@example.com",
    "name": "Free Example",
    "timeZone": "Asia/Dubai",
    "username": "free"
  },
  "attendees": [
    {
      "id": 40,
      "email": "hussam@shippercrm.com",
      "name": "Hussam",
      "timeZone": "Asia/Dubai",
      "locale": "en",
      "bookingId": 40,
      "noShow": false
    }
  ],
  "payment": [],
  "references": []
}

Example of payload

{
  "bookerUrl": "http://localhost:3001",
  "type": "1min",
  "title": "1min between Free Example and Hussam",
  "description": "",
  "additionalNotes": "",
  "customInputs": {},
  "startTime": "2024-08-11T20:54:14Z",
  "endTime": "2024-08-11T20:55:14Z",
  "organizer": {
    "id": 6,
    "name": "Free Example",
    "email": "free@example.com",
    "username": "free",
    "timeZone": "Asia/Dubai",
    "language": { "locale": "en" },
    "timeFormat": "h:mma"
  },
  "responses": {
    "name": { "label": "your_name", "value": "Hussam", "isHidden": false },
    "email": { "label": "email_address", "value": "hussam@shippercrm.com", "isHidden": false },
    "location": {
      "label": "location",
      "value": { "value": "integrations:daily", "optionValue": "" },
      "isHidden": false
    },
    "title": { "label": "what_is_this_meeting_about", "isHidden": true },
    "notes": { "label": "additional_notes", "isHidden": false },
    "guests": { "label": "additional_guests", "value": [], "isHidden": false },
    "rescheduleReason": { "label": "reason_for_reschedule", "isHidden": false }
  },
  "userFieldsResponses": {},
  "attendees": [
    {
      "email": "hussam@shippercrm.com",
      "name": "Hussam",
      "firstName": "",
      "lastName": "",
      "timeZone": "Asia/Dubai",
      "language": { "locale": "en" }
    }
  ],
  "location": "integrations:daily",
  "destinationCalendar": null,
  "hideCalendarNotes": false,
  "requiresConfirmation": false,
  "eventTypeId": 1131,
  "seatsShowAttendees": true,
  "seatsPerTimeSlot": null,
  "seatsShowAvailabilityCount": true,
  "schedulingType": null,
  "iCalUID": "cZQKhWVNzyX6EkuFPZwwRU@Cal.com",
  "iCalSequence": 0,
  "uid": "cZQKhWVNzyX6EkuFPZwwRU",
  "conferenceData": { "createRequest": { "requestId": "2db644eb-37a5-581a-99fa-ebe6ce513834" } },
  "eventTitle": "1min",
  "eventDescription": "",
  "price": 0,
  "currency": "usd",
  "length": 1,
  "bookingId": 40,
  "metadata": {},
  "status": "ACCEPTED"
}

github-actions · 2024-08-26T00:19:40Z

This PR is being marked as stale due to inactivity.

anikdhabal · 2024-10-04T06:59:11Z

@hussamkhatib could you pls fix the conflict

anikdhabal · 2024-10-22T10:55:15Z

Closing due to staleness.

graphite-app bot added the community Created by Linear-GitHub Sync label May 14, 2024

graphite-app bot requested a review from a team May 14, 2024 21:49

github-actions bot added docs area: docs, documentation, cal.com/docs webhooks area: webhooks, callback, webhook payload labels May 14, 2024

dosubot bot added the 🐛 bug Something isn't working label May 14, 2024

hussamkhatib changed the title ~~fix body payload of MEETING_STARTED & MEETING_ENDED webhook~~ fix: body payload of MEETING_STARTED & MEETING_ENDED webhook May 14, 2024

Udit-takkar reviewed May 15, 2024

View reviewed changes

packages/features/webhooks/lib/handleWebhookScheduledTriggers.ts Outdated Show resolved Hide resolved

Udit-takkar added this to the v4.1 milestone May 15, 2024

Udit-takkar requested a review from CarinaWolli May 15, 2024 07:03

hussamkhatib force-pushed the fix-scheduled-webhooks branch from d8cac17 to 9f45c2e Compare May 15, 2024 11:37

hussamkhatib commented May 15, 2024

View reviewed changes

hussamkhatib requested a review from Udit-takkar May 15, 2024 13:01

CarinaWolli requested changes May 15, 2024

View reviewed changes

keithwillcode modified the milestones: v4.1, v4.2 May 15, 2024

dosubot bot modified the milestone: v4.2 May 15, 2024

CarinaWolli mentioned this pull request May 16, 2024

[CAL-3752] Unreliable Triggering and Payload Inconsistencies in Calcom Webhooks #15078

Open

hussamkhatib force-pushed the fix-scheduled-webhooks branch from 14acdc8 to a00bede Compare May 17, 2024 19:30

github-actions bot added High priority Created by Linear-GitHub Sync Urgent Created by Linear-GitHub Sync labels May 17, 2024

hussamkhatib force-pushed the fix-scheduled-webhooks branch from a00bede to 4053f66 Compare May 21, 2024 11:24

github-actions bot added the Stale label Jul 2, 2024

Udit-takkar removed the Stale label Jul 16, 2024

dosubot bot removed this from the v4.4 milestone Jul 17, 2024

keithwillcode removed the Urgent Created by Linear-GitHub Sync label Jul 22, 2024

PeerRich modified the milestones: Community Only, v4.4 Jul 31, 2024

dosubot bot modified the milestones: v4.4, Community Only Jul 31, 2024

CarinaWolli reviewed Aug 1, 2024

View reviewed changes

hussamkhatib added 5 commits August 12, 2024 00:07

fix body payload of MEETING_STARTED & MEETING_ENDED webhook

4f00b0d

revert back headers

a71de5d

fix unit tests

2116354

fix payload of scheduledTriggers while ot breaking existing workflows

247c0c1

add payload conditionally as it was not part in legacy payloads

d32306e

hussamkhatib force-pushed the fix-scheduled-webhooks branch from 8c8379a to d32306e Compare August 11, 2024 20:14

move the fetching of the booking and creation of the webhookData outs…

bd9185a

…ide addedEventTriggers.map

hussamkhatib marked this pull request as draft August 11, 2024 21:23

fix grammer

10742ed

jonnyboyC mentioned this pull request Aug 21, 2024

[Cal.com] Handle case of empty webhook payload SkillArc-App/skillarc-server#488

Merged

github-actions bot added the Stale label Aug 26, 2024

anikdhabal closed this Oct 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: body payload of MEETING_STARTED & MEETING_ENDED webhook #15036

fix: body payload of MEETING_STARTED & MEETING_ENDED webhook #15036

hussamkhatib commented May 14, 2024 •

edited by CarinaWolli

Loading

vercel bot commented May 14, 2024

github-actions bot commented May 14, 2024 •

edited

Loading

graphite-app bot commented May 14, 2024

github-actions bot commented May 14, 2024 •

edited

Loading

Udit-takkar commented May 15, 2024

hussamkhatib May 15, 2024

CarinaWolli left a comment

CarinaWolli May 15, 2024

CarinaWolli commented May 16, 2024

hussamkhatib commented May 16, 2024

CarinaWolli commented May 17, 2024

PeerRich commented Jul 31, 2024

algora-pbc bot commented Jul 31, 2024

zomars commented Jul 31, 2024

CarinaWolli Aug 1, 2024

hussamkhatib Aug 11, 2024

hussamkhatib commented Aug 11, 2024 •

edited

Loading

github-actions bot commented Aug 26, 2024

anikdhabal commented Oct 4, 2024

anikdhabal commented Oct 22, 2024

fix: body payload of MEETING_STARTED & MEETING_ENDED webhook #15036

fix: body payload of MEETING_STARTED & MEETING_ENDED webhook #15036

Conversation

hussamkhatib commented May 14, 2024 • edited by CarinaWolli Loading

What does this PR do?

Mandatory Tasks (DO NOT REMOVE)

How should this be tested?

Checklist

vercel bot commented May 14, 2024

github-actions bot commented May 14, 2024 • edited Loading

graphite-app bot commented May 14, 2024

Graphite Automations

github-actions bot commented May 14, 2024 • edited Loading

📦 Next.js Bundle Analysis for @calcom/web

Udit-takkar commented May 15, 2024

hussamkhatib May 15, 2024

Choose a reason for hiding this comment

CarinaWolli left a comment

Choose a reason for hiding this comment

CarinaWolli May 15, 2024

Choose a reason for hiding this comment

CarinaWolli commented May 16, 2024

hussamkhatib commented May 16, 2024

CarinaWolli commented May 17, 2024

PeerRich commented Jul 31, 2024

algora-pbc bot commented Jul 31, 2024

zomars commented Jul 31, 2024

CarinaWolli Aug 1, 2024

Choose a reason for hiding this comment

hussamkhatib Aug 11, 2024

Choose a reason for hiding this comment

hussamkhatib commented Aug 11, 2024 • edited Loading

Example of legacy payload

Example of payload

github-actions bot commented Aug 26, 2024

anikdhabal commented Oct 4, 2024

anikdhabal commented Oct 22, 2024

hussamkhatib commented May 14, 2024 •

edited by CarinaWolli

Loading

github-actions bot commented May 14, 2024 •

edited

Loading

github-actions bot commented May 14, 2024 •

edited

Loading

hussamkhatib commented Aug 11, 2024 •

edited

Loading