Skip to content

Truncate oversized Teams webhook payloads to prevent HTTP 413 errors#2160

Merged
michael-myaskovsky merged 4 commits intomasterfrom
devin/1773995331-truncate-teams-webhook-payload
Mar 20, 2026
Merged

Truncate oversized Teams webhook payloads to prevent HTTP 413 errors#2160
michael-myaskovsky merged 4 commits intomasterfrom
devin/1773995331-truncate-teams-webhook-payload

Conversation

@devin-ai-integration
Copy link
Contributor

@devin-ai-integration devin-ai-integration bot commented Mar 20, 2026

Summary

Fixes a production issue where the send-alerts Airflow task for Prendio was failing because the alert adaptive card payload exceeded the Microsoft Teams webhook size limit, resulting in an HTTP 413 error that bubbled up as CouldNotSendAlertsError and failed the entire alert flow.

The fix proactively measures the JSON payload size before sending. If it exceeds 27KB (just under Teams' ~28KB limit for Adaptive Cards), it progressively removes body items from the end of the card and appends a truncation notice, then sends the reduced payload. If even a single body item is too large, it falls back to a minimal card.

Key changes in teams_webhook.py:

  • Extract _build_payload() helper from send_adaptive_card() for reuse
  • Add _truncate_card() which progressively drops trailing body items until the payload fits
  • Add _minimal_card() fallback when truncation alone isn't enough (single oversized body item)
  • Add _truncation_notice_item() for the user-facing truncation message
  • Add proactive size check + truncation + warning log in send_adaptive_card()

Review & Testing Checklist for Human

  • 27KB threshold — confirm this is appropriate for both old-style (status 200) and new-style (status 202) Teams webhooks. The actual Teams limit may differ.
  • _minimal_card uses **card spread — non-body fields (e.g. actions if ever added) are preserved, which could theoretically keep the payload oversized. In practice format_adaptive_card() only produces small top-level fields (type, version, msteams), so this is low risk but worth noting.
  • No unit tests included — consider adding tests for _truncate_card covering: payload under limit (no truncation), payload over limit (truncation occurs), single-body-item fallback to minimal card, empty body.
  • Recommended test plan: Trigger a Teams alert with a payload known to exceed 28KB (e.g. a grouped alert with many items) and verify the truncated message is delivered successfully with the truncation notice visible.

Notes

  • Triggered by a Prendio production incident (send-alerts-76bf501d-...). The Teams destination 5f726761-... was returning HTTP error 413 inside a 200 response, which the existing len(response.text) > 1 check correctly caught but couldn't recover from.
  • The fix is proactive only (pre-send size check). If Teams rejects at a different threshold, the original error path still applies.

Link to Devin session: https://app.devin.ai/sessions/d746de1f2743489995cc5070957c91c6

When an adaptive card payload exceeds the Teams webhook size limit (~28KB),
progressively remove body items from the card and append a truncation notice.
This prevents the entire alert flow from failing when a single message is
too large for the Teams endpoint.

Co-Authored-By: Michael Myaskovsky <michael@elementary-data.com>
@devin-ai-integration
Copy link
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@github-actions
Copy link
Contributor

👋 @devin-ai-integration[bot]
Thank you for raising your pull request.
Please make sure to add tests and document all user-facing changes.
You can do this by editing the docs files in this pull request.

…k markdown for italic

Co-Authored-By: Michael Myaskovsky <michael@elementary-data.com>
@coderabbitai
Copy link

coderabbitai bot commented Mar 20, 2026

📝 Walkthrough

Walkthrough

The change adds a 27 KiB payload size limit to the Teams webhook integration, builds and measures the full webhook payload before sending, and implements iterative truncation of adaptive card body elements with a truncation notice and a minimal-card fallback when needed.

Changes

Cohort / File(s) Summary
Teams Webhook Payload Truncation
elementary/messages/messaging_integrations/teams_webhook.py
Added TEAMS_PAYLOAD_SIZE_LIMIT and json/typing imports. Replaced direct card post with _build_payload + serialization size check. Implemented _truncate_card (iteratively removes last body items and appends a truncation notice) and _minimal_card fallback. Adjusted send_adaptive_card flow to use these helpers and log warnings when truncation occurs.

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant WebhookModule as TeamsWebhook
  participant TeamsAPI as "Microsoft Teams\nWebhook API"

  Caller->>WebhookModule: send_adaptive_card(card)
  WebhookModule->>WebhookModule: _build_payload(card) -> payload
  WebhookModule->>WebhookModule: json.dumps(payload) -> size
  alt size <= TEAMS_PAYLOAD_SIZE_LIMIT
    WebhookModule->>TeamsAPI: POST payload
  else size > TEAMS_PAYLOAD_SIZE_LIMIT
    WebhookModule->>WebhookModule: _truncate_card(card) (loop remove last body)
    WebhookModule->>WebhookModule: if still too large -> _minimal_card()
    WebhookModule->>TeamsAPI: POST truncated/minimal payload
  end
  TeamsAPI-->>WebhookModule: 200/response
  WebhookModule-->>Caller: result
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

Poem

🐰 I hopped a payload, big and round,
Nibbled bits off from the end I found.
A tiny note I left behind,
"Trimmed with care" — a kindly sign.
Off it went, snug and sound. 🥕✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and specifically describes the main change: truncating oversized Teams webhook payloads to prevent HTTP 413 errors, which directly matches the core objective of fixing a production issue caused by oversized payloads.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch devin/1773995331-truncate-teams-webhook-payload
📝 Coding Plan
  • Generate coding plan for human review comments

Comment @coderabbitai help to get the list of available commands and usage tips.

Co-Authored-By: Michael Myaskovsky <michael@elementary-data.com>
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
elementary/messages/messaging_integrations/teams_webhook.py (1)

84-93: Consider using list unpacking instead of concatenation.

Static analysis flags body + [_truncation_notice_item()] on lines 85 and 90. Using unpacking is more idiomatic and slightly more efficient.

♻️ Proposed refactor
     while len(body) > 1:
-        payload = _build_payload({**card, "body": body + [_truncation_notice_item()]})
+        payload = _build_payload({**card, "body": [*body, _truncation_notice_item()]})
         if len(json.dumps(payload)) <= TEAMS_PAYLOAD_SIZE_LIMIT:
             break
         body.pop()

-    truncated = {**card, "body": body + [_truncation_notice_item()]}
+    truncated = {**card, "body": [*body, _truncation_notice_item()]}
     if len(json.dumps(_build_payload(truncated))) > TEAMS_PAYLOAD_SIZE_LIMIT:
         return _minimal_card(card)
     return truncated
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@elementary/messages/messaging_integrations/teams_webhook.py` around lines 84
- 93, The code repeatedly concatenates lists via body +
[_truncation_notice_item()] which static analysis flags; modify the two usages
inside the loop and when constructing truncated to use list unpacking instead
(e.g., [*body, _truncation_notice_item()]) so you still pass a new list into
_build_payload and json.dumps while avoiding the concatenation idiom; ensure
changes reference the same symbols (_build_payload, _truncation_notice_item,
body, truncated, TEAMS_PAYLOAD_SIZE_LIMIT, _minimal_card) and preserve the
existing logic and size checks.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@elementary/messages/messaging_integrations/teams_webhook.py`:
- Around line 84-93: The code repeatedly concatenates lists via body +
[_truncation_notice_item()] which static analysis flags; modify the two usages
inside the loop and when constructing truncated to use list unpacking instead
(e.g., [*body, _truncation_notice_item()]) so you still pass a new list into
_build_payload and json.dumps while avoiding the concatenation idiom; ensure
changes reference the same symbols (_build_payload, _truncation_notice_item,
body, truncated, TEAMS_PAYLOAD_SIZE_LIMIT, _minimal_card) and preserve the
existing logic and size checks.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a29959c0-4d77-41eb-afe8-82a2cefb6711

📥 Commits

Reviewing files that changed from the base of the PR and between 31e7528 and 59c6b7b.

📒 Files selected for processing (1)
  • elementary/messages/messaging_integrations/teams_webhook.py

return {
"type": "TextBlock",
"text": "_... Content truncated due to message size limits. "
"View full details in Elementary Cloud._",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this pr is in the package, not all package users have elementary cloud

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — removed the Elementary Cloud references from both truncation messages.

Co-Authored-By: Michael Myaskovsky <michael@elementary-data.com>
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@elementary/messages/messaging_integrations/teams_webhook.py`:
- Around line 63-91: The current _minimal_card preserves all top-level fields
from card so large non-body fields (e.g., actions) can still make the payload
exceed TEAMS_PAYLOAD_SIZE_LIMIT; change _minimal_card to construct a strict
minimal payload (only keys required by _build_payload / Teams adaptive card:
type, body with the truncation TextBlock, and any mandatory schema/version keys)
instead of spreading **card, and update _truncate_card (and the similar fallback
at lines 94–106) to validate the final payload size by calling _build_payload
and, if still too large, return this strict minimal payload; reference
_minimal_card, _truncate_card, _build_payload, _truncation_notice_item and
TEAMS_PAYLOAD_SIZE_LIMIT when making the change.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 9ed56b2f-2b2c-40d4-8a58-30684a965ef2

📥 Commits

Reviewing files that changed from the base of the PR and between 59c6b7b and 97195a2.

📒 Files selected for processing (1)
  • elementary/messages/messaging_integrations/teams_webhook.py

Comment on lines +63 to +91
def _minimal_card(card: dict) -> dict:
return {
**card,
"body": [
{
"type": "TextBlock",
"text": "Alert content too large to display in Teams.",
"wrap": True,
"weight": "bolder",
}
],
}


def _truncate_card(card: dict) -> dict:
body: List[Dict[str, Any]] = list(card.get("body", []))
if not body:
return card

while len(body) > 1:
payload = _build_payload({**card, "body": body + [_truncation_notice_item()]})
if len(json.dumps(payload)) <= TEAMS_PAYLOAD_SIZE_LIMIT:
break
body.pop()

truncated = {**card, "body": body + [_truncation_notice_item()]}
if len(json.dumps(_build_payload(truncated))) > TEAMS_PAYLOAD_SIZE_LIMIT:
return _minimal_card(card)
return truncated
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Minimal fallback can still exceed limit if non-body fields are large.

_minimal_card() preserves **card, so large top-level fields (for example actions) can keep the payload oversized. In that case, send_adaptive_card() can still post a >27KiB payload and hit 413 again.

Suggested hard-limit fallback
 def _minimal_card(card: dict) -> dict:
-    return {
-        **card,
-        "body": [
-            {
-                "type": "TextBlock",
-                "text": "Alert content too large to display in Teams.",
-                "wrap": True,
-                "weight": "bolder",
-            }
-        ],
-    }
+    # Keep only required/adaptive-card identity fields to guarantee small payload.
+    minimal = {
+        "type": card.get("type", "AdaptiveCard"),
+        "version": card.get("version", "1.4"),
+        "body": [
+            {
+                "type": "TextBlock",
+                "text": "Alert content too large to display in Teams.",
+                "wrap": True,
+                "weight": "bolder",
+            }
+        ],
+    }
+    if "$schema" in card:
+        minimal["$schema"] = card["$schema"]
+    return minimal
 def send_adaptive_card(webhook_url: str, card: dict) -> requests.Response:
@@
     if len(payload_json) > TEAMS_PAYLOAD_SIZE_LIMIT:
@@
         card = _truncate_card(card)
         payload = _build_payload(card)
+        if len(json.dumps(payload)) > TEAMS_PAYLOAD_SIZE_LIMIT:
+            raise MessagingIntegrationError(
+                "Teams webhook payload still exceeds size limit after truncation."
+            )

Also applies to: 94-106

🧰 Tools
🪛 Ruff (0.15.6)

[warning] 83-83: Consider [*body, _truncation_notice_item()] instead of concatenation

Replace with [*body, _truncation_notice_item()]

(RUF005)


[warning] 88-88: Consider [*body, _truncation_notice_item()] instead of concatenation

Replace with [*body, _truncation_notice_item()]

(RUF005)

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@elementary/messages/messaging_integrations/teams_webhook.py` around lines 63
- 91, The current _minimal_card preserves all top-level fields from card so
large non-body fields (e.g., actions) can still make the payload exceed
TEAMS_PAYLOAD_SIZE_LIMIT; change _minimal_card to construct a strict minimal
payload (only keys required by _build_payload / Teams adaptive card: type, body
with the truncation TextBlock, and any mandatory schema/version keys) instead of
spreading **card, and update _truncate_card (and the similar fallback at lines
94–106) to validate the final payload size by calling _build_payload and, if
still too large, return this strict minimal payload; reference _minimal_card,
_truncate_card, _build_payload, _truncation_notice_item and
TEAMS_PAYLOAD_SIZE_LIMIT when making the change.

@michael-myaskovsky michael-myaskovsky merged commit ed170b7 into master Mar 20, 2026
23 checks passed
@michael-myaskovsky michael-myaskovsky deleted the devin/1773995331-truncate-teams-webhook-payload branch March 20, 2026 09:17
GuyEshdat added a commit that referenced this pull request Mar 21, 2026
…1-truncate-teams-webhook-payload"

This reverts commit ed170b7, reversing
changes made to 31e7528.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants