mqtt: return error when a too large packet is decoded

[bagder]

No description provided.

[Copilot]

Pull request overview

This PR enhances error handling in the MQTT protocol implementation by modifying the mqtt_decode_len function to properly detect and report when a packet's variable-length encoding exceeds the MQTT specification's 4-byte limit. The function signature is changed from returning the decoded length directly to returning an error code, with the length passed via an output parameter.

Key changes:

• Refactored mqtt_decode_len to return an error code (0 for success, non-zero for error) instead of the decoded length value
• Added validation to reject packets that attempt to use more than 4 bytes for length encoding
• Updated the call site to properly handle the new error return value

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.