Receive long MQTT payload

[gaco79]

What does this implement/fix?

MQTT fails when large message is delivered. Most noticeable with large JSON messages where JSON parsing then fails.

MQTT payload larger than TCP buffer will cause AsyncMQTTClient to split payload contents into separate parts.

The current ESPHome implementation assumes the whole payload will be delivered in one go

esphome/esphome/components/mqtt/mqtt_client.cpp

Lines 26 to 31 in 0f151a8

	this->mqtt_client_.onMessage([this](char *topic, char *payload, AsyncMqttClientMessageProperties properties,
	size_t len, size_t index, size_t total) {
	std::string payload_s(payload, len);
	std::string topic_s(topic);
	this->on_message(topic_s, payload_s);
	});

This can be fixed by moving the payload_s variable to a class variable and only calling MQTTClientComponent::on_message(topic_s, payload_s) when the full payload is available.

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Configuration change (this will require users to update their yaml configuraiton files to keep working)

Test Environment

• ESP32
• ESP8266
• Windows
• Mac OS
• Linux

Example entry for config.yaml:

# Example config.yaml - any standard MQTT setup will work 
mqtt:
  broker: !secret mqtt_broker
  username: !secret mqtt_username
  password: !secret mqtt_password
  on_json_message:
    - topic: test/data/
      then:
        logger.log: "JSON received"

Example long JSON message that causes error available here: https://pastebin.com/3DvKVeQR

Logs:

[18:37:06][D][main:174]: JSON received
[18:37:26][W][json:054]: Parsing JSON failed.
[18:37:26][W][json:054]: Parsing JSON failed.

Explain your changes

ESPHome can't be used if large MQTT messages are sent. This has presented a problem in my project where large MQTT payloads are sent containing colour information for strings of neopixel lights arranged in a panel. The only alternative has been to chunk the MQTT payloads into smaller sections, which in turn causes problems with not being able to update the whole "frame" of pixels at one go.

Checklist:

• The code change is tested and works locally.
• Tests have been added to verify that the new code works (under tests/ folder).

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated in esphome-docs.

[blejdfist]

Looks good to me

[OttoWinter]

LGTM, left you one comment about the shrink_to_fit()

[OttoWinter]

Thanks 🎉