Restored retained message have no payload

[neiledgar]

What happened?

After importing a previous export containing retained messages, all the retained messages reported an empty payload. The dashboard gave a 404 NOT_FOUND error. New MQTT clients did not receive any retained messages on subscription

Before import I cleared the retained messages in 3 ways. Each test had the same broken behaviour

1. Publish an empty, retained message
2. Clear all messages from the dashboard
3. Remove the persistent volume and re-initialise a new database

What did you expect to happen?

I expected the retained messages to be visible on the dashboard and delivered to any new MQTT client subscribing on that topic

How can we reproduce it (as minimally and precisely as possible)?

1. Populate some retained messages using an MQTT client
2. Backup using emqx ctl data export
3. Clear all retained messages using the EMQX dashboard
4. Restore using emqx ctl data import <File>
5. Verify using EMQX dashboard to view the payload of a restored message.

Anything else we need to know?

I managed to clear the fault by running the command

emqx ctl retainer reindex start

Starting reindexing
Reindexed 1000 messages
Reindexed 2000 messages
Reindexed 3000 messages
Reindexed 4000 messages
[snip]

Reindexing finished

After the command completed, I could view all retained messages on the dashboard and receive them on new MQTT client subscriptions

Perhaps this step is missing from the emqx ctl data import <File> command - or the Backup and Restore documentation should be updated to include this extra step. https://www.emqx.io/docs/en/latest/operations/backup-restore.html#cli-example

EMQX version

emqx-operator
emqx/emqx-operator-controller:2.2.21

emqx
emqx/emqx:5.6.1

$ ./bin/emqx_ctl broker
sysdescr  : EMQX
version   : 5.6.1
datetime  : 2024-05-10T08:57:33.011331190+00:00
uptime    : 15 minutes, 19 seconds

OS version

$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
$ uname -a
Linux emqx-core-5b7c9788f7-0 5.10.209-198.858.amzn2.x86_64 #1 SMP Tue Feb 13 18:46:41 UTC 2024 x86_64 GNU/Linux

Log files

Logs indicate a successful import

{"time":1715331224649834,"level":"debug","msg":"cluster_rpc_apply_result","tnx_id":18,"result":"{ok,#{config => #{},post_config_update => #{},raw_config => #{}}}","pid":"<0.2384.0>","kind":"initiate","entrypoint":"emqx:update_config/3"}
{"time":1715331224665468,"level":"debug","msg":"cluster_rpc_apply_result","tnx_id":19,"result":"{ok,#{config => #{},post_config_update => #{},raw_config => #{}}}","pid":"<0.2384.0>","kind":"initiate","entrypoint":"emqx:update_config/3"}
{"time":1715331224676261,"level":"debug","msg":"cluster_rpc_apply_result","tnx_id":20,"result":"{ok,#{config => #{},post_config_update => #{},raw_config => #{}}}","pid":"<0.2384.0>","kind":"initiate","entrypoint":"emqx:update_config/3"}
{"time":1715331224699652,"level":"debug","msg":"cluster_rpc_apply_result","tnx_id":21,"result":"{ok,#{config => #{ssl => #{default => #{acceptors => 16,access_rules => [\"allow all\"],authentication => [],bind => 8883,enable => false,enable_authn => true,max_connections => infinity,mountpoint => <<>>,proxy_protocol => false,proxy_protocol_timeout => 3000,ssl_options => #{cacertfile => <<\"${EMQX_ETC_DIR}/certs/cacert.pem\">>,certfile => <<\"${EMQX_ETC_DIR}/certs/cert.pem\">>,ciphers => [],client_renegotiation => true,depth => 10,enable_crl_check => false,fail_if_no_peer_cert => false,gc_after_handshake => false,handshake_timeout => 15000,hibernate_after => 5000,honor_cipher_order => true,keyfile => <<\"${EMQX_ETC_DIR}/certs/key.pem\">>,log_level => notice,ocsp => #{enable_ocsp_stapling => false,refresh_http_timeout => 15000,refresh_interval => 300000},reuse_sessions => true,secure_renegotiate => true,user_lookup_fun => {fun emqx_tls_psk:lookup/3,undefined},verify => verify_none,versions => ['tlsv1.3','tlsv1.2']},tcp_options => #{active_n => 100,backlog => 1024,buffer => 4096,high_watermark => 1048576,keepalive => \"none\",nodelay => true,reuseaddr => true,send_timeout => 15000,send_timeout_close => true},zone => default}},tcp => #{default => #{acceptors => 16,access_rules => [\"allow all\"],authentication => [],bind => 1883,enable => false,enable_authn => true,max_connections => infinity,mountpoint => <<>>,proxy_protocol => false,proxy_protocol_timeout => 3000,tcp_options => #{active_n => 100,backlog => 1024,buffer => 4096,high_watermark => 1048576,keepalive => \"none\",nodelay => true,reuseaddr => true,send_timeout => 15000,send_timeout_close => true},zone => default},external => #{acceptors => 64,access_rules => [\"allow all\"],authentication => [],bind => {{0,0,0,0},1883},enable => true,enable_authn => true,max_conn_rate => 100.0,max_connections => 1024000,mountpoint => <<>>,proxy_protocol => true,proxy_protocol_timeout => 3000,tcp_options => #{active_n => 100,backlog => 1024,buffer => 4096,high_watermark => 1048576,keepalive => \"none\",nodelay => true,reuseaddr => true,send_timeout => 15000,send_timeout_close => true},zone => default},internal => #{acceptors => 64,access_rules => [\"allow all\"],authentication => [],bind => {{0,0,0,0},11883},enable => true,enable_authn => true,max_connections => 1024000,mountpoint => <<>>,proxy_protocol => false,proxy_protocol_timeout => 3000,tcp_options => #{active_n => 100,backlog => 1024,buffer => 4096,high_watermark => 1048576,keepalive => \"none\",nodelay => true,reuseaddr => true,send_timeout => 15000,send_timeout_close => true},zone => default}},ws => #{default => #{acceptors => 16,access_rules => [\"allow all\"],authentication => [],bind => 8083,enable => false,enable_authn => true,max_connections => infinity,mountpoint => <<>>,proxy_protocol => false,proxy_protocol_timeout => 3000,tcp_options => #{active_n => 100,backlog => 1024,buffer => 4096,high_watermark => 1048576,keepalive => \"none\",nodelay => true,reuseaddr => true,send_timeout => 15000,send_timeout_close => true},websocket => #{allow_origin_absence => true,check_origin_enable => false,check_origins => [<<\"http://localhost:18083\">>,<<\"http://127.0.0.1:18083\">>],compress => false,deflate_opts => #{client_context_takeover => takeover,client_max_window_bits => 15,mem_level => 8,server_context_takeover => takeover,server_max_window_bits => 15,strategy => default},fail_if_no_subprotocol => true,idle_timeout => 7200000,max_frame_size => infinity,mqtt_path => \"/mqtt\",mqtt_piggyback => multiple,proxy_address_header => \"x-forwarded-for\",proxy_port_header => \"x-forwarded-port\",supported_subprotocols => [\"mqtt\",\"mqtt-v3\",\"mqtt-v3.1.1\",\"mqtt-v5\"],validate_utf8 => true},zone => default}},wss => #{default => #{acceptors => 16,access_rules => [\"allow all\"],authentication => [],bind => 8084,enable => false,enable_authn => true,max_connections => infinity,mountpoint => <<>>,proxy_protocol => false,proxy_protocol_timeout => 3000,ssl_options => #{cacertfile => <<\"${EMQX_ETC_DIR}/certs/cacert.pem\">>,certfile => <<\"${EMQX_ETC_DIR}/certs/cert.pem\">>,ciphers => [],client_renegotiation => true,depth => 10,fail_if_no_peer_cert => false,handshake_timeout => 15000,hibernate_after => 5000,honor_cipher_order => true,keyfile => <<\"${EMQX_ETC_DIR}/certs/key.pem\">>,log_level => notice,reuse_sessions => true,secure_renegotiate => true,user_lookup_fun => {fun emqx_tls_psk:lookup/3,undefined},verify => verify_none,versions => ['tlsv1.3','tlsv1.2']},tcp_options => #{active_n => 100,backlog => 1024,buffer => 4096,high_watermark => 1048576,keepalive => \"none\",nodelay => true,reuseaddr => true,send_timeout => 15000,send_timeout_close => true},websocket => #{allow_origin_absence => true,check_origin_enable => false,check_origins => [<<\"http://localhost:18083\">>,<<\"http://127.0.0.1:18083\">>],compress => false,deflate_opts => #{client_context_takeover => takeover,client_max_window_bits => 15,mem_level => 8,server_context_takeover => takeover,server_max_window_bits => 15,strategy => default},fail_if_no_subprotocol => true,idle_timeout => 7200000,max_frame_size => infinity,mqtt_path => \"/mqtt\",mqtt_piggyback => multiple,proxy_address_header => \"x-forwarded-for\",proxy_port_header => \"x-forwarded-port\",supported_subprotocols => [\"mqtt\",\"mqtt-v3\",\"mqtt-v3.1.1\",\"mqtt-v5\"],validate_utf8 => true},zone => default}}},post_config_update => #{},raw_config => #{<<\"ssl\">> => #{<<\"default\">> => #{<<\"acceptors\">> => 16,<<\"access_rules\">> => [<<\"allow all\">>],<<\"bind\">> => 8883,<<\"enable\">> => false,<<\"enable_authn\">> => true,<<\"max_connections\">> => <<\"infinity\">>,<<\"mountpoint\">> => <<>>,<<\"proxy_protocol\">> => false,<<\"proxy_protocol_timeout\">> => <<\"3s\">>,<<\"ssl_options\">> => #{<<\"cacertfile\">> => <<\"${EMQX_ETC_DIR}/certs/cacert.pem\">>,<<\"certfile\">> => <<\"${EMQX_ETC_DIR}/certs/cert.pem\">>,<<\"ciphers\">> => [],<<\"client_renegotiation\">> => true,<<\"depth\">> => 10,<<\"enable_crl_check\">> => false,<<\"fail_if_no_peer_cert\">> => false,<<\"gc_after_handshake\">> => false,<<\"handshake_timeout\">> => <<\"15s\">>,<<\"hibernate_after\">> => <<\"5s\">>,<<\"honor_cipher_order\">> => true,<<\"keyfile\">> => <<\"${EMQX_ETC_DIR}/certs/key.pem\">>,<<\"log_level\">> => <<\"notice\">>,<<\"ocsp\">> => #{<<\"enable_ocsp_stapling\">> => false,<<\"refresh_http_timeout\">> => <<\"15s\">>,<<\"refresh_interval\">> => <<\"5m\">>},<<\"reuse_sessions\">> => true,<<\"secure_renegotiate\">> => true,<<\"verify\">> => <<\"verify_none\">>,<<\"versions\">> => [<<\"tlsv1.3\">>,<<\"tlsv1.2\">>]},<<\"tcp_options\">> => #{<<\"active_n\">> => 100,<<\"backlog\">> => 1024,<<\"buffer\">> => <<\"4KB\">>,<<\"high_watermark\">> => <<\"1MB\">>,<<\"keepalive\">> => <<\"none\">>,<<\"nodelay\">> => true,<<\"reuseaddr\">> => true,<<\"send_timeout\">> => <<\"15s\">>,<<\"send_timeout_close\">> => true}}},<<\"tcp\">> => #{<<\"default\">> => #{<<\"acceptors\">> => 16,<<\"access_rules\">> => [<<\"allow all\">>],<<\"bind\">> => 1883,<<\"enable\">> => false,<<\"enable_authn\">> => true,<<\"max_connections\">> => <<\"infinity\">>,<<\"mountpoint\">> => <<>>,<<\"proxy_protocol\">> => false,<<\"proxy_protocol_timeout\">> => <<\"3s\">>,<<\"tcp_options\">> => #{<<\"active_n\">> => 100,<<\"backlog\">> => 1024,<<\"buffer\">> => <<\"4KB\">>,<<\"high_watermark\">> => <<\"1MB\">>,<<\"keepalive\">> => <<\"none\">>,<<\"nodelay\">> => true,<<\"reuseaddr\">> => true,<<\"send_timeout\">> => <<\"15s\">>,<<\"send_timeout_close\">> => true}},<<\"external\">> => #{<<\"acceptors\">> => 64,<<\"access_rules\">> => [<<\"allow all\">>],<<\"bind\">> => <<\"0.0.0.0:1883\">>,<<\"enable\">> => true,<<\"enable_authn\">> => true,<<\"max_conn_rate\">> => <<\"1000/s\">>,<<\"max_connections\">> => 1024000,<<\"mountpoint\">> => <<>>,<<\"proxy_protocol\">> => true,<<\"proxy_protocol_timeout\">> => <<\"3s\">>,<<\"tcp_options\">> => #{<<\"active_n\">> => 100,<<\"backlog\">> => 1024,<<\"buffer\">> => <<\"4KB\">>,<<\"high_watermark\">> => <<\"1MB\">>,<<\"keepalive\">> => <<\"none\">>,<<\"nodelay\">> => true,<<\"reuseaddr\">> => true,<<\"send_timeout\">> => <<\"15s\">>,<<\"send_timeout_close\">> => true}},<<\"internal\">> => #{<<\"acceptors\">> => 64,<<\"access_rules\">> => [<<\"allow all\">>],<<\"bind\">> => <<\"0.0.0.0:11883\">>,<<\"enable\">> => true,<<\"enable_authn\">> => true,<<\"max_connections\">> => 1024000,<<\"mountpoint\">> => <<>>,<<\"proxy_protocol\">> => false,<<\"proxy_protocol_timeout\">> => <<\"3s\">>,<<\"tcp_options\">> => #{<<\"active_n\">> => 100,<<\"backlog\">> => 1024,<<\"buffer\">> => <<\"4KB\">>,<<\"high_watermark\">> => <<\"1MB\">>,<<\"keepalive\">> => <<\"none\">>,<<\"nodelay\">> => true,<<\"reuseaddr\">> => true,<<\"send_timeout\">> => <<\"15s\">>,<<\"send_timeout_close\">> => true}}},<<\"ws\">> => #{<<\"default\">> => #{<<\"acceptors\">> => 16,<<\"access_rules\">> => [<<\"allow all\">>],<<\"bind\">> => 8083,<<\"enable\">> => false,<<\"enable_authn\">> => true,<<\"max_connections\">> => <<\"infinity\">>,<<\"mountpoint\">> => <<>>,<<\"proxy_protocol\">> => false,<<\"proxy_protocol_timeout\">> => <<\"3s\">>,<<\"tcp_options\">> => #{<<\"active_n\">> => 100,<<\"backlog\">> => 1024,<<\"buffer\">> => <<\"4KB\">>,<<\"high_watermark\">> => <<\"1MB\">>,<<\"keepalive\">> => <<\"none\">>,<<\"nodelay\">> => true,<<\"reuseaddr\">> => true,<<\"send_timeout\">> => <<\"15s\">>,<<\"send_timeout_close\">> => true},<<\"websocket\">> => #{<<\"allow_origin_absence\">> => true,<<\"check_origin_enable\">> => false,<<\"check_origins\">> => <<\"http://localhost:18083, http://127.0.0.1:18083\">>,<<\"compress\">> => false,<<\"deflate_opts\">> => #{<<\"client_context_takeover\">> => <<\"takeover\">>,<<\"client_max_window_bits\">> => 15,<<\"mem_level\">> => 8,<<\"server_context_takeover\">> => <<\"takeover\">>,<<\"server_max_window_bits\">> => 15,<<\"strategy\">> => <<\"default\">>},<<\"fail_if_no_subprotocol\">> => true,<<\"idle_timeout\">> => <<\"7200s\">>,<<\"max_frame_size\">> => <<\"infinity\">>,<<\"mqtt_path\">> => <<\"/mqtt\">>,<<\"mqtt_piggyback\">> => <<\"multiple\">>,<<\"proxy_address_header\">> => <<\"x-forwarded-for\">>,<<\"proxy_port_header\">> => <<\"x-forwarded-port\">>,<<\"supported_subprotocols\">> => <<\"mqtt, mqtt-v3, mqtt-v3.1.1, mqtt-v5\">>,<<\"validate_utf8\">> => true}}},<<\"wss\">> => #{<<\"default\">> => #{<<\"acceptors\">> => 16,<<\"access_rules\">> => [<<\"allow all\">>],<<\"bind\">> => 8084,<<\"enable\">> => false,<<\"enable_authn\">> => true,<<\"max_connections\">> => <<\"infinity\">>,<<\"mountpoint\">> => <<>>,<<\"proxy_protocol\">> => false,<<\"proxy_protocol_timeout\">> => <<\"3s\">>,<<\"ssl_options\">> => #{<<\"cacertfile\">> => <<\"${EMQX_ETC_DIR}/certs/cacert.pem\">>,<<\"certfile\">> => <<\"${EMQX_ETC_DIR}/certs/cert.pem\">>,<<\"ciphers\">> => [],<<\"client_renegotiation\">> => true,<<\"depth\">> => 10,<<\"fail_if_no_peer_cert\">> => false,<<\"handshake_timeout\">> => <<\"15s\">>,<<\"hibernate_after\">> => <<\"5s\">>,<<\"honor_cipher_order\">> => true,<<\"keyfile\">> => <<\"${EMQX_ETC_DIR}/certs/key.pem\">>,<<\"log_level\">> => <<\"notice\">>,<<\"reuse_sessions\">> => true,<<\"secure_renegotiate\">> => true,<<\"verify\">> => <<\"verify_none\">>,<<\"versions\">> => [<<\"tlsv1.3\">>,<<\"tlsv1.2\">>]},<<\"tcp_options\">> => #{<<\"active_n\">> => 100,<<\"backlog\">> => 1024,<<\"buffer\">> => <<\"4KB\">>,<<\"high_watermark\">> => <<\"1MB\">>,<<\"keepalive\">> => <<\"none\">>,<<\"nodelay\">> => true,<<\"reuseaddr\">> => true,<<\"send_timeout\">> => <<\"15s\">>,<<\"send_timeout_close\">> => true},<<\"websocket\">> => #{<<\"allow_origin_absence\">> => true,<<\"check_origin_enable\">> => false,<<\"check_origins\">> => <<\"http://localhost:18083, http://127.0.0.1:18083\">>,<<\"compress\">> => false,<<\"deflate_opts\">> => #{<<\"client_context_takeover\">> => <<\"takeover\">>,<<\"client_max_window_bits\">> => 15,<<\"mem_level\">> => 8,<<\"server_context_takeover\">> => <<\"takeover\">>,<<\"server_max_window_bits\">> => 15,<<\"strategy\">> => <<\"default\">>},<<\"fail_if_no_subprotocol\">> => true,<<\"idle_timeout\">> => <<\"7200s\">>,<<\"max_frame_size\">> => <<\"infinity\">>,<<\"mqtt_path\">> => <<\"/mqtt\">>,<<\"mqtt_piggyback\">> => <<\"multiple\">>,<<\"proxy_address_header\">> => <<\"x-forwarded-for\">>,<<\"proxy_port_header\">> => <<\"x-forwarded-port\">>,<<\"supported_subprotocols\">> => <<\"mqtt, mqtt-v3, mqtt-v3.1.1, mqtt-v5\">>,<<\"validate_utf8\">> => true}}}}}}","pid":"<0.2384.0>","kind":"initiate","entrypoint":"emqx:update_config/3"}
{"time":1715331224714292,"level":"debug","msg":"cluster_rpc_apply_result","tnx_id":22,"result":"{ok,#{config => #{},post_config_update => #{},raw_config => #{}}}","pid":"<0.2384.0>","kind":"initiate","entrypoint":"emqx:update_config/3"}
{"time":1715331224724848,"level":"debug","msg":"cluster_rpc_apply_result","tnx_id":23,"result":"{ok,#{config => #{},post_config_update => #{},raw_config => #{}}}","pid":"<0.2384.0>","kind":"initiate","entrypoint":"emqx:update_config/3"}
{"time":1715331224737343,"level":"debug","msg":"cluster_rpc_apply_result","tnx_id":24,"result":"{ok,#{config => [],post_config_update => #{},raw_config => []}}","pid":"<0.2384.0>","kind":"initiate","entrypoint":"emqx:update_config/3"}
{"time":1715331226808202,"level":"info","msg":"emqx_data_import_success","pid":"<0.4373.0>"}

[node_dump_20240510_090220.tar.gz](https://github.com/emqx/emqx/files/15273071/node_dump_20240510_090220.tar.gz)

[kjellwinblad]

Thanks for reporting this!

[id]

Thanks for the report, we've created internal issue to track this.