EMQX Data Bridge Drop Packets

[rajubish]

What happened?

Hi,
We planning to implement the EMQX Data Integration and Ingesting the data using Webhook.
As per our requirement we want to process 30k message per minute. What we are observing that 25% in-queue message are going under dropped state which never comes back for reprocessing.
Can you please help to understand the root cause of message is being dropped.
I have attached the screenshot of process and dropped message stats.

What did you expect to happen?

All in-queued message should get processed successfully.

How can we reproduce it (as minimally and precisely as possible)?

No response

Anything else we need to know?

No response

EMQX version

$ ./bin/emqx_ctl broker
# paste output here
<img width="1347" alt="Screenshot 2023-05-16 at 7 28 32 PM" src="https://github.com/emqx/emqx/assets/74891056/6696dde0-53c4-4965-8d76-de5406109b06">

OS version

Ubuntu 20.04

Log files

[keynslug]

Hi @rajubish,

On first look, these metrics suggest that remote webhook processing server does not handle that amount of messages well. Also, dropped messages are not eligible for retries, usually because they're either expired or did not fit into the queue buffer.

Otherwise, I'd need more information to tell what's going on, e.g. EMQX version, log file contents, etc.

[rajubish]

Hi @keynslug ,

Thanks for your response.
Even though remote webhook server processing not able to handle the amount of messages, it should be in queue buffer instead of dropped. In some scenarios server response may take time due to high traffic.

EMQX Version : 5.0.21

Logs

2023-05-16T13:49:27.234479+00:00 [error] msg: http_connector_do_request_failed, mfa: emqx_connector_http:on_query/3, line: 322, connector: <<"bridge:webhook:agent_update_channel">>, reason: normal, request: {<<"/api/sync">>,[{<<"content-type">>,<<"application/json">>}],<<31,139,8,0,217,137,99,100,2,255,171,86,202,75,204,77,85,178,82,80,242,72,205,201,201,87,40,207,47,202,73,209,83,170,5,0,145,197,86,198,24,0,0,0>>} 2023-05-16T13:49:27.234767+00:00 [error] msg: http_connector_do_request_failed, mfa: emqx_connector_http:on_query/3, line: 322, connector: <<"bridge:webhook:agent_update_channel">>, reason: normal, request: {<<"/api/sync">>,[{<<"content-type">>,<<"application/json">>}],<<31,139,8,0,219,137,99,100,2,255,171,86,202,75,204,77,85,178,82,80,242,72,205,201,201,87,40,207,47,202,73,209,83,170,5,0,145,197,86,198,24,0,0,0>>} 2023-05-16T13:49:27.235360+00:00 [error] msg: unrecoverable_error, mfa: emqx_resource_buffer_worker:handle_query_result_pure/3, line: 834, id: <<"bridge:webhook:agent_update_channel">>, reason: normal 2023-05-16T13:49:27.235335+00:00 [error] msg: unrecoverable_error, mfa: emqx_resource_buffer_worker:handle_query_result_pure/3, line: 834, id: <<"bridge:webhook:agent_update_channel">>, reason: normal 2023-05-16T13:49:32.332652+00:00 [error] msg: http_connector_do_request_failed, mfa: emqx_connector_http:on_query/3, line: 322, connector: <<"bridge:webhook:agent_update_channel">>, reason: normal, request: {<<"/api/sync">>,[{<<"content-type">>,<<"application/json">>}],<<31,139,8,0,223,137,99,100,2,255,171,86,202,75,204,77,85,178,82,80,242,72,205,201,201,87,40,207,47,202,73,209,83,170,5,0,145,197,86,198,24,0,0,0>>} 2023-05-16T13:49:32.332878+00:00 [error] msg: unrecoverable_error, mfa: emqx_resource_buffer_worker:handle_query_result_pure/3, line: 834, id: <<"bridge:webhook:agent_update_channel">>, reason: normal 2023-05-16T13:49:46.191668+00:00 [error] msg: http_connector_do_request_failed, mfa: emqx_connector_http:on_query/3, line: 322, connector: <<"bridge:webhook:agent_update_channel">>, reason: normal, request: {<<"/api/sync">>,[{<<"content-type">>,<<"application/json">>}],<<31,139,8,0,227,137,99,100,2,255,171,86,202,75,204,77,85,178,82,80,242,72,205,201,201,87,40,207,47,202,73,209,83,170,5,0,145,197,86,198,24,0,0,0>>} 2023-05-16T13:49:46.191881+00:00 [error] msg: unrecoverable_error, mfa: emqx_resource_buffer_worker:handle_query_result_pure/3, line: 834, id: <<"bridge:webhook:agent_update_channel">>, reason: normal 2023-05-16T13:50:28.268131+00:00 [error] msg: http_connector_do_request_failed, mfa: emqx_connector_http:on_query/3, line: 322, connector: <<"bridge:webhook:agent_update_channel">>, reason: normal, request: {<<"/api/sync">>,[{<<"content-type">>,<<"application/json">>}],<<31,139,8,0,231,137,99,100,2,255,171,86,202,75,204,77,85,178,82,80,242,72,205,201,201,87,40,207,47,202,73,209,83,170,5,0,145,197,86,198,24,0,0,0>>} 2023-05-16T13:50:28.268366+00:00 [error] msg: unrecoverable_error, mfa: emqx_resource_buffer_worker:handle_query_result_pure/3, line: 834, id: <<"bridge:webhook:agent_update_channel">>, reason: normal 2023-05-16T13:50:43.894272+00:00 [error] msg: http_connector_do_request_failed, mfa: emqx_connector_http:on_query/3, line: 322, connector: <<"bridge:webhook:agent_update_channel">>, reason: normal, request: {<<"/api/sync">>,[{<<"content-type">>,<<"application/json">>}],<<31,139,8,0,30,138,99,100,2,255,171,86,202,75,204,77,85,178,82,80,242,72,205,201,201,87,40,207,47,202,73,209,83,170,5,0,145,197,86,198,24,0,0,0>>} 2023-05-16T13:50:43.894509+00:00 [error] msg: unrecoverable_error, mfa: emqx_resource_buffer_worker:handle_query_result_pure/3, line: 834, id: <<"bridge:webhook:agent_update_channel">>, reason: normal 2023-05-16T13:50:51.881637+00:00 [error] msg: http_connector_do_request_failed, mfa: emqx_connector_http:on_query/3, line: 322, connector: <<"bridge:webhook:agent_update_channel">>, reason: normal, request: {<<"/api/sync">>,[{<<"content-type">>,<<"application/json">>}],<<31,139,8,0,255,137,99,100,2,255,171,86,202,75,204,77,85,178,82,80,242,72,205,201,201,87,40,207,47,202,73,209,83,170,5,0,145,197,86,198,24,0,0,0>>} 2023-05-16T13:50:51.882708+00:00 [error] msg: unrecoverable_error, mfa: emqx_resource_buffer_worker:handle_query_result_pure/3, line: 834, id: <<"bridge:webhook:agent_update_channel">>, reason: normal 2023-05-16T13:50:56.808349+00:00 [error] msg: http_connector_do_request_failed, mfa: emqx_connector_http:on_query/3, line: 322, connector: <<"bridge:webhook:agent_update_channel">>, reason: normal, request: {<<"/api/sync">>,[{<<"content-type">>,<<"application/json">>}],<<31,139,8,0,34,138,99,100,2,255,171,86,202,75,204,77,85,178,82,80,242,72,205,201,201,87,40,207,47,202,73,209,83,170,5,0,145,197,86,198,24,0,0,0>>} 2023-05-16T13:50:56.808600+00:00 [error] msg: unrecoverable_error, mfa: emqx_resource_buffer_worker:handle_query_result_pure/3, line: 834, id: <<"bridge:webhook:agent_update_channel">>, reason: normal 2023-05-16T13:51:24.396142+00:00 [error] msg: http_connector_do_request_failed, mfa: emqx_connector_http:on_query/3, line: 322, connector: <<"bridge:webhook:agent_update_channel">>, reason: normal, request: {<<"/api/sync">>,[{<<"content-type">>,<<"application/json">>}],<<31,139,8,0,27,138,99,100,2,255,171,86,202,75,204,77,85,178,82,80,242,72,205,201,201,87,40,207,47,202,73,209,83,170,5,0,145,197,86,198,24,0,0,0>>} 2023-05-16T13:51:24.396350+00:00 [error] msg: unrecoverable_error, mfa: emqx_resource_buffer_worker:handle_query_result_pure/3, line: 834, id: <<"bridge:webhook:agent_update_channel">>, reason: normal

[keynslug]

Looking at the logs, seems that you might find this fix useful. It will be included in the next release.

As for the message drops, you could look into either increasing the buffer queue size for the bridge, request timeout (called expiry in recent versions of dashboard), or both. These options should be configurable through the dashboard.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[rajubish]

Hi @keynslug ,

I have tried with increasing the buffer size and request timeout but still same issue.
Is there any other config parameter needs to check?
If I use subscription listener then all messages are processing successfully, so I am wondering why Webhook data bridge dropping the messages.
Can you please help and provide the solution for it.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.