Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(emqx_rule_engine): set inc_action_metrics as async_reply_fun #11126

Merged
merged 10 commits into from Jun 30, 2023

Conversation

sstrigler
Copy link
Contributor

@sstrigler sstrigler commented Jun 22, 2023

Fixes EMQX-8842

Next try. This time using a callback.

Summary

馃 Generated by Copilot at 5c23ed7

Added support for asynchronous replies from bridge resources in rule engine and resource buffer. This allows bridge actions to update metrics and tracing data after sending or receiving messages. Modified emqx_bridge, emqx_resource_buffer_worker, and emqx_rule_runtime modules to implement this feature.

PR Checklist

Please convert it to a draft if any of the following conditions are not met. Reviewers may skip over until all the items are checked:

  • Added tests for the changes
  • Changed lines covered in coverage report
  • Change log has been added to changes/{ce,ee}/(feat|perf|fix)-<PR-id>.en.md files
  • For internal contributor: there is a jira ticket to track this change
  • If there should be document changes, a PR to emqx-docs.git is sent, or a jira ticket is created to follow up
  • Schema changes are backward compatible

@sstrigler sstrigler force-pushed the EMQX-8842-fix-rule-metrics branch 2 times, most recently from 896890a to 778576b Compare June 27, 2023 10:27
Copy link
Contributor

@kjellwinblad kjellwinblad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The only comments that I have are the things we discussed before. Maybe async_reply_fun is not the optimal name for the field as we use it even when the "query" is not async. Maybe on_complete_reply_fun would be better? Is async_reply_fun used by something else? Could it be "overwritten" somewhere in the call chain? Also as we discussed before, it would be good with a test case.

@@ -220,14 +220,15 @@ send_to_matched_egress_bridges(Topic, Msg) ->
send_message(BridgeId, Message) ->
{BridgeType, BridgeName} = emqx_bridge_resource:parse_bridge_id(BridgeId),
ResId = emqx_bridge_resource:resource_id(BridgeType, BridgeName),
send_message(BridgeType, BridgeName, ResId, Message).
send_message(BridgeType, BridgeName, ResId, Message, undefined).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit:

Suggested change
send_message(BridgeType, BridgeName, ResId, Message, undefined).
Opts = #{reply_to => undefined},
send_message(BridgeType, BridgeName, ResId, Message, Opts).

@@ -158,7 +158,9 @@ simple_async_query(Id, Request, QueryOpts0) ->
Ref = make_request_ref(),
Result = call_query(async_if_possible, Id, Index, Ref, ?SIMPLE_QUERY(Request), QueryOpts),
_ = handle_query_result(Id, Result, _HasBeenSent = false),
Result.
maybe_apply_async_reply_fun(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should leave the calling of the reply function to do_reply_caller, and just inject the callback as the From here instead of undefined.

i.e., something like:

  ReplyTo = maps:get(reply_to, QueryOpts, undefined),
  Query = ?QUERY(ReplyTo, Request, _HasBeenSeen = false, _ExpireAt = infinity),
  ...

and then I think do_reply_caller will call {Fun, Args} from From without further changes.

@coveralls
Copy link
Collaborator

coveralls commented Jun 27, 2023

Pull Request Test Coverage Report for Build 5411692341

  • 26 of 28 (92.86%) changed or added relevant lines in 4 files are covered.
  • 36 unchanged lines in 13 files lost coverage.
  • Overall coverage decreased (-0.02%) to 81.759%

Changes Missing Coverage Covered Lines Changed/Added Lines %
apps/emqx_resource/src/emqx_resource_buffer_worker.erl 15 16 93.75%
apps/emqx_rule_engine/src/emqx_rule_runtime.erl 8 9 88.89%
Files with Coverage Reduction New Missed Lines %
apps/emqx_bridge_gcp_pubsub/src/emqx_bridge_gcp_pubsub_consumer_worker.erl 1 58.73%
apps/emqx_gateway/src/emqx_gateway_schema.erl 1 94.44%
apps/emqx/src/emqx_cm.erl 1 92.05%
apps/emqx/src/emqx_limiter/src/emqx_limiter_schema.erl 1 81.48%
apps/emqx/src/emqx_quic_data_stream.erl 1 75.21%
apps/emqx_gateway_mqttsn/src/emqx_mqttsn_frame.erl 2 64.04%
apps/emqx_resource/src/emqx_resource_buffer_worker.erl 2 93.4%
apps/emqx_rule_engine/src/emqx_rule_runtime.erl 2 76.68%
apps/emqx/src/emqx_channel.erl 2 87.56%
apps/emqx/src/emqx_quic_connection.erl 2 78.57%
Totals Coverage Status
Change from base Build 5410523518: -0.02%
Covered Lines: 30524
Relevant Lines: 37334

馃挍 - Coveralls

@sstrigler sstrigler marked this pull request as ready for review June 29, 2023 08:58
@sstrigler sstrigler requested a review from a team as a code owner June 29, 2023 08:58
actions => [BridgeId]
}
),
timer:sleep(100),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor comment: to avoid the risk of creating flake test and not making the tests slow to run one could loop until the the expected condition is met and fail if it is not met after a certain amount of time (see https://github.com/klarna/snabbkaffe/blob/master/src/snabbkaffe.erl#L313)

emqx_resource_metrics:matched_inc(Id),
Ref = make_request_ref(),
Result = call_query(force_sync, Id, Index, Ref, ?SIMPLE_QUERY(Request), QueryOpts),
ReplyTo = maps:get(reply_to, QueryOpts0, undefined),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: now we have 2 different query option keys to specify async reply functions. Maybe we could choose only one of async_reply_fun and reply_to for this query option to avoid confusion?

@sstrigler sstrigler merged commit 07cf250 into emqx:master Jun 30, 2023
140 checks passed
@sstrigler
Copy link
Contributor Author

I鈥檒l create a follow-up ticket

@sstrigler sstrigler deleted the EMQX-8842-fix-rule-metrics branch June 30, 2023 18:08
@sstrigler
Copy link
Contributor Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants