Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support send msgs to exchange and flush msgs from exchange to parquet #1560

Merged
merged 20 commits into from Dec 14, 2023
Merged

Support send msgs to exchange and flush msgs from exchange to parquet #1560

merged 20 commits into from Dec 14, 2023

Conversation

wanghaEMQ
Copy link
Member

  1. Support sends msgs to exchange in parallel
  2. Support flush msgs got from exchange to parquet.

@wanghaEMQ
* NEW [hook] Support flush msgs to disk.
4afea24 
Signed-off-by: wanghaemq <wangwei@emqx.io>
@wanghaEMQ
* NEW [hook] Add process of exchange initial.
3394f00 
Signed-off-by: wanghaemq <wangwei@emqx.io>
@wanghaEMQ
* NEW [hook] Add keys as an input of parquet write batch.
bcfcc89 
Signed-off-by: wanghaemq <wangwei@emqx.io>
@wanghaEMQ
* NEW [hook] Add lengths of datas.
2c6e337 
Signed-off-by: wanghaemq <wangwei@emqx.io>
@wanghaEMQ
* FIX [hook] Add handler for the case that aio busy.
a4f55e9 
Signed-off-by: wanghaemq <wangwei@emqx.io>
@wanghaEMQ
* FIX [hook] Flush msgs in callback of send exchange.
0226115 
Signed-off-by: wanghaemq <wangwei@emqx.io>
@wanghaEMQ
* FIX [hook] Update code with latest exchange structure,
560bc39 
Signed-off-by: wanghaemq <wangwei@emqx.io>
@wanghaEMQ
* NEW [hook] Create aio for each exchange sender.
eb67f28 
Signed-off-by: wanghaemq <wangwei@emqx.io>
@wanghaEMQ
* NEW [hook] Send msgs to exchange in parallel.
1aef1f7 
Signed-off-by: wanghaemq <wangwei@emqx.io>
@wanghaEMQ
* NEW [hook] Fix the warnings when building.
e9a0bbf 
Signed-off-by: wanghaemq <wangwei@emqx.io>
@wanghaEMQ
* NEW [hook] Set payload ptr when decode publish package.
0869266 
Signed-off-by: wanghaemq <wangwei@emqx.io>
@wanghaEMQ
* NEW [hook] Update the location of input parameters.
af33131 
Signed-off-by: wanghaemq <wangwei@emqx.io>
@wanghaEMQ
* FIX [hook] Add lock for keys in hook and remove the check of busy o…
9fc2a1b 
…f aio due to queue flush in parquet.

Signed-off-by: wanghaemq <wangwei@emqx.io>
@wanghaEMQ
Copy link
Member Author

Needs PR nanomq/NanoNNG#765 to be merged

@wanghaEMQ
* FIX [hook] Fix the memory leakage of msgs.
18bf292 
Signed-off-by: wanghaemq <wangwei@emqx.io>
@wanghaEMQ
* SYNC nng.
9104473 
Signed-off-by: wanghaemq <wangwei@emqx.io>
github-advanced-security[bot]
github-advanced-security bot found potential problems Dec 13, 2023
View reviewed changes
nanomq/webhook_post.c Fixed Show fixed Hide fixed
nanomq/webhook_post.c Fixed Show fixed Hide fixed
nanomq/webhook_post.c Fixed Show fixed Hide fixed
nanomq/webhook_post.c Fixed Show fixed Hide fixed
nanomq/webhook_post.c Fixed Show fixed Hide fixed
nanomq/webhook_post.c Fixed Show fixed Hide fixed
nanomq/webhook_post.c Fixed Show fixed Hide fixed
@codecov Codecov
Copy link

codecov bot commented Dec 13, 2023 •
edited

Codecov Report

Attention: 76 lines in your changes are missing coverage. Please review.

Comparison is base (5e2883f) 57.83% compared to head (7aaf953) 57.26%.

❗ Current head 7aaf953 differs from pull request most recent head 4afdde9. Consider uploading reports for the commit 4afdde9 to get more accurate results

Files Patch % Lines
nanomq/webhook_post.c 0.00% 72 Missing ⚠️
nanomq/apps/broker.c 33.33% 4 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #1560      +/-   ##
==========================================
- Coverage   57.83%   57.26%   -0.57%     
==========================================
  Files          18       18              
  Lines        6557     6622      +65     
==========================================
  Hits         3792     3792              
- Misses       2765     2830      +65

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@wanghaEMQ
* FIX [hook] Fix the warnings found by codeql.
6df49eb 
Signed-off-by: wanghaemq <wangwei@emqx.io>
JaylinYu
JaylinYu requested changes Dec 13, 2023
View reviewed changes
nanomq/apps/broker.c Outdated Show resolved Hide resolved
nanomq/apps/broker.c Outdated Show resolved Hide resolved
nanomq/pub_handler.c
@@ -1593,6 +1593,7 @@ decode_pub_message(nano_work *work, uint8_t proto)
// payload
pub_packet->payload.len =
(uint32_t) (msg_len - (size_t) used_pos);
nng_msg_set_payload_ptr(msg, msg_body + pos);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool

OdyWayne
OdyWayne reviewed Dec 14, 2023
View reviewed changes
nanomq/webhook_post.c Outdated Show resolved Hide resolved
@wanghaEMQ
* FIX [hook] Fix the warning on windows.
5331e41 
Signed-off-by: wanghaemq <wangwei@emqx.io>
@wanghaEMQ
* NEW [hook] Update the number of aios for sending msg to exchange.
349c26f 
Signed-off-by: wanghaemq <wangwei@emqx.io>
@wanghaEMQ
* FIX [hook] Fix the warnings in building on windows .
7aaf953 
Signed-off-by: wanghaemq <wangwei@emqx.io>
@wanghaEMQ
* NEW [hook] Exchange as an indepent service to start.
4afdde9 
* FIX [hook] Fix the wrong num_ctx.

Signed-off-by: wanghaemq <wangwei@emqx.io>
JaylinYu
JaylinYu approved these changes Dec 14, 2023
View reviewed changes
nanomq/webhook_post.c
*nkey = g_msg_index++;
nng_mtx_unlock(hook_conf->ex_mtx);

if (nng_aio_busy(aio))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to check busy

@JaylinYu JaylinYu merged commit 592dd3d into master Dec 14, 2023
81 of 82 checks passed
@JaylinYu JaylinYu deleted the wangha/dev branch December 14, 2023 07:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JaylinYu JaylinYu JaylinYu approved these changes

@xinyi-xs xinyi-xs Awaiting requested review from xinyi-xs

@RanMaoyi RanMaoyi Awaiting requested review from RanMaoyi

@OdyWayne OdyWayne Awaiting requested review from OdyWayne

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@wanghaEMQ @OdyWayne @JaylinYu