Upgrade rocketmq for pubsub #1169

zach030 · 2021-09-25T15:58:39Z

Description

Upgrade rocketmq client with apache/rocketmq-client-go/v2 and remove dependence on third-party client.

Issue reference

Please reference the issue this PR will close: #[issue number]

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

Code compiles correctly
Created/updated tests
Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

…components-contrib into feature_pubsub_rocketmq

ghost · 2021-09-25T15:58:50Z

All CLA requirements met.

daixiang0 · 2021-09-26T01:06:47Z

Could share why need to upgrade?

CodeMonkeyLeet · 2021-09-27T16:32:25Z

@zach030 can you also look at the build failures? It looks like the resulting go.sum is incomplete so a go mod tidy is needed

zach030 · 2021-09-27T17:35:11Z

Could share why need to upgrade?

The previous pubsub component rocketmq depends on third-party libraries. According to the opinions of the RocketMQ community, I remove it and make it purely base on apache/rocketmq-client-go

zach030 · 2021-10-06T16:32:27Z

@zach030 can you also look at the build failures? It looks like the resulting go.sum is incomplete so a go mod tidy is needed

done

pubsub/rocketmq/rocketmq.go

pubsub/rocketmq/metadata.go

CodeMonkeyLeet · 2021-10-06T18:20:29Z

tests/config/pubsub/rocketmq/pubsub.yml

+    - name: accessKey
+      value: "RocketMQ"
+    - name: secretKey
+      value: "12345"


Nit: Missing newline at EOF for linux text files.

CodeMonkeyLeet · 2021-10-06T18:23:56Z

@yaron2 @artursouza This is a fairly significant change, have the maintainers approved this? It looks in line with previous requests such as #890

CodeMonkeyLeet · 2021-10-06T18:24:37Z

@zach030 Still hitting build failures with the go.sum issues

artursouza · 2021-10-06T18:38:18Z

This component was contributed by Alibaba.

/cc @LaurenceLiZhixin Is this change OK for Alibaba?

…components-contrib into feature_pubsub_rocketmq

Co-authored-by: xujie <xujie1@shein.com> Signed-off-by: zach <zachchou016@gmail.com>

) Co-authored-by: Long Dai <long.dai@intel.com> Co-authored-by: Artur Souza <artursouza.ms@outlook.com> Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com> Signed-off-by: zach <zachchou016@gmail.com>

* support tracestate can be propagated in PubSub * add trace state Co-authored-by: Long Dai <long.dai@intel.com> Co-authored-by: Ian Luo <ian.luo@gmail.com> Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com> Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com> Signed-off-by: zach <zachchou016@gmail.com>

* bugfix for sns topic deletion upon termination * removed upstream github workflow files * Update snssqs.go * dapr bot schedule * read and append queue attributes * unnecessary escaping in json tag * unexporting structs Signed-off-by: zach <zachchou016@gmail.com>

Signed-off-by: zach <zachchou016@gmail.com>

…1345) Signed-off-by: zach <zachchou016@gmail.com>

* fix: add config subscription gr done * unexpected comments * fix comment Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com> Signed-off-by: zach <zachchou016@gmail.com>

* Adds retry on CosmosDB Init in case of TooManyRequests error * Use backoff v4 * missed some permanent errors * clean up go.mod * fix error type casting * Add constant for HTTP 429 Signed-off-by: zach <zachchou016@gmail.com>

Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com> Signed-off-by: zach <zachchou016@gmail.com>

…#1319 (#1349) Signed-off-by: zach <zachchou016@gmail.com>

* bugfix for sns topic deletion upon termination * removed upstream github workflow files * wrapping errors * fifo and naming sanitization * fifo and name santization tests * component carrying id * Update snssqs.go removed unneeded import * Update snssqs.go * add missing github actions file * linting * fix go.mod * schedule yml file * const max lenght, bugfix in ack * dapr bot schedule Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com> Signed-off-by: zach <zachchou016@gmail.com>

* Add certification tests for ASB Queue binding This commit adds certification tests for the Azure Service Bus Queue Input/Output binding. This utilizes the new certification framework and performs a series of tests as described in the test plan readme. The general purpose of these tests is to serve as an integration test and as such requires an actual Azure Service Bus connection and a Dapr sidecar. #957 * Add to github workflow and fix formatting * Fix dependencies Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com> Signed-off-by: zach <zachchou016@gmail.com>

Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com> Signed-off-by: zach <zachchou016@gmail.com>

Signed-off-by: zach <zachchou016@gmail.com>

* bugfix for sns topic deletion upon termination * removed upstream github workflow files * Update snssqs.go * dapr bot schedule * read and append queue attributes * unnecessary escaping in json tag * unexporting structs * bugfix in policy * bugfix in policy. merged from master * fifo suffix as const Signed-off-by: zach <zachchou016@gmail.com>

Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com> Signed-off-by: zach <zachchou016@gmail.com>

* Fix test infrastructure setup script Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com> * use more unique ACR resource name Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com> Co-authored-by: Looong Dai <long.dai@intel.com> Signed-off-by: zach <zachchou016@gmail.com>

Signed-off-by: zach <zachchou016@gmail.com>

Signed-off-by: Amit Mor <amit.mor@hotmail.com> Signed-off-by: zach <zachchou016@gmail.com>

* Adds CosmosDB Binding authentication tests Signed-off-by: GitHub <noreply@github.com> Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com> * Enable Cosmos DB Binding Conformance test Signed-off-by: GitHub <noreply@github.com> Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com> * Initial cosmosdb binding certification plan WIP Signed-off-by: GitHub <noreply@github.com> Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com> * Go mod tidy Signed-off-by: GitHub <noreply@github.com> Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com> * Update library and go mod tidy Signed-off-by: GitHub <noreply@github.com> Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com> * make modtidy-all Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com> * CosmosDB Binding test plan details Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com> Signed-off-by: zach <zachchou016@gmail.com>

Signed-off-by: Dmitry Shmulevich <dmitry.shmulevich@gmail.com> Signed-off-by: zach <zachchou016@gmail.com>

Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com> Signed-off-by: zach <zachchou016@gmail.com>

…ts-contrib into feature_pubsub_rocketmq

codecov · 2021-12-18T17:04:03Z

Codecov Report

Merging #1169 (a47cc39) into master (870fedb) will increase coverage by 0.29%.
The diff coverage is 37.65%.

❗ Current head a47cc39 differs from pull request most recent head 1e939c5. Consider uploading reports for the commit 1e939c5 to get more accurate results

@@            Coverage Diff             @@
##           master    #1169      +/-   ##
==========================================
+ Coverage   34.95%   35.24%   +0.29%     
==========================================
  Files         149      149              
  Lines       12923    12962      +39     
==========================================
+ Hits         4517     4569      +52     
+ Misses       7922     7898      -24     
- Partials      484      495      +11

Impacted Files	Coverage Δ
tests/conformance/common.go	`16.60% <0.00%> (-0.13%)`	⬇️
pubsub/rocketmq/rocketmq.go	`27.81% <31.15%> (+27.81%)`	⬆️
pubsub/rocketmq/metadata.go	`81.81% <81.81%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7a631ef...1e939c5. Read the comment docs.

zach030 · 2021-12-18T17:30:08Z

@yaron2 I find it's quite confused for me to fix conflict while rebasing. So can I checkout a new branch and open another PR?

yaron2 · 2021-12-18T17:31:09Z

@yaron2 I find it's quite confused for me to fix conflict while rebasing. So can I checkout a new branch and open another PR?

Yep go for it

zach030 · 2021-12-18T18:22:42Z

new pr #1383

zach030 added 12 commits September 25, 2021 21:41

[add] implement pubsub for rocketmq-client

ea2fcd8

[add] parse rocket metadata

1b1a4cc

[add] producer consumer init

3f3f97a

[add] rocketmq pubsub test

9f6af59

[add] rocketmq support publish mode

f704605

[fix] setup consumer err

b0a84e8

add few comment

086e025

feat producer cache

7b04803

feat producer pool

569984e

feat add metadata test

a3d6a5c

Merge branch 'feature_pubsub_rocketmq' of https://github.com/zach030/…

ed7dcfc

…components-contrib into feature_pubsub_rocketmq

feat integrate test for rocketmq

c47fa3b

zach030 requested review from a team as code owners September 25, 2021 15:58

artursouza and others added 4 commits September 28, 2021 19:19

Merge branch 'master' into feature_pubsub_rocketmq

9334bac

Merge branch 'master' into feature_pubsub_rocketmq

927d1ce

Merge branch 'master' into feature_pubsub_rocketmq

2d1a2dd

Update go.sum

e3cf0af

CodeMonkeyLeet reviewed Oct 6, 2021

View reviewed changes

zach030 added 3 commits October 7, 2021 08:52

fix add license and format

8c3a04f

Merge branch 'feature_pubsub_rocketmq' of https://github.com/zach030/…

f348c62

…components-contrib into feature_pubsub_rocketmq

fix format yml config

6a67d24

xj524598 and others added 24 commits December 19, 2021 00:54

update some ratelimit_middleware.go comments (#1330)

37f0de9

Co-authored-by: xujie <xujie1@shein.com> Signed-off-by: zach <zachchou016@gmail.com>

fixed typo in Query JSON (#1344)

12d94c8

Signed-off-by: zach <zachchou016@gmail.com>

Forcing dapr/dapr commit to fix compatibility w/ components-contrib (#…

359a416

…1345) Signed-off-by: zach <zachchou016@gmail.com>

fix: add config subscription gr done (#1315)

5cd2c62

* fix: add config subscription gr done * unexpected comments * fix comment Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com> Signed-off-by: zach <zachchou016@gmail.com>

Ignore go.work for Go 1.18 Workspaces (#1324)

e448baa

Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com> Signed-off-by: zach <zachchou016@gmail.com>

upgrade dapr's version to fix go mod tidy issue in tests/certification …

7efc735

…#1319 (#1349) Signed-off-by: zach <zachchou016@gmail.com>

Tidy & Pin certification tests to latest Dapr/Dapr (#1361)

7cc0afd

Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com> Signed-off-by: zach <zachchou016@gmail.com>

Adds missing ACR resource definition for test infrastructure (#1359)

596d6b2

Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com> Signed-off-by: zach <zachchou016@gmail.com>

Enables autoupdate for PRs to be in sync with master. (#1363)

264dbbe

Signed-off-by: zach <zachchou016@gmail.com>

Update azure-sdk-for-go to v59.3.0 for EventGrid (#1325)

6669154

Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com> Signed-off-by: zach <zachchou016@gmail.com>

Fix dependency issue in service bus queue tests (#1374)

e556dec

Signed-off-by: zach <zachchou016@gmail.com>

disable entity creation (#1375)

428b75a

Signed-off-by: Amit Mor <amit.mor@hotmail.com> Signed-off-by: zach <zachchou016@gmail.com>

pub/sub rabbitmq: fix subscribe error (#1377)

4b533ab

Signed-off-by: Dmitry Shmulevich <dmitry.shmulevich@gmail.com> Signed-off-by: zach <zachchou016@gmail.com>

fix binding list test (#1317)

6404e60

Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com> Signed-off-by: zach <zachchou016@gmail.com>

Merge branch 'feature_pubsub_rocketmq' of github.com:zach030/componen…

1e939c5

…ts-contrib into feature_pubsub_rocketmq

zach030 mentioned this pull request Dec 18, 2021

pub/sub rocketmq:upgrade with v2 #1383

Merged

3 tasks

zach030 closed this Dec 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade rocketmq for pubsub #1169

Upgrade rocketmq for pubsub #1169

zach030 commented Sep 25, 2021 •

edited

ghost commented Sep 25, 2021 •

edited by ghost

daixiang0 commented Sep 26, 2021

CodeMonkeyLeet commented Sep 27, 2021

zach030 commented Sep 27, 2021

zach030 commented Oct 6, 2021

CodeMonkeyLeet Oct 6, 2021

CodeMonkeyLeet commented Oct 6, 2021

CodeMonkeyLeet commented Oct 6, 2021

artursouza commented Oct 6, 2021 •

edited

codecov bot commented Dec 18, 2021

zach030 commented Dec 18, 2021

yaron2 commented Dec 18, 2021

zach030 commented Dec 18, 2021

Upgrade rocketmq for pubsub #1169

Upgrade rocketmq for pubsub #1169

Conversation

zach030 commented Sep 25, 2021 • edited

Description

Issue reference

Checklist

ghost commented Sep 25, 2021 • edited by ghost

daixiang0 commented Sep 26, 2021

CodeMonkeyLeet commented Sep 27, 2021

zach030 commented Sep 27, 2021

zach030 commented Oct 6, 2021

CodeMonkeyLeet Oct 6, 2021

Choose a reason for hiding this comment

CodeMonkeyLeet commented Oct 6, 2021

CodeMonkeyLeet commented Oct 6, 2021

artursouza commented Oct 6, 2021 • edited

codecov bot commented Dec 18, 2021

Codecov Report

zach030 commented Dec 18, 2021

yaron2 commented Dec 18, 2021

zach030 commented Dec 18, 2021

zach030 commented Sep 25, 2021 •

edited

ghost commented Sep 25, 2021 •

edited by ghost

artursouza commented Oct 6, 2021 •

edited