Support configurable MaxOutstanding* and NumGoroutines settings for GCP PubSub component

[nathandl86]

Description

Adds metadata settings for GCP PubSub component to configure the MaxOutstandingMessages, MaxOutstandingBytes, and NumGoroutines settings in the underlying Google Cloud pubsub package.

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: #3441

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

• Code compiles correctly
• Created/updated tests
• Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#4232

[berndverst]

If a work in progress, please open as a draft PR. I'll convert to draft for now. (Most maintainers do not review draft PRs by the way)

[berndverst]

Can you explore adding some sort of test that shows these settings are used correctly? Our "certification test" (which is a kind of end to end integration test) lives here: https://github.com/dapr/components-contrib/tree/main/tests/certification/pubsub/gcp/pubsub

[nathandl86]

If a work in progress, please open as a draft PR. I'll convert to draft for now. (Most maintainers do not review draft PRs by the way)

Ah, sorry about that 🙇🏻 I'll add a commit updating the CONTRIBUTING.md to call out that the PR should be draft where it mentions the [WIP] prefix. I wasn't able to add the do not merge label. Do you want that removed from the readme?

[nathandl86]

Can you explore adding some sort of test that shows these settings are used correctly? Our "certification test" (which is a kind of end to end integration test) lives here: https://github.com/dapr/components-contrib/tree/main/tests/certification/pubsub/gcp/pubsub

Yep! I'm going to look into adding the tests tonight. Part of the reason I opened it as [WIP] was I knew that wasn't there. The other reason it's currently a [WIP] is I haven't been able to manually test yet either.

[nathandl86]

Can you explore adding some sort of test that shows these settings are used correctly? Our "certification test" (which is a kind of end to end integration test) lives here: https://github.com/dapr/components-contrib/tree/main/tests/certification/pubsub/gcp/pubsub

Yep! I'm going to look into adding the tests tonight. Part of the reason I opened it as [WIP] was I knew that wasn't there. The other reason it's currently a [WIP] is I haven't been able to manually test yet either.

@berndverst One hang-up I'm running into is how to iterate on the tests to prove this out. The certification tests look tied to GitHub Actions. Is there a way y'all typically run/iterate on these locally? It's entirely possible this is just me not being as comfortable in golang 😅

[yaron2]

@nathandl86 we don't have certification test infrastructure for GCP pub/sub today, so you're not required to add a test for this that interacts with GCP infra. You can update the existing Init certification test to check for the new metadata field, and this test you can run locally.

[nathandl86]

@nathandl86 we don't have certification test infrastructure for GCP pub/sub today, so you're not required to add a test for this that interacts with GCP infra. You can update the existing Init certification test to check for the new metadata field, and this test you can run locally.

@yaron2 I updated the unit tests living next to the GCP pubsub module. I'm not sure if those are the Init tests you intended though since they did not live in the certification directory?

[nathandl86]

I'd mentioned in the Discord that we had some challenges in manual testing. Closing that loop, I worked with my DevOps team and they setup an isolated cluster for us to test. Testing was successful 🎉 Where before we were seeing scale up of pods that were not processing messages,

We now see full participation of our pods because there are available messages for their sidecar to be served 🙌🏻

We also saw a massive improvement in our unacked age metric. Before we would see it climb above 20 minutes during a load test.

Across 5 tests with higher volume yesterday and today it stayed just below 4:

[ItalyPaleAle]

Thank you!

Can you please update the docs in https://github.com/dapr/docs too?

[nathandl86]

Thank you!

Can you please update the docs in https://github.com/dapr/docs too?
Thank you for the feedback and merging it 🙇🏻

Yep! I'll work on that now. A couple things I want to confirm:

• I assume this is the area where I should add the new settings: https://github.com/dapr/docs/blob/7625eee6a9e3334f4406f496929e47d75ad6ed04/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-gcp-pubsub.md?plain=1 ?
• Based on the 1.14 milestone I assume I should target the 1.14 branch in the docs repo?

[ItalyPaleAle]

Yes to both, thank you!