feat(pubsub): add opentelemetry tracing for publish #8317

hongalex · 2023-07-24T18:02:07Z

This will one of a few of PRs to implement OpenTelemetry Tracing into the Pub/Sub client library, starting with publish side spans.

Sample trace using Cloud Trace:

…sub-otel-trace-publish

quartzmo

LGTM

feywind

Overall this looks good to me :)

pubsub/topic.go

feywind · 2023-08-14T19:45:59Z

pubsub/trace.go

+	publisherSpanName          = "send"
+	publishFlowControlSpanName = "publisher flow control"
+	publishSchedulerSpanName   = "publish scheduler"
+	publishRPCSpanName         = "send Publish"


The spec doc seems to imply that this should be just "publish"?

Yeah this should just be Publish, I went back and forth on this for a bit.

…sub-otel-trace-publish

…#8439) * feat(datastore): Support aggregation query in transaction * feat(datastore): Refactoring integration test * feat(datastore): Integration tests for sum and average

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

…pis#8467) * fix: Transaction was started in a different session Retrying a "Session not found" error could cause a "Transaction was started in a different session" error. This happened because: 1. The detection of a "Session not found" error would remove the session from the pool, and also remove the session ID from the session handle 2. The retry mechanism would check out a new session from the pool, but not assign it to the transaction yet 3. The retry would then proceed to retry the transaction with an explicit BeginTransaction RPC. This function would however pick a new session from the pool, because step 2 had not yet assigned the transaction a new session. 4. The higher level retry loop would then after executing the BeginTransaction RPC assign the session that was picked in step 2 to the transaction. 5. The transaction would then proceed to use the session from step 2 with the transaction from step 3. * chore: remove unused code * chore: fix import order

Co-authored-by: rahul2393 <irahul@google.com>

…eapis#8473) * fix(datastore): Truncate transaction read time to millisecond * fix(datastore): Truncate transaction read time to microsecond

This package provides: - A TokenProvider interface - A Token type - A standard auth Error type - Configuration and TokenProvider for 2L0 oauth2 flows - Configuration and TokenProvider for 3L0 oauth2 flows - A means to cache tokens This code has been adapted from the golang oauth2 repo. In particular the types here are analogous to: oauth2, authhandler, and jwt packages. It should feel familiar for anyone who as worked with that library before, but it only provides a subset of the features that we require for our client libraries and auth stack as defined by our AIPs.

…is#8475)

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

…ons and GENERATE_UUID (googleapis#8482) * feat(spanner/spansql): add support for bit functions and sequence functions * feat(spanner/spansql): add support for function GENERATE_UUID

) Increasing timeout to give some leeway. If the test continues flaking, we should investigate further why it fails.

…oogleapis#8478) Fixes: https://togithub.com/googleapis/google-cloud-go/issues/7942

…pis#8479) Fixes: https://togithub.com/googleapis/google-cloud-go/issues/7454

…s#8481) * feat(spanner/spansql): add support for SEQUENCE statements * feat(spanner/spansql): remove using generics for CI env --------- Co-authored-by: Sri Harsha CH <57220027+harshachinta@users.noreply.github.com> Co-authored-by: rahul2393 <irahul@google.com>

feat: publicize tpu topology in v1 API PiperOrigin-RevId: 560116372 Source-Link: googleapis/googleapis@cf6f9ee Source-Link: googleapis/googleapis-gen@b618aaf Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYjYxOGFhZmE2NDE3NDFhNTYxNWMyMTZiODc4NjAwNjViZGMzMGY4ZCJ9 BEGIN_NESTED_COMMIT feat(bigquery/datapolicies): support using custom UDF in the data policy PiperOrigin-RevId: 559480224 Source-Link: googleapis/googleapis@4ffd6cd Source-Link: googleapis/googleapis-gen@2fa3b74 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMmZhM2I3NDk2ZjkyNzQ1YjE4N2QxYmI1ZjllNjVlNWQwMzZhNWQwNyJ9 END_NESTED_COMMIT BEGIN_NESTED_COMMIT docs(optimization): Minor formatting fix PiperOrigin-RevId: 559474510 Source-Link: googleapis/googleapis@5b3202b Source-Link: googleapis/googleapis-gen@7b6e921 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiN2I2ZTkyMTA4NTEzZDBmZjIxNjE0ZTBkYTZmYzgyNzkxNjFlNTVhMyJ9 END_NESTED_COMMIT BEGIN_NESTED_COMMIT docs(datacatalog): fix typo PiperOrigin-RevId: 559249168 Source-Link: googleapis/googleapis@f49882a Source-Link: googleapis/googleapis-gen@08d36c5 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMDhkMzZjNTkzYWI2MGYxODRhZmUxNDIzYTg0ZGY5NDJlMmFmMmRmMiJ9 END_NESTED_COMMIT BEGIN_NESTED_COMMIT docs(billing): update comments PiperOrigin-RevId: 559241398 Source-Link: googleapis/googleapis@6fe53ad Source-Link: googleapis/googleapis-gen@11b03ab Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMTFiMDNhYjZmOWNmMDBjYmNkMTY3MWFmN2Y1ZDcxMzQ4ZjI1NmE3NCJ9 END_NESTED_COMMIT BEGIN_NESTED_COMMIT feat(billing/budgets): Supported project-level-budgets in Public Budget API V1 Added scope for project scope filter in ListBudgetsRequest. When this field is set to a project's resource name, the budgets returned are tracking the costs for the given project. feat: Added enable_project_level_recipients for project owner budget emails Setting this field to true will enable email notification to project owners for project-level budgets. PiperOrigin-RevId: 559152052 Source-Link: googleapis/googleapis@113867b Source-Link: googleapis/googleapis-gen@017d177 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMDE3ZDE3N2Y1MTQ2OWQxNTA0MTdiN2I2YTQ1ZDk5ZDk5NjNmNzgxMyJ9 END_NESTED_COMMIT BEGIN_NESTED_COMMIT feat(billing/budgets): Supported project-level-budgets in Public Budget API V1Beta Added scope for project scope filter in ListBudgetsRequest. When this field is set to a project's resource name, the budgets returned are tracking the costs for the given project. feat: Added enable_project_level_recipients for project owner budget emails Setting this field to true will enable email notification to project owners for project-level budgets. PiperOrigin-RevId: 559151636 Source-Link: googleapis/googleapis@138b0f1 Source-Link: googleapis/googleapis-gen@41c0ba8 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNDFjMGJhODk5Zjc2Y2JiZmFjZGRlY2NmN2M3Njk3NDdhY2FmZTI2ZCJ9 END_NESTED_COMMIT BEGIN_NESTED_COMMIT feat(dataproc): support min_num_instances for primary worker and InstanceFlexibilityPolicy for secondary worker PiperOrigin-RevId: 559135594 Source-Link: googleapis/googleapis@4a5a6fd Source-Link: googleapis/googleapis-gen@5c91126 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNWM5MTEyNjFmMDAyNTdmNzY4YThhMzIzYWExYTNlNDgzNjQwYzFkMiJ9 END_NESTED_COMMIT BEGIN_NESTED_COMMIT docs(batch): Revert HTML formats in comments docs: Expand compute resource API docs to match with VM's machine type field docs: Clarify Batch API proto doc about pubsub notifications PiperOrigin-RevId: 558940776 Source-Link: googleapis/googleapis@a7ee700 Source-Link: googleapis/googleapis-gen@4f3a485 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNGYzYTQ4NWYzY2ViMTNiNjdlMDUyMmI3NGE3ZmQ5Y2NkZTJhYTZhMSJ9 END_NESTED_COMMIT

…s#8498)

…leapis#8489)

hongalex added 3 commits July 20, 2023 15:07

feat(pubsub): add publish tracing

c439058

refactor tests to use InMemoryExporter

217ab44

update copyright year

128c6b1

hongalex requested review from a team and shollyman as code owners July 24, 2023 18:02

product-auto-label bot added size: l Pull request size is large. api: pubsub Issues related to the Pub/Sub API. labels Jul 24, 2023

Merge branch 'main' of github.com:googleapis/google-cloud-go into pub…

17f41b0

…sub-otel-trace-publish

hongalex requested review from a team as code owners July 24, 2023 18:12

product-auto-label bot added size: xl Pull request size is extra large. and removed size: l Pull request size is large. labels Jul 24, 2023

googleapis deleted a comment from snippet-bot bot Jul 24, 2023

hongalex removed the size: xl Pull request size is extra large. label Jul 24, 2023

run go mod tidy

d4a9ef3

product-auto-label bot added the size: xl Pull request size is extra large. label Jul 24, 2023

hongalex added 4 commits July 27, 2023 15:25

rename PubsubMessageCarrier -> MessageCarrier, fix race

229c80e

make messageCarrier not exported

e1fe160

downgrade otel sdk version to support go1.17

c2bee2a

add short delay to ensure spans are exported

0f03c49

product-auto-label bot added size: l Pull request size is large. and removed size: xl Pull request size is extra large. labels Aug 3, 2023

merge with slush branch

47693c4

quartzmo approved these changes Aug 4, 2023

View reviewed changes

feywind approved these changes Aug 14, 2023

View reviewed changes

hongalex added 2 commits August 16, 2023 14:20

context renaming

247465a

Merge branch 'main' of github.com:googleapis/google-cloud-go into pub…

f3938f2

…sub-otel-trace-publish

hongalex requested review from a team as code owners August 18, 2023 22:23

kokoro-team removed kokoro:run Add this label to force Kokoro to re-run the tests. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Aug 22, 2023

bhshkh and others added 10 commits August 23, 2023 01:13

feat(datastore): Support aggregation query in transaction (googleapis…

37681ff

…#8439) * feat(datastore): Support aggregation query in transaction * feat(datastore): Refactoring integration test * feat(datastore): Integration tests for sum and average

chore(main): release datastore 1.14.0 (googleapis#8351)

911f31e

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

add helper functions for creating publish spans

ca01559

refactor error tests

567ed2d

feat(spanner/spannertest): support INSERT DML (googleapis#7820)

3dda7b2

Co-authored-by: rahul2393 <irahul@google.com>

test(datastore): Truncate transaction read time to millisecond (googl…

14aa857

…eapis#8473) * fix(datastore): Truncate transaction read time to millisecond * fix(datastore): Truncate transaction read time to microsecond

test(datastore): Correct read time before creating entities (googleap…

2c65345

…is#8475)

chore(main): release spanner 1.49.0 (googleapis#8468)

d773366

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

product-auto-label bot added the stale: old Pull request is old and needs attention. label Aug 24, 2023

noahdietz and others added 14 commits August 24, 2023 11:22

chore(ci): increase new-client fetch depth (googleapis#8483)

8c1a363

feat(spanner/spansql): add support for bit functions, sequence functi…

3789882

…ons and GENERATE_UUID (googleapis#8482) * feat(spanner/spansql): add support for bit functions and sequence functions * feat(spanner/spansql): add support for function GENERATE_UUID

test(storage): increase timeout for requester pays test (googleapis#8421

3d8c2c8

) Increasing timeout to give some leeway. If the test continues flaking, we should investigate further why it fails.

test(pubsublite): fix flaky TestRoutingPublisherUnloadIdlePublisher (g…

236904e

…oogleapis#8478) Fixes: https://togithub.com/googleapis/google-cloud-go/issues/7942

test(pubsublite): fix flaky TestIntegration_SeekSubscription (googlea…

ee274c7

…pis#8479) Fixes: https://togithub.com/googleapis/google-cloud-go/issues/7454

feat(spanner): allow non-default service accounts (googleapis#8488)

c90dd00

chore(config): add config to generate apiv1 (googleapis#8490)

d2a9d82

feat(config): new clients (googleapis#8493)

9874485

feat(spanner/spansql): add support for aggregate functions (googleapi…

d440d75

…s#8498)

fix exactly once delivery test

2fef801

fix(pubsub): make AckWithResult return success when constructed (goog…

38a040e

…leapis#8489)

pull in changes from main

098ffa5

product-auto-label bot added size: xl Pull request size is extra large. and removed size: l Pull request size is large. labels Aug 28, 2023

hongalex merged commit db03297 into googleapis:pubsub-otel-trace Aug 31, 2023
9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(pubsub): add opentelemetry tracing for publish #8317

feat(pubsub): add opentelemetry tracing for publish #8317

hongalex commented Jul 24, 2023 •

edited

quartzmo left a comment

feywind left a comment

feywind Aug 14, 2023

hongalex Aug 16, 2023

feat(pubsub): add opentelemetry tracing for publish #8317

feat(pubsub): add opentelemetry tracing for publish #8317

Conversation

hongalex commented Jul 24, 2023 • edited

quartzmo left a comment

Choose a reason for hiding this comment

feywind left a comment

Choose a reason for hiding this comment

feywind Aug 14, 2023

Choose a reason for hiding this comment

hongalex Aug 16, 2023

Choose a reason for hiding this comment

hongalex commented Jul 24, 2023 •

edited