Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cdc: add telemetry logging for create changefeed #77283

Closed
amruss opened this issue Mar 2, 2022 · 3 comments · Fixed by #79513
Closed

cdc: add telemetry logging for create changefeed #77283

amruss opened this issue Mar 2, 2022 · 3 comments · Fixed by #79513
Assignees
@samiskin
Labels
A-cdc Change Data Capture C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-cdc
Projects
[DEPRECATED] CDC

Comments

@amruss
Copy link
Contributor

amruss commented Mar 2, 2022 •
edited by cockroach-jira-scripts

Since the new logging framework is only for serverless. We should do some more detailed logging of create changefeed for majority of users going into 22.1 launch. This will help us better understand what are customers are doing and where they are struggling.

Create changefeed event:
The biggest thing we should log is the sink_type (gs, s3, http, webhook, kafka, gc pubsub). Additionally, logging the number of non-default configurations would be a great thing to log, but its a nice to have not a must have.

Changefeed fail event
There are three things we should log

  • changefeed failed before changefeed create succeeded
  • changefeed failed due to closed connection (core)
  • All other failures

Jira issue: CRDB-13517
@amruss amruss added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-cdc Change Data Capture T-cdc labels Mar 2, 2022
@blathers-crl
Copy link

blathers-crl bot commented Mar 2, 2022

cc @cockroachdb/cdc

@amruss amruss added this to Triage in [DEPRECATED] CDC via automation Mar 2, 2022
@amruss amruss moved this from Triage to Stability Period in [DEPRECATED] CDC Mar 2, 2022
@samiskin samiskin mentioned this issue Apr 6, 2022
Merged
samiskin added a commit to samiskin/cockroach that referenced this issue Apr 6, 2022
@samiskin
changefeedccl: add structured logging for changefeed create and failed
54b5acc 
Resolves cockroachdb#77283

Previously we didn't have any logging related to the new Telemetry
channel in the structured logs which are exported to Snowflake. This
patch adds events around changefeed creation and failure, informing us
of some creation options and a general reason for a failure.

Release justification: low risk logging related changes

Release note (ops change): Changefeed creations and failures event logs
are now emitted to the telemetry channel
samiskin added a commit to samiskin/cockroach that referenced this issue Apr 7, 2022
@samiskin
changefeedccl: add structured logging for changefeed create and failed
08fb8a7 
Resolves cockroachdb#77283

Previously we didn't have any logging related to the new Telemetry
channel in the structured logs which are exported to Snowflake. This
patch adds events around changefeed creation and failure, informing us
of some creation options and a general reason for a failure.

Release justification: low risk logging related changes

Release note (ops change): Changefeed creations and failures event logs
are now emitted to the telemetry channel
@samiskin samiskin mentioned this issue Apr 11, 2022
Merged
samiskin added a commit to samiskin/cockroach that referenced this issue Apr 11, 2022
@samiskin
changefeedccl: add structured logging for changefeed create and failed
53870c8 
Resolves cockroachdb#77283

Previously we didn't have any logging related to the new Telemetry
channel in the structured logs which are exported to Snowflake. This
patch adds events around changefeed creation and failure, informing us
of some creation options and a general reason for a failure.

Release justification: low risk logging related changes

Release note (ops change): Changefeed creations and failures event logs
are now emitted to the telemetry channel
@craig craig bot closed this as completed in 5c1ae72 Apr 11, 2022
[DEPRECATED] CDC automation moved this from Stability Period to Closed Apr 11, 2022
samiskin added a commit to samiskin/cockroach that referenced this issue Apr 11, 2022
@samiskin
changefeedccl: add structured logging for changefeed create and failed
2d88bb6 
Resolves cockroachdb#77283

Previously we didn't have any logging related to the new Telemetry
channel in the structured logs which are exported to Snowflake. This
patch adds events around changefeed creation and failure, informing us
of some creation options and a general reason for a failure.

Release justification: low risk logging related changes

Release note (ops change): Changefeed creations and failures event logs
are now emitted to the telemetry channel
@mari-crl
Copy link
Contributor

ℹ️ Hello! I am a human and not at all a robot! Look at my very human username! 🤖 🎶
🤔 Although I tried very hard to figure out what to do with this issue, more powerful human brains will need to help me. (specifically: Both Github and Jira issues were modified after exalate problems) 😖
🔄 Please visit this issue's mirror at CRDB-13517 and try to sync the two sides up manually. 🌟
✅ When you're finished, comment saying as much asn a member of Developer Infrastructure will be along to finish linking. 🔗
🚫 Note that until this is done, this issue is not and will not be synced to Jira with Exalate. 🚫
😅 Feeling lost? Don't worry about it! A member of @cockroachdb/exalate-22-cleanup-team will be along shortly to help! 👍
👷 Developer Infrastructure members: when ready, open Exalate from the right-hand menu of the mirror issue in Jira, then choose Connect and enter this issue's URN: cockroachdb/cockroach-77283. Either way, delete this comment when you're done. 🔑
🙏 Thank you for your compliance, my fellow humans! 🤖 👋

@rail
Copy link
Member

rail commented May 25, 2022

Manually synced with Jira

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@samiskin samiskin

Labels
A-cdc Change Data Capture C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-cdc
Projects
[DEPRECATED] CDC
  
Closed
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

changefeedccl: add structured logging for changefeed create and failed samiskin/cockroach
4 participants
@rail @samiskin @amruss @mari-crl