Changefeed monitoring guide #19296

katmayb · 2025-01-10T21:18:20Z

Fixes DOC-11998

This PR adds a guide for monitoring changefeeds, particularly as a pipeline. Includes recommended metrics, information on potential impact of high values, and suggested dashboards.

Preview

https://deploy-preview-19296--cockroachdb-docs.netlify.app/docs/v25.1/changefeed-monitoring-guide.html

github-actions · 2025-01-10T21:18:47Z

Files changed:

src/current/_includes/v24.3/sidebar-data/stream-data.json
src/current/_includes/v25.1/sidebar-data/stream-data.json
src/current/images/v24.3/changefeed-pipeline.svg:

src/current/images/v25.1/changefeed-pipeline.svg:

src/current/v24.3/change-data-capture-overview.md
src/current/v24.3/changefeed-monitoring-guide.md
src/current/v25.1/change-data-capture-overview.md
src/current/v25.1/changefeed-monitoring-guide.md

netlify · 2025-01-10T21:19:03Z

✅ Deploy Preview for cockroachdb-interactivetutorials-docs canceled.

Name	Link
🔨 Latest commit	`c81d7f0`
🔍 Latest deploy log	https://app.netlify.com/sites/cockroachdb-interactivetutorials-docs/deploys/67d88082a7255c000829f50f

netlify · 2025-01-10T21:19:04Z

✅ Deploy Preview for cockroachdb-api-docs canceled.

Name	Link
🔨 Latest commit	`c81d7f0`
🔍 Latest deploy log	https://app.netlify.com/sites/cockroachdb-api-docs/deploys/67d880824ef423000899efb4

netlify · 2025-01-10T21:26:08Z

✅ Netlify Preview

Name	Link
🔨 Latest commit	`f41db60`
🔍 Latest deploy log	https://app.netlify.com/sites/cockroachdb-docs/deploys/67818e9fe862350008649ae2
😎 Deploy Preview	https://deploy-preview-19296--cockroachdb-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

netlify · 2025-01-10T21:33:10Z

✅ Netlify Preview

Name	Link
🔨 Latest commit	`c81d7f0`
🔍 Latest deploy log	https://app.netlify.com/sites/cockroachdb-docs/deploys/67d880824c780b00087ef5f4
😎 Deploy Preview	https://deploy-preview-19296--cockroachdb-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

rohan-joshi

Made some comments - really appreciate all of this!

rohan-joshi · 2025-01-28T20:24:42Z

src/current/v24.3/changefeed-monitoring-guide.md

+    - [Sink errors over time](#sink-errors)
+    - [Retry counts](#downstream-delivery)
+
+## Common troubleshooting scenarios


after some thinking - let's remove the common troubleshooting scenario section, including all the subsections (high end to end latency, rangefeed pressure, sink perf issues).

rohan-joshi · 2025-01-28T20:25:38Z

src/current/v24.3/changefeed-monitoring-guide.md

+- Scoped by `changefeed_job_id`
+- Supported Versions: v23.2.13+, v24.1.6+, v24.2.4+, v24.3.0+
+
+## Suggested dashboards


Let's remove this section entirely

rohan-joshi · 2025-01-28T20:27:36Z

src/current/v24.3/changefeed-monitoring-guide.md

+    - Resource usage during catch-up after restarts.
+- Supported Versions: v23.2.3+, v24.1.0+
+
+### End-to-end performance metrics


bc we want to limit the verbosity of metrics to readers, let's remove the entire end to end performance metrics section

katmayb · 2025-01-31T19:33:20Z

@rohan-joshi I've removed the requested sections. When we chatted, you mentioned keeping the diagram as is — is that still the case? I suppose the end-to-end component of that could be a source of confusion if we're not calling it out anywhere. Happy to update the diagram, if you think best. Let me know!

(p.s. I do think we actually mention most of the end-to-end metrics elsewhere in the cdc docs)

katmayb · 2025-02-05T15:27:54Z

@rohan-joshi Just a friendly ping on the question above!

rohan-joshi · 2025-02-19T19:36:16Z

@kathancox yikes. Sorry for not responding.
I think it's okay to keep the diagram as is even if we don't reference all of the metrics in the write-up.

I'd rather not hide the subcomponents

katmayb · 2025-02-19T20:07:48Z

OK thanks! @rohan-joshi, @asg0451 any other changes you want me to make, before I push to a docs writer for a final review?

rohan-joshi

lgtm

asg0451 · 2025-01-21T21:25:44Z

src/current/v24.3/changefeed-monitoring-guide.md

+
+### High-level performance metrics
+
+- Metric: `(now() - changefeed.checkpoint_progress)`


replace this with changefeed.max_behind_nanos, which is now scoped in the next backport releases

asg0451 · 2025-01-21T21:37:51Z

src/current/v24.3/changefeed-monitoring-guide.md

+
+#### Batch latency
+
+- Metric: `changefeed.sink_batch_hist_nanos`


may need to note that some names differ in the datadog integration. eg losing the _nanos suffix, and in prometheus they replace dots with underscores, etc. applicable for all metrics

src/current/v24.3/changefeed-monitoring-guide.md

katmayb · 2025-03-04T19:22:51Z

I also just realized the "Supported versions" bullet for each metric doesn't make sense in versioned docs. I'm going to backport the page to the relevant versions and then have that bullet point list the correct version.

katmayb · 2025-03-05T16:19:07Z

@asg0451 I've worked on some of your feedback in the latest commit. However, it looks like your feedback preceded the edits I made for Rohan, which removed a lot of content. Can you PTAL and see if the change to max_behind_nanos is still what we want?

katmayb · 2025-03-13T15:18:44Z

In a Slack conversation, I suggested + confirmed the following changes, which would resolve any discrepancy in feedback @rohan-joshi / @asg0451:

Keep the End-to-end performance section (all of these metrics are in the docs anyway and this will also match the diagram).
As per this commit ^, remove the Supported dashboards and Common troubleshooting scenarios sections.
As per Miles' suggestion, move up changefeed.max_behind_nanos to the High-level performance metrics section because this metric is now scoped + alter the diagram to include this in the correct place.

@asg0451 @rohan-joshi These are now implemented. Kept both checkpoint_progress + max_behind_nanos in both the high-level section and progress tracking section. (My thought was that it maybe better to show these as useful in both context, but please correct me if I'm wrong.) I've adapted the diagram to represent this.

I'll push to a docs review later this afternoon (3/13) Thanks both!

rmloveland

LGTM, non-blocking comment is i don't think v25.1 docs are the right place to litigate all this per-metric old supported versions info but i'm guessing there are $reasons so .. LGTM

src/current/v25.1/changefeed-monitoring-guide.md

katmayb · 2025-03-17T19:58:57Z

LGTM, non-blocking comment is i don't think v25.1 docs are the right place to litigate all this per-metric old supported versions info but i'm guessing there are $reasons so .. LGTM

Yes, you're right! My mistake, I removed from the v24.3 docs and then forgot to remove from the v25.1 docs.

rohan-joshi · 2025-03-25T17:11:50Z

yay

katmayb force-pushed the cdc-monitoring-guide branch from f41db60 to 8db2635 Compare January 10, 2025 21:21

katmayb force-pushed the cdc-monitoring-guide branch 2 times, most recently from edad6e5 to 1cd9186 Compare January 16, 2025 20:39

katmayb requested review from asg0451, rohan-joshi and shannonbradshaw January 16, 2025 20:59

rohan-joshi reviewed Jan 28, 2025

View reviewed changes

katmayb marked this pull request as ready for review February 5, 2025 15:27

katmayb requested a review from rohan-joshi February 5, 2025 15:28

katmayb changed the title ~~WIP changefeed monitoring guide~~ Changefeed monitoring guide Feb 7, 2025

rohan-joshi approved these changes Feb 27, 2025

View reviewed changes

katmayb requested a review from rmloveland March 4, 2025 15:24

asg0451 reviewed Mar 4, 2025

View reviewed changes

rmloveland removed their request for review March 4, 2025 19:41

katmayb requested a review from rmloveland March 13, 2025 18:41

rmloveland approved these changes Mar 17, 2025

View reviewed changes

WIP changefeed monitoring guide

cc96820

Kathryn May added 4 commits March 17, 2025 16:05

Remove requested sections

4d6ab3d

Feedback

496bc0b

Overall feedbakc TL/PM

985277b

Feedback fixes

c81d7f0

katmayb force-pushed the cdc-monitoring-guide branch from 849c82e to c81d7f0 Compare March 17, 2025 20:05

katmayb merged commit 3399c10 into main Mar 17, 2025
6 checks passed

katmayb deleted the cdc-monitoring-guide branch March 17, 2025 20:12


		### High-level performance metrics

		- Metric: `(now() - changefeed.checkpoint_progress)`


		#### Batch latency

		- Metric: `changefeed.sink_batch_hist_nanos`

Changefeed monitoring guide #19296

Changefeed monitoring guide #19296

Uh oh!

Conversation

katmayb commented Jan 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Preview

Uh oh!

github-actions bot commented Jan 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Files changed:

Uh oh!

netlify bot commented Jan 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for cockroachdb-interactivetutorials-docs canceled.

Uh oh!

netlify bot commented Jan 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for cockroachdb-api-docs canceled.

Uh oh!

netlify bot commented Jan 10, 2025

✅ Netlify Preview

Uh oh!

netlify bot commented Jan 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Netlify Preview

Uh oh!

rohan-joshi left a comment

Choose a reason for hiding this comment

Uh oh!

rohan-joshi Jan 28, 2025

Choose a reason for hiding this comment

Uh oh!

rohan-joshi Jan 28, 2025

Choose a reason for hiding this comment

Uh oh!

rohan-joshi Jan 28, 2025

Choose a reason for hiding this comment

Uh oh!

katmayb commented Jan 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

katmayb commented Feb 5, 2025

Uh oh!

rohan-joshi commented Feb 19, 2025

Uh oh!

katmayb commented Feb 19, 2025

Uh oh!

rohan-joshi left a comment

Choose a reason for hiding this comment

Uh oh!

asg0451 Jan 21, 2025

Choose a reason for hiding this comment

Uh oh!

asg0451 Jan 21, 2025

Choose a reason for hiding this comment

Uh oh!

katmayb Mar 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

katmayb commented Mar 4, 2025

Uh oh!

katmayb commented Mar 5, 2025

Uh oh!

katmayb commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rmloveland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

katmayb commented Mar 17, 2025

Uh oh!

katmayb commented Jan 10, 2025 •

edited

Loading

github-actions bot commented Jan 10, 2025 •

edited

Loading

netlify bot commented Jan 10, 2025 •

edited

Loading

netlify bot commented Jan 10, 2025 •

edited

Loading

netlify bot commented Jan 10, 2025 •

edited

Loading

katmayb commented Jan 31, 2025 •

edited

Loading

katmayb commented Mar 13, 2025 •

edited

Loading