Skip to content

Update clickpipes error reporting section to mention the system table #3550

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jul 1, 2025

Conversation

tpanetti
Copy link
Contributor

Summary

Updated error reporting section on clickpipes to include the system error table for clickpipes.

Notes

Marking as draft until the notification email has been sent out to users.

@tpanetti tpanetti requested a review from a team March 24, 2025 19:47
Copy link

vercel bot commented Mar 24, 2025

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
clickhouse-docs ❌ Failed (Inspect) Jun 30, 2025 5:34am
2 Skipped Deployments
Name Status Preview Comments Updated (UTC)
clickhouse-docs-ru ⬜️ Ignored (Inspect) Visit Preview Jun 30, 2025 5:34am
clickhouse-docs-zh ⬜️ Ignored (Inspect) Visit Preview Jun 30, 2025 5:34am

@@ -70,8 +70,8 @@ Steps:
<Image img={cp_custom_role} alt="Assign a custom role" size="lg" border/>

## Error reporting {#error-reporting}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it could be nice to add a bit of structure and elaborate a bit:

Logging and Errors

And then have two separate sections:

Record Errors Table

ClickPipes will create a table next to your destination table with the postfix <destination_table_name>_clickpipes_error. This table will contain any errors from malformed data or mismatched schema and will include the entirety of the invalid message. These errors are by default retained for ....

ClickPipe Logs

yada yada

This can be accessed via the UI or queried directly with clusterAllReplicas and maybe provide an example query?

@tpanetti tpanetti marked this pull request as ready for review June 25, 2025 00:33
@tpanetti tpanetti requested review from a team as code owners June 25, 2025 00:33
@tpanetti tpanetti requested a review from heavycrystal June 25, 2025 00:33
### System Errors {#system-errors}
Errors related to the operation of the ClickPipe will be stored in the `system.clickpipes_log` table. This will store all other errors related to the operation of your ClickPipe (network, connectivity, etc.). This table has a [TTL](/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-ttl) of 7 days.

If ClickPipes cannot connect to a data source or destination after 15 min., the ClickPipes instance stops and stores an appropriate message in the system error table (providing the ClickHouse instance is available).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

15min for data source connectivity is correct, but we give 1h for destination connectivity.

@tpanetti tpanetti merged commit 75f176a into main Jul 1, 2025
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants