Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 87 additions & 0 deletions knowledgebase/restore-replica-after-storage-failure.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
---
date: 2025-11-19
title: How to restore a replica after storage failure
tags: ['Deployments and Scaling']
keywords: ['restore', 'replica', 'storage failure', 'atomic database']
description: 'This article explains how to recover data when using replicated tables in atomic databases in ClickHouse and disks/storage on one of the replica is lost/currupted.'
---

{frontMatter.description}
{/* truncate */}

<br/>
<br/>

<VerticalStepper headerLevel="h2">

:::note
This guide assumes that the `<path>` parameter in your config.xml file is set to:

```text
<path>/var/lib/clickhouse/</path>
```

If you have configured a different data path, replace all instances of `/var/lib/clickhouse` in the below commands with the actual value of your `<path>` setting.
:::

## Copy access configuration from the healthy replica {#copy-access-config}

Copy the contents of the `access` folder which contains local users from the healthy replica:

```text
/var/lib/clickhouse/access
```

## Back up the metadata folder from the healthy replica

1. Navigate to the ClickHouse data directory:

```text
cd /var/lib/clickhouse
```

2. Create a backup of the metadata folder (including symbolic links): The metadata directory contains DDLs for databases and tables.
The database directory has symlinks to `/var/lib/clickhouse/store/..` which contains all the table DDLs.

```bash
{ find metadata -type f; find metadata -type l; find metadata -type l | xargs readlink -f; } | tar -cPf backup.tar --files-from=-
```

:::note
This command ensures that both the **metadata files**, and the symlink architecture are preserved in the backup.
:::

## Restore the metadata on the faulty replica {#restore-the-metadata-on-the-faulty-replica}

1. Copy the generated `backup.tar` file to the faulty replica.
2. Extract it to the ClickHouse data directory:

```text
cd /var/lib/clickhouse/
tar -xvPf backup.tar
```

## Create the force restore flag {#create-force-restore-flag}

To trigger automatic data synchronization from other replicas, create the following flag:

```text
sudo -u clickhouse touch /var/lib/clickhouse/flags/force_restore_data
```

## Restart the faulty replica {#restart-faulty-replica}

1. Restart the ClickHouse server on the faulty node.
2. Check the server logs, you should observe parts being downloaded from the healthy replicas:

```bash
2025.11.02 00:00:04.047097 [ 682 ] {} <Debug> analytics.events_local (...) (Fetcher): Downloading files 23
2025.11.02 00:00:04.055542 [ 682 ] {} <Debug> analytics.events_local (...) (Fetcher): Download of part 202511_0_0_0 onto disk disk2 finished.
2025.11.02 00:00:04.101888 [ 687 ] {} <Debug> warehouse.customers_local (...) (Fetcher): Downloading part 2025_0_0_1 onto disk default.
2025.11.02 00:00:04.102005 [ 687 ] {} <Debug> warehouse.customers_local (...) (Fetcher): Downloading files 11
2025.11.02 00:00:04.102210 [ 690 ] {} <Debug> warehouse.customers_local (...) (Fetcher): Downloading part 2022_0_0_1 onto disk disk1.
2025.11.02 00:00:04.102247 [ 688 ] {} <Debug> warehouse.customers_local (...) (Fetcher): Downloading part 2021_0_0_1 onto disk disk2.
2025.11.02 00:00:04.102331 [ 690 ] {} <Debug> warehouse.customers_local (...) (Fetcher): Downloading files 11
```

</VerticalStepper>
2 changes: 1 addition & 1 deletion plugins/remark-custom-blocks.js
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ const plugin = (options) => {
visit(tree, 'mdxJsxFlowElement', (node, index, parent) => {
// Look specifically for the <VerticalStepper> tag used in the markdown file
if (node.name === 'VerticalStepper') {
try {
try{
// --- 1. Parse <VerticalStepper> Attributes ---
const jsxAttributes = node.attributes || [];
let type = "numbered"; // Default type
Expand Down
4 changes: 4 additions & 0 deletions scripts/build.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
#!/bin/bash

# Set locale to prevent perl warnings in child processes
export LC_ALL=C
export LANG=C

function parse_args() {
locale="" # Default: No locale specified
out_dir="" # Default: No custom output directory
Expand Down
4 changes: 4 additions & 0 deletions scripts/sed_links.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
#!/bin/bash

# Set locale to prevent perl warnings
export LC_ALL=C
export LANG=C

# Sometimes we need to change links which are reporting as broken
# but they are from the ClickHouse/ClickHouse repo. In this case
# it's useful to sed the link so that the build can pass and then
Expand Down
4 changes: 2 additions & 2 deletions src/components/CodeViewer/CodeInterpreter.tsx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { Button, Icon, RadioGroup, Tooltip } from '@clickhouse/click-ui/bundled'
import { createClient as createWebClient } from '@clickhouse/client-web'
import { parse } from 'json5'
import JSON5 from 'json5'
import { useEffect, useState } from 'react'
import short from 'short-uuid'
import CodeResults, { DefaultView } from './CodeResults'
Expand Down Expand Up @@ -90,7 +90,7 @@ function CodeInterpreter({
params.forEach((param) => {
if (param.type && /^(Array|Map|Tuple|Nested)/.test(param.type)) {
try {
query_params[param.name] = parse(param.value)
query_params[param.name] = JSON5.parse(param.value)
} catch (e) {
// just send and let clickhouse error
query_params[param.name] = param.value
Expand Down
25 changes: 25 additions & 0 deletions src/theme/BlogLayout/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
import React from 'react';
import BlogLayout from '@theme-original/BlogLayout';
import {ClickUIProvider} from '@clickhouse/click-ui/bundled';
import {useColorMode} from "@docusaurus/theme-common";

function BlogContentWithProvider({children}) {
const { colorMode } = useColorMode();
return (
<ClickUIProvider theme={colorMode}>
{children}
</ClickUIProvider>
);
}

export default function BlogLayoutWrapper(props) {
const {children, ...otherProps} = props;

return (
<BlogLayout {...otherProps}>
<BlogContentWithProvider>
{children}
</BlogContentWithProvider>
</BlogLayout>
);
}
4 changes: 3 additions & 1 deletion src/theme/MDXComponents.js
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ const enhancedComponents = {
// Map to the components expected from the remark plugin
Stepper: VStepper,
Step: VStepper.Step,
// Also map VerticalStepper directly for cases where remark plugin doesn't transform it
VerticalStepper: VStepper,
};

export default enhancedComponents;
export default enhancedComponents;