Skip to content

Failed to add new rollup rule to existing GraphiteMergeTree table in v25.3 #100814

@opoliako

Description

@opoliako

Company or project name

No response

Question

Hi team,
We are running a cluster of 6 nodes (2 shards, 3 replicas each) on v25.3.6.10034 altinitystable release with 3 zookeeper nodes as coordination layer.
Recently we wanted to add new rollup rule to test min and max functions.
Here is snippet of our retention.xml file before testing:

<?xml version="1.0"?>
<yandex>
	<graphite_rollup>
		<!-- general test rule -->
		<pattern>
			<regexp>^test\.</regexp>
			<function>avg</function>
			<retention>
				<age>0</age>
				<precision>60</precision>
			</retention>
			<retention>
				<age>86400</age>
				<precision>7200</precision>
			</retention>
		</pattern>
. . . 

We tried to add additional rule above the one you have seen now, so file looked like this:

?xml version="1.0"?>
<yandex>
	<graphite_rollup>
		<!-- testing save of minimal metric -->
		<pattern>
			<regexp>^test\.minimal\.</regexp>
			<function>min</function>
			<retention>
				<age>0</age>
				<precision>60</precision>
			</retention>
		</pattern>
		<!-- general test rule -->
		<pattern>
			<regexp>^test\.</regexp>
			<function>avg</function>
			<retention>
. . .

The issue is that after retention file is updated and clickhouse-server is stop-started, replica becomes read-only, throwing this error:

<Error> data.graphite (ReplicatedMergeTreeAttachThread): Initialization failed, table will remain readonly. Error: Code: 342. DB::Exception: Existing table metadata in ZooKeeper differs in graphite params. Stored in ZooKeeper hash: 314***350, local hash: 136***314. (METADATA_MISMATCH)...

We tried updating every node one-by-one, but whole cluster becomes read-only.
We also tried running SYSTEM RESTORE REPLICA and SYSTEM RESTART REPLICA commands, with no success.
Is this expected behaviour?
Or am I missing something and there is a way to change rollup without manipulations with zookeeper's metadata?

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions