Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix wrong restoration of Materialized views with view name starting with digits #942

Merged
merged 1 commit into from
Jun 30, 2024

Conversation

praveenthuwat
Copy link
Contributor

@praveenthuwat praveenthuwat commented Jun 30, 2024

Hi, If the materialized view names start with a digit and are enclosed in a backtick(`) during the view creation, the regex group capture for matching the TO database is missing. This further leads to the wrong restoration of materialized views, as the TO still points to the source database!

For example:- If we run the following restore command,
clickhouse-backup restore_remote --rm -m default:praveen <backup-name>

The resulting materialized view will be the following. As you can see, the TO is still pointing to the source DB name, i.e., default instead of praveen)

CREATE MATERIALIZED VIEW praveen.`32da477b_0eb9_4aa6_bcbb_7f511649861e_v0_5_min_mv_shard` TO default.`32da477b_0eb9_4aa6_bcbb_7f511649861e_v0_5_min_mv_inner_shard` (

    `__created_at` AggregateFunction(max,
 DateTime),

    `__occurred_at` DateTime,

    `environment_id` UUID,

    `aggregation` String,

    `value` AggregateFunction(sum,
 Float64)
) AS
SELECT
    maxState(__created_at) AS __created_at,

    toStartOfFiveMinute(__occurred_at) AS __occurred_at,

    environment_id,

    aggregation,

    sumState(value) AS value
FROM praveen.`32da477b_0eb9_4aa6_bcbb_7f511649861e_v0_5_min_shard` GROUP BY
    environment_id,

    aggregation,

    __occurred_at
ORDER BY
    environment_id ASC,

    aggregation ASC,

    __occurred_at ASC;

The expected one:-

CREATE MATERIALIZED VIEW praveen.`32da477b_0eb9_4aa6_bcbb_7f511649861e_v0_5_min_mv_shard` TO praveen.`32da477b_0eb9_4aa6_bcbb_7f511649861e_v0_5_min_mv_inner_shard` (

    `__created_at` AggregateFunction(max,
 DateTime),

    `__occurred_at` DateTime,

    `environment_id` UUID,

    `aggregation` String,

    `value` AggregateFunction(sum,
 Float64)
) AS
SELECT
    maxState(__created_at) AS __created_at,

    toStartOfFiveMinute(__occurred_at) AS __occurred_at,

    environment_id,

    aggregation,

    sumState(value) AS value
FROM praveen.`32da477b_0eb9_4aa6_bcbb_7f511649861e_v0_5_min_shard` GROUP BY
    environment_id,

    aggregation,

    __occurred_at
ORDER BY
    environment_id ASC,

    aggregation ASC,

    __occurred_at ASC;

…n case the table names are enclosed in a "`". This further leads to wrong restore of materialized views(the TO still point to default database!).

Restore command used:
clickhouse-backup restore_remote --rm -m praveen:default <backup-name>

For example:- If we run the following restore command, the resulting materialized view will be the following.

CREATE MATERIALIZED VIEW praveen.`32da477b_0eb9_4aa6_bcbb_7f511649861e_v0_5_min_mv_shard` TO default.`32da477b_0eb9_4aa6_bcbb_7f511649861e_v0_5_min_mv_inner_shard`
(

    `__created_at` AggregateFunction(max,
 DateTime),

    `__occurred_at` DateTime,

    `environment_id` UUID,

    `aggregation` String,

    `value` AggregateFunction(sum,
 Float64)
) AS
SELECT
    maxState(__created_at) AS __created_at,

    toStartOfFiveMinute(__occurred_at) AS __occurred_at,

    environment_id,

    aggregation,

    sumState(value) AS value
FROM praveen.`32da477b_0eb9_4aa6_bcbb_7f511649861e_v0_5_min_shard`
GROUP BY
    environment_id,

    aggregation,

    __occurred_at
ORDER BY
    environment_id ASC,

    aggregation ASC,

    __occurred_at ASC;

The expected one:-

CREATE MATERIALIZED VIEW praveen.`32da477b_0eb9_4aa6_bcbb_7f511649861e_v0_5_min_mv_shard` TO praveen.`32da477b_0eb9_4aa6_bcbb_7f511649861e_v0_5_min_mv_inner_shard`
(

    `__created_at` AggregateFunction(max,
 DateTime),

    `__occurred_at` DateTime,

    `environment_id` UUID,

    `aggregation` String,

    `value` AggregateFunction(sum,
 Float64)
) AS
SELECT
    maxState(__created_at) AS __created_at,

    toStartOfFiveMinute(__occurred_at) AS __occurred_at,

    environment_id,

    aggregation,

    sumState(value) AS value
FROM praveen.`32da477b_0eb9_4aa6_bcbb_7f511649861e_v0_5_min_shard`
GROUP BY
    environment_id,

    aggregation,

    __occurred_at
ORDER BY
    environment_id ASC,

    aggregation ASC,

    __occurred_at ASC;
@coveralls
Copy link

coveralls commented Jun 30, 2024

Pull Request Test Coverage Report for Build 9731076406

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 265 unchanged lines in 13 files lost coverage.
  • Overall coverage decreased (-2.0%) to 65.543%

Files with Coverage Reduction New Missed Lines %
pkg/backup/create.go 2 75.58%
pkg/backup/restore.go 2 67.08%
pkg/backup/backuper.go 2 65.12%
cmd/clickhouse-backup/main.go 2 77.19%
pkg/backup/list.go 2 63.44%
pkg/clickhouse/clickhouse.go 4 78.71%
pkg/storage/object_disk/object_disk.go 5 70.95%
pkg/status/status.go 9 70.2%
pkg/config/config.go 10 71.24%
pkg/storage/general.go 23 61.46%
Totals Coverage Status
Change from base Build 9699341182: -2.0%
Covered Lines: 8552
Relevant Lines: 13048

💛 - Coveralls

@Slach Slach added this to the 2.5.17 milestone Jun 30, 2024
@Slach
Copy link
Collaborator

Slach commented Jun 30, 2024

thanks for contribution

@Slach Slach merged commit 57834ab into Altinity:master Jun 30, 2024
20 checks passed
@praveenthuwat praveenthuwat deleted the restore-db-mapping-regex-fix branch July 1, 2024 00:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants